v3.0 (bug?) - swagger compliance assertion fails in v3.0 but is ok in v2.8

[JoostDG]

Hi,

I recently update to v3.0 of readyAPI. Looks nice!

However, I notice a difference in behavior on the Swagger Compliance Assertion. Might this be a bug at your end or is this assertion suddenly more strict in the v3.0? And if the latter, why does it flags this a failure?

Scenario:

Applying swagger compliance assertion to a rest response. In the assertion, I have a swagger url which contains a relative path to a yaml file. Strict validation is enabled (disabling this makes no diffrence to this issue). The validation happens and provides now these failures (note: some items are renamed for privacy reasons):

1) Instance failed to match all required schemas (matched only 0 out of 1) checked node: [/references] corresponding schema: [/properties/references]

If this is a valid assertion failure then it is not a clear error message in my opinion. In my JSON response I get back the following for the reference node:

Response of my usedResource:

   "references" : {
      "aReference" : "93269491-e89b-4f19-9d12-ec09b71aeaec",
      "bReference" : null,
      "cReference" : null
   }

Is the error saying that bReference and cReference cannot be null? In my swagger/yaml file reference is defined as: 

  usedResource:
    type: object
    description: Description of the request
    properties:
...
       references:
          readOnly: true
           description: References description
           allOf:
             - $ref: '#/definitions/ourReferences'
...
...
ourReferences:
    type: object
    description: Contains the references
    properties:
      aReference:
        type: string
        description: The reference a 
      bReference:
        type: string
        description: The reference b
      cReference:
        type: string
        description: The reference c

2) Instance type (string) does not match any allowed primitive type (allowed: ["object"]) checked node [/priority] corresponding schema [properties/priority]

In my JSON response I get back the following for the reference node:

Response of my usedResource:

"priority" : "NORMAL"

In my swagger/yaml file priority is defined as: 

usedResource:
    type: object
    description: Description of the request
    properties:
...  
    priority:
        description: The urgency.
        allOf:
          - $ref: '#/definitions/RequestPriority'
...
  RequestPriority:
    type: string
    enum:
      - NORMAL
      - URGENT

I cannot see what's wrong with this. Other similar nodes with similar type & enum are not resulting in a failure. Again, also v2.8 did not flag this as a failed compliance.

I could provide more examples, but I hope these suffice. 

[JoostDG]

Support request ticket logged where I refer to this post : 00413347

[TanyaGorbunova]

Thanks for submitting the support ticket. Please share the solution once you find it together with the support team.

[JoostDG]

Got a reply from support. Basically, it's not a SmartBear bug but a more strict assertion since 3.0.

Full response from support:

---

Hi Joost,

In ReadyAPI 3.0.0, we updated some Swagger libraries. These libraries had a bug with handling the allOf statements. The issue has been fixed, so now you get a failed assertion in ReadyAPI. Please see the below comments about the assertion messages: 

1. "instance failed to match all required schemas (matched only 0 out of 1) checked node: [/externalRefs] corresponding schema: [/properties/externalRefs]"
The properties in the ExternalReference schema have the string type. But, in the response, some properties have the null value which is not valid. In this case, you need to either correct the definition or the response.

2. "instance type (string) does not match any allowed primitive type (allowed: ["object"]) checked node: [/priority] corresponding schema: [/properties/priority]"
There is an incorrect reference in the enum types. You can't use the allOf statement if you want to refer to the enum type only, because enum is a simple value, and the allOf statement is used for objects. In this case, you need to fix the definition.

3. "instance failed to match all required schemas (matched only 0 out of 1) checked node: [/study] corresponding schema: [/properties/study]"
The studyId property of the Study object is a string, but the response returns an integer value for this property. In this case, you need to either correct the definition or the response.

For more information, see the Swagger 2.0 specification (https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md) and the JSON Schema specification (https://json-schema.org/specification.html).

Also, we have created an improvement to provide more detailed error messages in assertions.

Let me know if you have any additional questions!

Regards,
Anastasia Batazova
Customer Care Engineer

---

[TanyaGorbunova]

Thanks for the update, @JoostDG 

[JoostDG · Accepted solution]

Further replying to this topic as the solution was a bit too quick. If anyone from the community is interested -- there is a bug in the swagger compliance assertion currently.

Full details:

Point 2  from above reply from smartbear:

2. "instance type (string) does not match any allowed primitive type (allowed: ["object"]) checked node: [/priority] corresponding schema: [/properties/priority]"
There is an incorrect reference in the enum types. You can't use the allOf statement if you want to refer to the enum type only, because enum is a simple value, and the allOf statement is used for objects. In this case, you need to fix the definition.

Above point 2 from smartbear support was contested by our Swagger creator David:

I do not agree with point 2. allOf includes an array of schema objects. ‘{“type”: “string” } is a valid schema object.

Swagger specification screenshot:

 

JSON Schema draft 4 spec screenshot:

Smartbear came back with:

[...] according to the Swagger Specification, the inline or referenced schema must be a schema object, not a standard JSON schema: https://1drv.ms/u/s!AnCx6DcSb33_nRQ4LJCgqd1OAXiK.
Please see this article for reference: https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/#allof

So, it is not correct to apply the allOf keyword to this definition. [...]

David's reply:

When looking at the OpenAPI 3.0.2 documentation I read the following for allOf:

Link: https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#schemaObject

• allOf - Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema.

When I look at the definition of a Schema Object (not a standard JSON Schema) I see the following:

The Schema Object allows the definition of input and output data types. These types can be objects, but also primitives and arrays.

Properties

The following properties are taken directly from the JSON Schema definition and follow the same specifications:

• required
• enum

When looking for the definition of the type property we see the following:

type - Value MUST be a string. Multiple types via an array are not supported.

This means that unlike in the JSON Schema definition only one type is supported. ‘string‘ is a valid value for the type property according to https://tools.ietf.org/html/draft-wright-json-schema-00#section-4.2

And the example section includes the following example of a Schema Object:

JSON

{

  "type": "string",

  "format": "email"

}

YAML format:

type: stringformat: email

Adding the enum property to the the example above while removing the format yields our definition:

Our definition is a schema object describing a primitive string value.

type: string
enum:
    - NORMAL
    - URGENT

This looks to be perfectly according to spec. Am I missing something that indicates that the schema objects used in the allOf array must describe objects and not primitives or arrays?

Too which finally Smartbear replied:

Yes, you're right - the schema is valid. And, there is a bug in ReadyAPI for the Swagger Compliance Assertion. I've linked your case to this bug to let you know once it's fixed.

[TanyaGorbunova]

Thanks for your deep investigation, @JoostDG!

I hope that the issue will be fixed soon.