$ref in response example relative to response example

I am using a $ref in a Swagger 2.0 API response example that, when the response is returned, is interpreted correctly as relative to the response itself:

responses:
  200:
    description: OK
    schema:
      "$ref": "#/definitions/CollectionOfFoos"
    examples:
      application/json:
        foos:
          - id: 00000000-0000-0000-0000-000000000000
            name: sample foo 0
            bar:
              id: 0
              "$ref": "#/refs/bars/0"
            baz:
              id: 42
              "$ref": "#/refs/bazzes/myBaz"
          - id: 00000000-0000-0000-0000-000000000001
            name: sample foo 1
            bar:
              "$ref": "#/refs/bars/0"
            baz:
              "$ref": "#/refs/bazzes/myOtherBaz"
        refs:
          bars:
            0:
              detail: "Additional detail about this bar..."
          bazzes:
            myBaz:
              detail: "Additional detail about this baz..."
            myOtherBaz:
              detail: "Additional detail about this other baz..."

So, were that example response to be sent to a consumer, the pointer in those $refs would point to the refs member on the response. This is a contrived example, but we do this for "normalization" purposes, so that more than one foo can refer to the same bar or baz.

The problem is that this doesn't comply with the spec, since those $ref elements that will later resolve correctly don't do so when the API including examples is checked for compliance. That first $ref("$ref": "#/definitions/CollectionOfFoos") is perfectly compliant, since in the API doc there is something at definitions/CollectionOfFoos, but there isn't anything at refs/bazzes/myBaz, etc., so those lines throw an error with the description $refs must reference a valid location in the document. Is there any way around this, or a better way to approach this problem?