Swagger path ref remote URL of another Swagger document

I am try to set a path's ref to another document in the remote server like

swagger: '2.0'
info:
  title: REST API specification
  description: >-
    test
host: 'localhost:3000'
basePath: /rest/abc/v4
schemes:
  - http
paths:
  /proxy2:
    $ref: "http://localhost:8080/api-docs/abc-rest-api-spec.yaml#/config"

Where abc-rest-api-spec.yaml is in another server. But, I do not see the API /proxy2 in swagger document. Am I missing something here ?