Can we used swagger.io for normal java library? or its only for REST APIs?
Swagger is meant to work based on a OpenAPI specification. But certainly it can be used as a library, it's just up to you.
OpenAPI Specification and Swagger tools are designed for REST APIs only.
To document arbitrary Java libraries, you can use Javadoc, Doxygen, DocFX, or similar tools.