Which certificate should your server use?
Anyone who connects to your server sees its certificate, so one probe can give the server away. Use a trusted certificate if you can. If you cannot, an honest self-signed certificate can still be better than a false one.
What each certificate tells a network
-
Trusted
A valid certificate for your domain, signed by a certificate authority.
It looks like an ordinary website or service.
-
Self-signed
A certificate no authority signed, with a made-up name, as on a new server.
It looks like a private server, such as a router or a test machine.
-
Invalid
A self-signed certificate that names a real site, at an address that is not that site's.
It looks like a server pretending to be something it is not.
Why self-signed can beat invalid
A browser rejects both. The difference is what each one claims.
A made-up name claims nothing, so there is nothing to disprove. Routers, test machines and private tools show the same kind of certificate every day, and a network that blocks them all cuts those off too.
A real site's name is a claim anyone can check: that site has a certificate from an authority and addresses of its own, and your server has neither. No ordinary server does that, so blocking it costs a network nothing.
A real site's name can still help on a network that goes by the name alone, where it can pass and a made-up name cannot. To try it, add one on a managed server or on a standalone server.
There is no point dressing up a self-signed certificate. Its organization, how long it lasts and its extensions change nothing, as a probe sees at once that no authority signed it.
Get a domain if you can
A trusted certificate is the real answer to a probe, and it costs little: a domain is cheap, and Let's Encrypt's certificates are free.
Use a host of the domain, such as api.example.com. Many sites keep an API or their files on a host of their own that refuses strangers, as your server does, so it looks ordinary. www.example.com stays yours for any site you like, on any hosting.
A domain does not get your users past a blocked address, though, as it leads to the same server. Only a reverse proxy does that.