Ask HN: How can I determine who is using a given public IP address without DPI?

Sounds like a dumb question but hear me out: I'd like to determine who the service provider is for a given public IP address. I'm not talking about who owns it–that information is readily available–but instead who is actually using it. Let's take Netflix for example: if I see a stream of packets to a Netflix server and look up the owner of that destination IP I will invariably get Amazon because Netflix (like many service providers) run on Amazon. How could I identify that it is Netflix who is providing the service without doing DPI on the TLS handshake? (you can guess at it from the raw data stream because the server's hostname will be in plain text)