10 ms·

This week I literally dug through the documentation of whois query APIs of all registrars and registries. The RIPE servers (and therefore APNIC, AFRINIC, too)

This week I literally dug through the documentation of whois query APIs of all registrars and registries. The RIPE servers (and therefore APNIC, AFRINIC, too) are much more detailed in what you can find out about their assigned number ranges via inverse queries. [1] ARIN and LACNIC both use their own response format (which is outdated, sucks and is super hard to parse). But now to the interesting part: You can query "the internet" for all customers that have specific IP ranges by that specific ISP. For example, this will show all customers with a static IP at Unitymedia, which is a VODANET subsidary ISP in Germany. I've chosen this example to reduce the wall of text, use VODANET with expectancy to get blocked due to too large response: whois -h whois.ripe.net -- "-i mnt-by UNITYMEDIA-MNT" The responses are - from an OSINT perspective - a nightmare, too. Because I know that literally all of those customers probably have an outdated router hardware that they cannot patch or update. It also nails down the possible hardware to less than 5 models (from ARRIS, AVM and UPC), which makes it very open to router exploits. (Remember UBEEkey? yeah, that's those routers [2]). Also, what the article only mentions marginally: WHOIS got deprecated since GDPR, back then by RIPE which seems to lead in terms of specification compliance. The new protocol that is pretty much the same, but in JSON is RDAP; which also allows to search for nameservers, for domains etc. and therefore superseeds the featureset of the WHOIS protocol. [3] [1] https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-database-documentation/types-of-queries/inverse-queries https://www.ripe.net/manage-ips-and-asns/db/support/document... [2] https://github.com/yolosec/upcgen https://github.com/yolosec/upcgen [3] https://datatracker.ietf.org/doc/html/rfc7482 https://datatracker.ietf.org/doc/html/rfc7482

tephra 5y ago
There must be around 4-5000 registries and registrars out there. That seems like an ungodly amount of work..
- cookiengineer 5y ago
  Actually, it's not that many that have documentation available. /s But, given that Donuts Inc are the overlord of the internet in terms of (g)TLDs, there's not as many as some might think. The native whois client on most distributions is on github and contains a self-maintained list of whois servers in case you're curious [1]. It also contains a list of all new gTLDs [2], which, for the most part are registered via Donuts Inc. in the end. It's kinda ridiculous that something like ".gmbh" is owned, hosted, and distributed by an American company in Texas :-/ The interesting part is that some companies like "dotBERLIN GmbH & Co KG" (for .berlin) operate their own whois servers but seemingly reuse the infrastructure that Donuts Inc. provides for them. The ICANNWiki also has some nice side-facts about registrars if you're more curious about their organizational structure [3]. Donuts Inc. also has some chinese TLDs (like .游戏 or .xn--unup4y which means "games") registered via the Spring Fields LLC company that is a subsidary of Donuts Inc. [4] and [5] So yeah, I guess I kind of agree with the centralization of the internet argument here. [1] https://github.com/rfc1036/whois/blob/next/tld_serv_list https://github.com/rfc1036/whois/blob/next/tld_serv_list [2] https://github.com/rfc1036/whois/blob/next/new_gtlds_list https://github.com/rfc1036/whois/blob/next/new_gtlds_list [3] https://icannwiki.org/Donuts https://icannwiki.org/Donuts [4] https://icannwiki.org/.%E6%B8%B8%E6%88%8F https://icannwiki.org/.%E6%B8%B8%E6%88%8F [5] https://newgtlds.icann.org/en/program-status/delegated-strings https://newgtlds.icann.org/en/program-status/delegated-strin...
  - tephra 5y ago
    Yes if we're talking about _registries_ (like the one for .se and .nu which I work for :) ) there are not that many (I'm on mobile so hard to search for up to date information but in 2018 it was ~800). But if you also looked at all the _registrars_ those numbered ~2500 in 2018, but yes those numbers are for registry operators many of which outsource to donuts et al.
    - cookiengineer 5y ago
      Touché. I now know what you meant with the previous reply. I used registries and registrar wrong in the previous comment. In my project I was focussing on getting the organization information for specific IP ranges in my project. So I dealt only with NICs and RIRs (or ICANN directly as they bootstrap the ASNs) in that case.
      - tephra 5y ago
        Ah right :) May I ask what you are doing? Your original comment read as some osint fun stuff.
        cookiengineer 5y ago
        I'm building a peer to peer Browser network that relies on trust ratios in order to find out seed/leech ratios or the trustworthiness of other peers in the sense of whether they could be malicious actors (sharing content too much, producing/modifying content too much etc). The problem I'm currently trying to solve is that I had the idea to have a pre-shipped vendor profile that contains the necessary information for IP ranges (ASN, organization, region, country, ISP/NAT etc) so that the discovery service for that doesn't have to do this and maybe I can get rid of a centralized peer discovery service completely. It's like the basic idea of an offline "map of the internet" that should be an approximation of who does what in which amount of network bandwidth and data. For example, data center IPs aren't as trustworthy or peers in the same ISP-NAT could also be censored when it comes to blocked websites, modified DNS responses etc). At this point it's a big experiment and I'm not sure whether I'm fundamentally wrong about this as I don't have any data to back it up. If you're curious, it's part of the Stealth Browser I'm building [1] and [2] [1] https://github.com/tholian-network/stealth https://github.com/tholian-network/stealth [2] https://github.com/tholian-network/stealth-vendor https://github.com/tholian-network/stealth-vendor

tephra 5y ago

There must be around 4-5000 registries and registrars out there. That seems like an ungodly amount of work..

cookiengineer 5y ago
Actually, it's not that many that have documentation available. /s But, given that Donuts Inc are the overlord of the internet in terms of (g)TLDs, there's not as many as some might think. The native whois client on most distributions is on github and contains a self-maintained list of whois servers in case you're curious [1]. It also contains a list of all new gTLDs [2], which, for the most part are registered via Donuts Inc. in the end. It's kinda ridiculous that something like ".gmbh" is owned, hosted, and distributed by an American company in Texas :-/ The interesting part is that some companies like "dotBERLIN GmbH & Co KG" (for .berlin) operate their own whois servers but seemingly reuse the infrastructure that Donuts Inc. provides for them. The ICANNWiki also has some nice side-facts about registrars if you're more curious about their organizational structure [3]. Donuts Inc. also has some chinese TLDs (like .游戏 or .xn--unup4y which means "games") registered via the Spring Fields LLC company that is a subsidary of Donuts Inc. [4] and [5] So yeah, I guess I kind of agree with the centralization of the internet argument here. [1] https://github.com/rfc1036/whois/blob/next/tld_serv_list https://github.com/rfc1036/whois/blob/next/tld_serv_list [2] https://github.com/rfc1036/whois/blob/next/new_gtlds_list https://github.com/rfc1036/whois/blob/next/new_gtlds_list [3] https://icannwiki.org/Donuts https://icannwiki.org/Donuts [4] https://icannwiki.org/.%E6%B8%B8%E6%88%8F https://icannwiki.org/.%E6%B8%B8%E6%88%8F [5] https://newgtlds.icann.org/en/program-status/delegated-strings https://newgtlds.icann.org/en/program-status/delegated-strin...
- tephra 5y ago
  Yes if we're talking about _registries_ (like the one for .se and .nu which I work for :) ) there are not that many (I'm on mobile so hard to search for up to date information but in 2018 it was ~800). But if you also looked at all the _registrars_ those numbered ~2500 in 2018, but yes those numbers are for registry operators many of which outsource to donuts et al.
  - cookiengineer 5y ago
    Touché. I now know what you meant with the previous reply. I used registries and registrar wrong in the previous comment. In my project I was focussing on getting the organization information for specific IP ranges in my project. So I dealt only with NICs and RIRs (or ICANN directly as they bootstrap the ASNs) in that case.
    - tephra 5y ago
      Ah right :) May I ask what you are doing? Your original comment read as some osint fun stuff.
      - cookiengineer 5y ago
        I'm building a peer to peer Browser network that relies on trust ratios in order to find out seed/leech ratios or the trustworthiness of other peers in the sense of whether they could be malicious actors (sharing content too much, producing/modifying content too much etc). The problem I'm currently trying to solve is that I had the idea to have a pre-shipped vendor profile that contains the necessary information for IP ranges (ASN, organization, region, country, ISP/NAT etc) so that the discovery service for that doesn't have to do this and maybe I can get rid of a centralized peer discovery service completely. It's like the basic idea of an offline "map of the internet" that should be an approximation of who does what in which amount of network bandwidth and data. For example, data center IPs aren't as trustworthy or peers in the same ISP-NAT could also be censored when it comes to blocked websites, modified DNS responses etc). At this point it's a big experiment and I'm not sure whether I'm fundamentally wrong about this as I don't have any data to back it up. If you're curious, it's part of the Stealth Browser I'm building [1] and [2] [1] https://github.com/tholian-network/stealth https://github.com/tholian-network/stealth [2] https://github.com/tholian-network/stealth-vendor https://github.com/tholian-network/stealth-vendor