4 ms·

Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that

Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea. I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.

jeromechoo 2mo ago
It feels like we've hit a point where search engines can become what "todo list apps" were for devs 10 years ago. What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.
- marginalia_nu 2mo ago
  To be fair they are a supremely interesting problem to hack away at, and one that will meet you where you are. Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence. Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing. It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later.
alightsoul 2mo ago
I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving them money There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
- burger_moon 2mo ago
  > There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages Where do you get stats on this info? Not refuting your claims but it's interesting data. I would've thought a wider gap in all biz to active websites since some countries run nearly all their biz on whatsapp/telegram/wechat.
  - alightsoul 2mo ago
    I remember 300 million is the number of businesses on Google maps. I forgot how I got that number though. I didn't crawl Google maps. A lot or even most of those businesses run via chat often have no physical presence besides the houses of their owners so houses on Google maps are not counted. There are a lot of businesses on shopping malls and B2B businesses that aren't on Google maps often because they don't know they can add themselves to Google maps. I did count the number of ICANN domains on their zone files on czds and there are a few services selling pre compiled domain lists that do crawling for marketers (it is forbidden by ICANN to use zone files for marketing) I don't remember how I got the active website count of 100 to 150 million, maybe I interpolated by checking some ICANN domains or there's a domain list service that does it. Domain list services usually say how many domains are active regardless of whether they have a website or not Of course parked domains add up to the active website count even though they have no relevant content
- ccgreg 2mo ago
  Both IA and CCF are private foundations and accept donations.

jeromechoo 2mo ago

It feels like we've hit a point where search engines can become what "todo list apps" were for devs 10 years ago. What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.

marginalia_nu 2mo ago
To be fair they are a supremely interesting problem to hack away at, and one that will meet you where you are. Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence. Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing. It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later.

alightsoul 2mo ago

I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving them money There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages

burger_moon 2mo ago
> There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages Where do you get stats on this info? Not refuting your claims but it's interesting data. I would've thought a wider gap in all biz to active websites since some countries run nearly all their biz on whatsapp/telegram/wechat.
- alightsoul 2mo ago
  I remember 300 million is the number of businesses on Google maps. I forgot how I got that number though. I didn't crawl Google maps. A lot or even most of those businesses run via chat often have no physical presence besides the houses of their owners so houses on Google maps are not counted. There are a lot of businesses on shopping malls and B2B businesses that aren't on Google maps often because they don't know they can add themselves to Google maps. I did count the number of ICANN domains on their zone files on czds and there are a few services selling pre compiled domain lists that do crawling for marketers (it is forbidden by ICANN to use zone files for marketing) I don't remember how I got the active website count of 100 to 150 million, maybe I interpolated by checking some ICANN domains or there's a domain list service that does it. Domain list services usually say how many domains are active regardless of whether they have a website or not Of course parked domains add up to the active website count even though they have no relevant content
ccgreg 2mo ago
Both IA and CCF are private foundations and accept donations.