3 ms·
Hello! Zack Maril here. I've been working on this research for over two years now. I got started working on it after I read everybody on Hacker News talking ab
by knuckleheads 6y ago
Hello! Zack Maril here.
I've been working on this research for over two years now. I got started working on it after I read everybody on Hacker News talking about how they kept getting blocked when they were crawling websites[0] and also recent antitrust developments[1]. This has been a huge part of my life these past few years and it wouldn't have happened without the Hacker News community, so, from the bottom of my heart, thank you.
If you would like to read more about this research, please check out Knuckleheads' Club[2]. I've published a lot of what I have so far there and plan on publishing more soon. You're all invited to join the club and I'll be checking this thread all day and responding to folks as best I can.
[0] https://news.ycombinator.com/item?id=9612345 https://news.ycombinator.com/item?id=9612345
[1] https://news.ycombinator.com/item?id=17941804 https://news.ycombinator.com/item?id=17941804
[2] http://knuckleheads.club http://knuckleheads.club
- marban 6y agoNever heard of this initiative but kudos for bringing it up. I run a popular news site[0] and merely aggregating ~150 sources costs me upwards of $1k/m just to crawl the basics. [0] https://upstract.com/ https://upstract.com/
- knuckleheads 6y agoYes!! Exactly!! Crawling is super expensive to do and it's hard to even get the money to start doing the crawling. And even then once you get started, a lot of websites block you out or only let Googlebot through the paywall. And we are launching the club today with the publishing of the NYT article, we have been pretty quiet about it so far for the last two years or so, doing the research in private.