3 ms·
No yesterday, I just tried to spin up an old project using common crawl, which is hosted on AWS open data, seems like it's gotten quite popular since I lasted w
by winddude 3y ago
No yesterday, I just tried to spin up an old project using common crawl, which is hosted on AWS open data, seems like it's gotten quite popular since I lasted worked on it, and they're experiencing throttling from over usage, maybe even ddos (or just a really bad developer).
But seems to be a pretty common thing with stuff related to data and crawling. Everyone is cracking down on crawlers, adding paywalls, removing previously free APIs, so coming back to old projects in that domain seems to be impossible.
- syndicatedjelly 3y agoWhat's the lesson to be learned from that experience?
- winddude 3y agoFinish something and exploit (I mean market) the shit out of it. Or find a lot more money somewhere and build a lot more from scratch.