5 ms·
That's why I built Magnetissimo: https://github.com/sergiotapia/magnetissimo https://github.com/sergiotapia/magnetissimo It's easy to build a crawler, the hard
by sergiotapia 10y ago
That's why I built Magnetissimo: https://github.com/sergiotapia/magnetissimo https://github.com/sergiotapia/magnetissimo
It's easy to build a crawler, the hard part is spending time on bullshit like DMCA takedowns and such. Even if you say: "I just crawl, I don't provide download links, I don't even know what is indexed", you have to deal with legal issues.
So, host your own locally or for your community. No big deal.
Even Google has to deal with DMCA!
- TuringTest 10y agoHow large is the database compiled by Magnetissimo? Is it Wikipedia-sized (a few dozens of Gigabytes), or several orders of magnitude above or below?
- sergiotapia 10y agoHaven't really measured to be honest. But it's only 1 model with a few fields. Shouldn't be humongous.