3 ms·
I mean. Lets be real. Search engines/a publically traversable & reality reflecting of the physically accessible WWW is by design outlawed. DMCA saw to that. No
by salawat 2y ago
I mean. Lets be real.
Search engines/a publically traversable & reality reflecting of the physically accessible WWW is by design outlawed. DMCA saw to that. Now, if you want that, you have to spider. The moment you create that representation and make it available to anyone else for any level of financial gain, and it's all over.
...And given all the assholery of the typical scraper these days, yhe Web is increasingly having to do away with good faith serving to novel bots; because assholes gonna asshole.
Can't have noce things.
- Terr_ 2y agoI'm having a little trouble parsing this comment. > Search engines/a publically traversable & reality reflecting of the physically accessible WWW is by design outlawed. DMCA saw to that. Do you mean in the sense that the DMCA forces everyone to censor their data on demand, and therefore data cannot be both accurate (containing certain items) and public? > Now, if you want that, you have to spider. Isn't "that" already the same thing as "spidering"? Or do you mean that "you" (i.e. the data consumer) have to do your own in-house spidering, because nobody is allowed to centralize the work and then redistribute (complete) data?