3 ms·
This website has a back-end, imo. It has crawlers, indexers, jobs, etc. Really the only thing it doesn't have is an api, a monolithic request handler, or a data
by markbnj 11y ago
This website has a back-end, imo. It has crawlers, indexers, jobs, etc. Really the only thing it doesn't have is an api, a monolithic request handler, or a database. Well they don't show a db but they probably have at least a list of sites to crawl. Also worth pointing out that, unless they are doing geolocation or otherwise trying to pin down what a valid request is for a specific IP, then someone should theoretically be able to extract the entire dataset as static files.
Lastly I agree with the other poster who found the first sentence disagreeable. I am not offended, but it just seemed out of place in a semi-smarmy way.
- oxplot 11y ago> someone should theoretically be able to extract the entire dataset as static files Yep, not only possible but trivial. One has to just figure out where the hash links are in each blob and follow them.