3 ms·
Most probably the github repo will be suspended, even though it's self hosted, I think. And you still need to account for upstream changes.
by alanfranzoni 9y ago
Most probably the github repo will be suspended, even though it's self hosted, I think. And you still need to account for upstream changes.
- TeMPOraL 9y agoForget Github. It might be just a regular git repo on someone's server. IANAL, but could companies really go against a person hosting the source code of a software that scraps a website? Even if, that's probably solvable (by turning it into cat-and-mouse game, TPB/SciHub-style). Have a non-turing-complete set of conversion rules that do scrapping -> standardized-data-format via a general-purpose scrapper. Distribute those through P2P. There might be problems with malicious actors, but generally this should be possible to implement in a way that's mostly transparent for end users. > And you still need to account for upstream changes. Upstream changes will never be too extreme to handle manually, if we're talking scrapping solutions - because any such extreme changes would also piss off the regular browser users.