2 ms·
How about: if you want me not to scrape it, keep it off my internet?? Actually I'm considering building "API-fication" of websites with bindings for major lang
by paulus_magnus2 9y ago
How about: if you want me not to scrape it, keep it off my internet??
Actually I'm considering building "API-fication" of websites with bindings for major languages (Java, Python, JS). With luck websites could participate by providing & maintaining a parseable API-sitemap.
This would open door to my 2nd project: orchestration a-la BPEL on top of websites. visual editor, macros, scripting. Call this PIPES 2.0
- _mhr_ 9y agoCan you provide some use-cases for why this would be useful in a way that wouldn't violate most sites' ToS?
- paulus_magnus2 9y ago- a lot of online stores, hotels need to constantly update prices based on what competitors do. - cleaning (big) data. Automatically reconcile data to canonical format / names using authoritative source (say wikipedia) Can you understand even the simplest TOS? I'd argue most (all?) are too restrictive to be enforceable. https://tosdr.org/ https://tosdr.org/