4 ms·
Web Scraping: Data for Everyone
- bernardlunn 3y agoSearch engines scrape data. Why not simply mandate that they have to make that data available to all?
- renegat0x0 3y agoI do not believe that Google would share anything if it was not required by law. I heard there are other public community driven solutions, not necessarily search engines. For example marginalia search engine provides some domain data, but I think it was distributed as some sort of binary info. I have developed myself an engine to scrap data, which is using beautiful soup. I export my data to a github repository, where it is available. All domains receive a page rating, based on their content. I believe that all pages should provide scrap-able title, description and a valid status code. We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code.
- marginalia_nu 3y agoMost of what I make available are just sample data for testing the search engine, it's a tiny fraction of the entire crawl set for purely logistical reasons. I don't have the bandwidth to upload datasets on the order of terabytes. Though I've begun distributing dumps of the domain and link database if anyone is interested: https://downloads.marginalia.nu/exports/ https://downloads.marginalia.nu/exports/ Might be persuaded to share the the document metadata databases as well in some form, as that's only order of 90 GB. Would contain titles and summaries and a few other flags for each URL. > We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code. Well there's already the WARC format for that.
- krauses 3y agoYou have unlocked the key to making the front page of YC, post about web scraping.
- deleted 3y ago[deleted]
- omneity 3y agoThat's an interesting observation indeed.
- xnx 3y agoAre there any existing or developing solutions for visual web scraping? It feels like the only way to capture data from emerging WEI and canvas-powered websites will soon be screen readers designed for accessibility and automated GUI interaction.
- tepitoperrito 3y agoYou could also explore approaches like what puppeteer-heap-snapshot does and extract the data you're interested in from the in memory Javascript objects that the canvas powered website uses to generate its contents.
- djsegal 3y agoYou mean something like octoparse.com or serpapi.com?
- remmargorp64 3y agooctoparse extracts data from DOM elements. It can't OCR or visually process a dynamic page. The page has to have a structured and predictable HTML DOM. Unless I'm missing something? And serpapi is just for google searches?
- datadrivenangel 3y agoOptical Character Recognition unfortunately.
- matheusmoreira 3y ago> canvas-powered websites Is this really happening right now? Are people really rendering their entire websites into an opaque canvas? Decades of moving the web platform forward and it has come to this?
- flashback2199 3y agoIf it's a flutter web app, yes. Reminds me of flash, kinda
- 3y ago
- CoffeeTails 3y agoAs someone who has only heard about web scraping but has had no idea about how it works this has been a really interesting read! Very easy to understand. Thanks!
- omneity 3y agoHey, your comment made my day! This is exactly why I write. Thanks!
- streamfunk191 3y agoAn interesting development is AI bots that can act as human agents and visit a site and extract information. I imagine these are only needed when there are barriers to use all other solutions, otherwise it’s a bit overlillt
- lopis 3y agoAnd barriers there are... Lots of websites only allow the Google bot and no one else.
- irishgeoff22 3y ago[dead]