3 ms·

I'm always surprised by how many web scraping frameworks/libraries I see sprout here on HN on a regular basis. Is web scraping something people are doing, or is

I'm always surprised by how many web scraping frameworks/libraries I see sprout here on HN on a regular basis. Is web scraping something people are doing, or is web scraping the new high-concurrency version of the "to do list" utility everyone used to write as an exercise? This is an honest question, I'm not trying to take a dig at anyone in particular.

mlevental 9y ago
i worked at a startup for 6 months (fintech) whose entire value prop was built on the back of scraping.
- dguaraglia 9y ago
  Oh, I know there are very niche cases of companies making money out of it (one of the two jobs in my life that required web scraping was a freelance job for a company doing sector-specific news aggregation) I just find the stream of new libraries for doing this kind of fascinating.
CyberShadow 9y ago
Just yesterday I wrote a program to scrape some Amazon search and product pages [0]. Why? Because Amazon's search is outright broken. The number of results changes when you change sorting mode, and sometimes, sorting by a different criteria will just serve you a "no products found" error page. I'll generally write a personal product comparison program when it becomes clear that I can't be certain that I can find the best product by hand. Often, even specialized websites that should have parametrized product search/filtering don't have their data properly indexed, so you have to scrape and parse it yourself. Another reason is to cross-reference with data from other sources. E.g. what laptop can I buy that has the best single-threaded CPU performance (within some other restrictions) [1]? [0]: https://github.com/CyberShadow/choose-product/blob/master/amazon/common.d https://github.com/CyberShadow/choose-product/blob/master/am... [1]: https://github.com/CyberShadow/choose-product/blob/master/lenovo/choose.d https://github.com/CyberShadow/choose-product/blob/master/le...
- dguaraglia 9y ago
  Dammit, I see D code and I want to learn the language so badly. I just need to find the right project for it (I might have something in the pipeline, as there's some stuff I need to do with libav and I really hate the idea of using C++.)
curun1r 9y ago
I find that I frequently want to dump a table from some webpage into a sqlite db file and run queries on it. Most sites don't offer a CSV or XLS download option, so a simple script to scrape the data usually ends up being the simplest option.
- ospider 9y ago
  I wrote web scrapers as my daily work. For so many times, people(coworkers) just came to me saying "Hi, can you scrape this site for me?", however, what they want might just be `extracting` some table out of a login-protected page. It made no sense to simulate the login in a script or something and extract it by xpath or css. So I just came up with a Chrome Extension(100 SLOC) that tries to extracting the main table from a web page, then you can copy the table to Excel or anything you like. It's kinda like Readability, but only for tables, not articles. [1] https://github.com/yifeikong/listify/ https://github.com/yifeikong/listify/
feelin_googley 9y ago
This framework has been posted before. The solutions I write using homegrown utilities are both more elegant and faster than any framework or library I have ever seen posted to HN or recommended elsewhere. Not to mention smaller and more agile. IMHO. All frameworks and libraries that I have seen will all fail given the right input i.e. fuzzing. I think parsing and transforming the content from webpages is just viewed as work that no one wants to do because, for whatever reason, webpages are still unpredictable.
973015077 9y ago
Wapo

mlevental 9y ago

i worked at a startup for 6 months (fintech) whose entire value prop was built on the back of scraping.

dguaraglia 9y ago
Oh, I know there are very niche cases of companies making money out of it (one of the two jobs in my life that required web scraping was a freelance job for a company doing sector-specific news aggregation) I just find the stream of new libraries for doing this kind of fascinating.

CyberShadow 9y ago

Just yesterday I wrote a program to scrape some Amazon search and product pages [0]. Why? Because Amazon's search is outright broken. The number of results changes when you change sorting mode, and sometimes, sorting by a different criteria will just serve you a "no products found" error page. I'll generally write a personal product comparison program when it becomes clear that I can't be certain that I can find the best product by hand. Often, even specialized websites that should have parametrized product search/filtering don't have their data properly indexed, so you have to scrape and parse it yourself. Another reason is to cross-reference with data from other sources. E.g. what laptop can I buy that has the best single-threaded CPU performance (within some other restrictions) [1]? [0]: https://github.com/CyberShadow/choose-product/blob/master/amazon/common.d https://github.com/CyberShadow/choose-product/blob/master/am... [1]: https://github.com/CyberShadow/choose-product/blob/master/lenovo/choose.d https://github.com/CyberShadow/choose-product/blob/master/le...

dguaraglia 9y ago
Dammit, I see D code and I want to learn the language so badly. I just need to find the right project for it (I might have something in the pipeline, as there's some stuff I need to do with libav and I really hate the idea of using C++.)

curun1r 9y ago

I find that I frequently want to dump a table from some webpage into a sqlite db file and run queries on it. Most sites don't offer a CSV or XLS download option, so a simple script to scrape the data usually ends up being the simplest option.

ospider 9y ago
I wrote web scrapers as my daily work. For so many times, people(coworkers) just came to me saying "Hi, can you scrape this site for me?", however, what they want might just be `extracting` some table out of a login-protected page. It made no sense to simulate the login in a script or something and extract it by xpath or css. So I just came up with a Chrome Extension(100 SLOC) that tries to extracting the main table from a web page, then you can copy the table to Excel or anything you like. It's kinda like Readability, but only for tables, not articles. [1] https://github.com/yifeikong/listify/ https://github.com/yifeikong/listify/

feelin_googley 9y ago

This framework has been posted before. The solutions I write using homegrown utilities are both more elegant and faster than any framework or library I have ever seen posted to HN or recommended elsewhere. Not to mention smaller and more agile. IMHO. All frameworks and libraries that I have seen will all fail given the right input i.e. fuzzing. I think parsing and transforming the content from webpages is just viewed as work that no one wants to do because, for whatever reason, webpages are still unpredictable.

973015077 9y ago

Wapo