5 ms·

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.co

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

agumonkey 13y ago
It also depends on a coherent structure in HTML websites. Domains running websites which are more like javascript frontend modules shouldn't be scraped at all, it screams for a public API.
- uptown 13y ago
  "it screams for a public API" But many content owners would never provide their data in this format even if doing-so would be trivial.
- jdavis703 13y ago
  These single page sites do have a public, albeit, undocumented API. If you analyze the network requests via the dev tools in your browser you'll have an XML/JSON data source that is probably structured better than the markup.
  - agumonkey 13y ago
    Of course, I should have thought about it that way.
- CMCDragonkai 13y ago
  Try using https://snapsearch.io/ https://snapsearch.io/ It is designed for JS sites.
yaph 13y ago
Use the tools suitable for the task. There are a lot of those "simple" sites and I'm pretty sure a lot of people will stick to those "dated" methods of building sites, because search traffic still matters.
johndavi 13y ago
The long tail is tough, but rules are useful when you only need to work with a small number of sites. And assuming, as you point out, less "modern" sites. (News sites tend to be mostly consistently manageable but, yes, smaller e-commerce players tend to adopt more modern techniques -- as befitting fashion-forward product lines, naturally). Our (Diffbot) approach is to learn what news and product (and other) pages look like, and obviate the rules-management -- we also fully execute JS when rendering. The web keeps evolving though, dang it. Tricky thing!
- lsh 13y ago
  Unfortunately Diffbot is not open source. Are you planning any F/OSS offerings?
egb 13y ago
Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?
- checker659 13y ago
  PhantomJS?
  - egb 13y ago
    "PhantomJS is a headless WebKit scriptable with a JavaScript API", so it's a browser. Is it also a webscraper that can pull data out of a page for me?
    - nols 13y ago
      I've heard of people using PhantomJS with CasperJS to scrape, not sure if it can be done solely with PhantomJS.
      - vaviloff 13y ago
        CasperJS is a higher-level wrapper for PhantomJS, so - yes, it could be done with PhantomJS solely... But you wouldn't want to, because CasperJS makes automation easier.
        uptown 13y ago
        Are there any libraries to facilitate database-connectivity to SQL Server or MySQL from javascript? I've used CasperJS to scrape some sites, but always fall back on post-processing the scraped data with another program in order to get it into my database. I'd love to be able to do it all from one piece of code.
        gee_totes 13y ago
        You could always have your CasperJS scraping script make an AJAX request to a RESTful API for your MySQL DB. You won't be doing it all from one piece of code, but you'll be doing about 90% of it.
alttab 13y ago
There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue. In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need. To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, that you could place inside any page and utilize local storage, you could scrape these dynamic sites by viewing the page first, and still get all of the cool gadetry provided by this tool. In essence, the value of this tool could be built as a bookmarklet. THAT SIR - I would use every, single, day.
- stedaniels 13y ago
  Why would scrape JIRA when they have a perfectly workable API?
  - lepht 13y ago
    While their APIs are nice, they require separate permissions and having access to them isn't always a given depending on the company that runs/owns the Jira instance.
    - deleted 13y ago
      [deleted]
    - alttab 13y ago
      This. I also would rather just grab it in the browser instead of having to run a server or something else.
- shaneofalltrad 13y ago
  Great idea on the bookmarklet. I could see a tool for building custom readers with clippings from various sites. Say I want to organize JavaScript array patterns and ideas. Throw in a way to clip parts of my PDF books into this "reader" and you have an amazing product worth millions.
CHsurfer 13y ago
At first, this seems correct. It's definately easier to get scraping with something like Capybara and a suitable js enabled driver, but in my experience, this solution is less reliable. Async loaded data can time out and don't get me started on the difficulties of running the scraper with cron jobs. In the end, I migrated even my JS heavy pages to Mechanize based solutions. It takes a few extra requests to get the async data, but once you get that figured out, it's rock solid - till they update the site design ;-)
CHsurfer 13y ago
At first, this seems correct. It's definately easier to get scraping with something like Capybara and a suitable js enabled driver, but in my experience, this solution is less reliable. Async loaded data can time out and don't get me started on the difficulties of running the scraper with cron jobs. In the end, I migrated even my JS heavy pages to Mechanize based solutions. It takes a few extra requests to get the async data, but once you get that figured out, it's rock solid - till they update the site design ;-)
CMCDragonkai 13y ago
I built SnapSearch for JS/SPA sites that need SEO. But it works for scraping as well. https://snapsearch.io/ https://snapsearch.io/ You can try the demo. I tried it with "http://www.asos.com/" http://www.asos.com/" and it worked properly. Note that empty content actually means that the webserver returned with no body content. The real API will return the headers as well the body. It works via Firefox, and it's load balanced and multithreaded. It takes care of all the thorny issues regarding async content... etc.

agumonkey 13y ago

It also depends on a coherent structure in HTML websites. Domains running websites which are more like javascript frontend modules shouldn't be scraped at all, it screams for a public API.

uptown 13y ago
"it screams for a public API" But many content owners would never provide their data in this format even if doing-so would be trivial.
jdavis703 13y ago
These single page sites do have a public, albeit, undocumented API. If you analyze the network requests via the dev tools in your browser you'll have an XML/JSON data source that is probably structured better than the markup.
- agumonkey 13y ago
  Of course, I should have thought about it that way.
CMCDragonkai 13y ago
Try using https://snapsearch.io/ https://snapsearch.io/ It is designed for JS sites.

yaph 13y ago

Use the tools suitable for the task. There are a lot of those "simple" sites and I'm pretty sure a lot of people will stick to those "dated" methods of building sites, because search traffic still matters.

johndavi 13y ago

The long tail is tough, but rules are useful when you only need to work with a small number of sites. And assuming, as you point out, less "modern" sites. (News sites tend to be mostly consistently manageable but, yes, smaller e-commerce players tend to adopt more modern techniques -- as befitting fashion-forward product lines, naturally). Our (Diffbot) approach is to learn what news and product (and other) pages look like, and obviate the rules-management -- we also fully execute JS when rendering. The web keeps evolving though, dang it. Tricky thing!

lsh 13y ago
Unfortunately Diffbot is not open source. Are you planning any F/OSS offerings?

egb 13y ago

Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?

checker659 13y ago
PhantomJS?
- egb 13y ago
  "PhantomJS is a headless WebKit scriptable with a JavaScript API", so it's a browser. Is it also a webscraper that can pull data out of a page for me?
  - nols 13y ago
    I've heard of people using PhantomJS with CasperJS to scrape, not sure if it can be done solely with PhantomJS.
    - vaviloff 13y ago
      CasperJS is a higher-level wrapper for PhantomJS, so - yes, it could be done with PhantomJS solely... But you wouldn't want to, because CasperJS makes automation easier.
      - uptown 13y ago
        Are there any libraries to facilitate database-connectivity to SQL Server or MySQL from javascript? I've used CasperJS to scrape some sites, but always fall back on post-processing the scraped data with another program in order to get it into my database. I'd love to be able to do it all from one piece of code.
        gee_totes 13y ago
        You could always have your CasperJS scraping script make an AJAX request to a RESTful API for your MySQL DB. You won't be doing it all from one piece of code, but you'll be doing about 90% of it.

alttab 13y ago

There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue. In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need. To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, that you could place inside any page and utilize local storage, you could scrape these dynamic sites by viewing the page first, and still get all of the cool gadetry provided by this tool. In essence, the value of this tool could be built as a bookmarklet. THAT SIR - I would use every, single, day.

stedaniels 13y ago
Why would scrape JIRA when they have a perfectly workable API?
- lepht 13y ago
  While their APIs are nice, they require separate permissions and having access to them isn't always a given depending on the company that runs/owns the Jira instance.
  - deleted 13y ago
    [deleted]
  - alttab 13y ago
    This. I also would rather just grab it in the browser instead of having to run a server or something else.
shaneofalltrad 13y ago
Great idea on the bookmarklet. I could see a tool for building custom readers with clippings from various sites. Say I want to organize JavaScript array patterns and ideas. Throw in a way to clip parts of my PDF books into this "reader" and you have an amazing product worth millions.

CHsurfer 13y ago

At first, this seems correct. It's definately easier to get scraping with something like Capybara and a suitable js enabled driver, but in my experience, this solution is less reliable. Async loaded data can time out and don't get me started on the difficulties of running the scraper with cron jobs. In the end, I migrated even my JS heavy pages to Mechanize based solutions. It takes a few extra requests to get the async data, but once you get that figured out, it's rock solid - till they update the site design ;-)

CHsurfer 13y ago

CMCDragonkai 13y ago

I built SnapSearch for JS/SPA sites that need SEO. But it works for scraping as well. https://snapsearch.io/ https://snapsearch.io/ You can try the demo. I tried it with "http://www.asos.com/" http://www.asos.com/" and it worked properly. Note that empty content actually means that the webserver returned with no body content. The real API will return the headers as well the body. It works via Firefox, and it's load balanced and multithreaded. It takes care of all the thorny issues regarding async content... etc.