5 ms·

Point #2 seems to contradict point #1…?

TekMol 4y ago
True. To some extend. But I can usually grasp very quickly "Ah yes, inside an article-section, the name comes first, and the description comes second". Writers of scraping software don't like ":nth-of-type(3)" though. They prefer ".price". As it is easier to write and will survive more changes to the site.
- bryanrasmussen 4y ago
  I think from my experience .price is not to be preferred either, but rather XPath as that tends to be more robust, as per the following https://sepl.dibris.unige.it/publications/2016-leotta-JSEP.pdf https://sepl.dibris.unige.it/publications/2016-leotta-JSEP.p... discussed here https://news.ycombinator.com/item?id=24055733 https://news.ycombinator.com/item?id=24055733
  - TekMol 4y ago
    If you have a structure like this: <div class=article-section> <div>Hammer</div> <div> Buy this and everything will start to look like a nail </div> </div> <div class=price>123.45</div> </div> How would you use XPath, and how would it be more rubust than '.price'?
    - bryanrasmussen 4y ago
      first off I would probably want to use price as the starting point, if price is there and the contents of price when trimmed etc. is a decimal, cool. But if there isn't a .price because someone has changed it to .amount for some reason. Or hey started using Material-UI or some other framework and they no longer have any meaningful classes anywhere which is what I most often see people whining about. then you would probably get an absolute xpath to the item you want, then you would use the getRobustXPath(element, document) function for that element, using the linked library under discussion https://github.com/cyluxx/robula-plus https://github.com/cyluxx/robula-plus to get a robust xpath. This xpath will probably be pretty unreadable but you can then test to see it gets your stuff. How will it be more robust - well that depends but if I have something like this <div class=article-section> <div>Hammer</div> <div> Buy this and everything will start to look like a nail </div> </div> <div class=price>123.45</div> </div> I might first off want to get products and their descriptions and their prices together. And I would probably say hey it might be that they are under this kind of class based structure but that can change but what is less likely to change is that a 'price' will be A number with either . or , as the potential decimal delimiter, with potential currency symbols either directly in the element with the number, not found, or in a directly preceding or following element. It would be useful to find prices on the page before everything else, and from there to figure out where product names and where descriptions where in relation to the price. Dependent on language of page product layout will probably change, but in a western language the price will probably come 'last' in our product container, which is exactly what you have done here. A non-robust xpath for matching the above might look something like product = //[local-name()='article' || local-name()='main']//[child::*[last()][xs:decimal(text())] The above might have an error in the xpath because hey, writing without testing in environment, and also obviously really expensive as have not optimized. But it should also be clear enough that by removing the need to have matching elements be based on the incidentals of how we style those elements we can add up with data extraction that is less likely to break with site redesigns. on edit: obv. article or main check would be optimized beforehand, just some sites use both, some use one, and some neither so would have to do all sorts of things first, etc. etc.
      - TekMol 4y ago
        Your XPath seems to have an error: http://xpather.com/OIJidwZw http://xpather.com/OIJidwZw
        bryanrasmussen 4y ago
        yes, I said it probably would have since not testing it before writing it, think of it as pseudocode to make a point, not code we're going to put in production.
    - benibela 4y ago
      I, for one, would use neither CSS nor XPath for that, but my own selector language: <div class=article-section> <div></div> <div class=price>{.}</div> </div> It is the most robust. It picks the price in the article and not some other price that might on the page. And if there is no price, you get an exception that there is no price, rather than an empty result

True. To some extend. But I can usually grasp very quickly "Ah yes, inside an article-section, the name comes first, and the description comes second". Writers of scraping software don't like ":nth-of-type(3)" though. They prefer ".price". As it is easier to write and will survive more changes to the site.

bryanrasmussen 4y ago
I think from my experience .price is not to be preferred either, but rather XPath as that tends to be more robust, as per the following https://sepl.dibris.unige.it/publications/2016-leotta-JSEP.pdf https://sepl.dibris.unige.it/publications/2016-leotta-JSEP.p... discussed here https://news.ycombinator.com/item?id=24055733 https://news.ycombinator.com/item?id=24055733
- TekMol 4y ago
  If you have a structure like this: <div class=article-section> <div>Hammer</div> <div> Buy this and everything will start to look like a nail </div> </div> <div class=price>123.45</div> </div> How would you use XPath, and how would it be more rubust than '.price'?
  - bryanrasmussen 4y ago
    first off I would probably want to use price as the starting point, if price is there and the contents of price when trimmed etc. is a decimal, cool. But if there isn't a .price because someone has changed it to .amount for some reason. Or hey started using Material-UI or some other framework and they no longer have any meaningful classes anywhere which is what I most often see people whining about. then you would probably get an absolute xpath to the item you want, then you would use the getRobustXPath(element, document) function for that element, using the linked library under discussion https://github.com/cyluxx/robula-plus https://github.com/cyluxx/robula-plus to get a robust xpath. This xpath will probably be pretty unreadable but you can then test to see it gets your stuff. How will it be more robust - well that depends but if I have something like this <div class=article-section> <div>Hammer</div> <div> Buy this and everything will start to look like a nail </div> </div> <div class=price>123.45</div> </div> I might first off want to get products and their descriptions and their prices together. And I would probably say hey it might be that they are under this kind of class based structure but that can change but what is less likely to change is that a 'price' will be A number with either . or , as the potential decimal delimiter, with potential currency symbols either directly in the element with the number, not found, or in a directly preceding or following element. It would be useful to find prices on the page before everything else, and from there to figure out where product names and where descriptions where in relation to the price. Dependent on language of page product layout will probably change, but in a western language the price will probably come 'last' in our product container, which is exactly what you have done here. A non-robust xpath for matching the above might look something like product = //[local-name()='article' || local-name()='main']//[child::*[last()][xs:decimal(text())] The above might have an error in the xpath because hey, writing without testing in environment, and also obviously really expensive as have not optimized. But it should also be clear enough that by removing the need to have matching elements be based on the incidentals of how we style those elements we can add up with data extraction that is less likely to break with site redesigns. on edit: obv. article or main check would be optimized beforehand, just some sites use both, some use one, and some neither so would have to do all sorts of things first, etc. etc.
    - TekMol 4y ago
      Your XPath seems to have an error: http://xpather.com/OIJidwZw http://xpather.com/OIJidwZw
      - bryanrasmussen 4y ago
        yes, I said it probably would have since not testing it before writing it, think of it as pseudocode to make a point, not code we're going to put in production.
  - benibela 4y ago
    I, for one, would use neither CSS nor XPath for that, but my own selector language: <div class=article-section> <div></div> <div class=price>{.}</div> </div> It is the most robust. It picks the price in the article and not some other price that might on the page. And if there is no price, you get an exception that there is no price, rather than an empty result