4 ms·

I was reading another thread about webscraping, someone mentioned CSS selectors being way quicker than xpath. I'm easy either way but apart from a more powerful

I was reading another thread about webscraping, someone mentioned CSS selectors being way quicker than xpath. I'm easy either way but apart from a more powerful syntax what other benefits are there?

byteface 4y ago
I'm not sure about quicker. Doesn't scrapy use elementpath?. which converts a css query to an xpath under the hood as there is no complete CSSOM available for python. Likely as there is no modern standards based python dom to operate on so doing it on lxml tree is probably the best option. I find the main difference is xpath can return an attribute value where as css returns the node. You can use either from the terminal in my lib... https://github.com/byteface/domonic https://github.com/byteface/domonic (as it uses elementpath like scrapy)
- byteface 4y ago
  sorry i meant cssselect... https://pypi.org/project/cssselect/ https://pypi.org/project/cssselect/ which converts to xpath.
mdaniel 4y ago
In my experience, it's not that CSS selectors are "more powerful," but rather "more legible." XPath is for sure more powerful, but also usually lower signal to noise ratio response.css("#the-id") # vs response.xpath("//*[@id='the-id']") Thankfully, Scrapy (well, pedantically "parsel") allows mixing and matching, using the one which makes the most sense response.css(".someClass").xpath(".//*[starts-with(text(), 'Price')]")
showerst 4y ago
CSS is nice because it's more readable than XPATH for longer queries, and is friendlier to newer programmers who didn't come up when XML was big. XPATH is generally more powerful for really gnarly things and for backtracking. "Show me the 3rd paragraph that's a sibling of the fourth div id="subhed" and contains the text "starting".
- dotancohen 4y ago
  > XPATH is generally more powerful... That is a convincing argument is you can back it up with an XPATH expression.
  - mdaniel 4y ago
    Well, the rest of their sentence summed it up pretty well; try and implement that example using CSS selectors Hell, even "find id=subhead and _go up one element_" isn't possible in CSS because that's not a problem it was designed to solve
  - showerst 4y ago
    Here’s an example of parsing some particularly annoying old school html. I’m not claiming it’s the _best_ way to do it, just that you can, and I’m not sure this one is doable with selectors. https://github.com/openstates/openstates-scrapers/blob/40246156dcd56ecaeb6130d6b610813992d90093/scrapers/mo/bills.py#L578 https://github.com/openstates/openstates-scrapers/blob/40246...
PigiVinci83 4y ago
Having a large codebase like ours, we find out that XPATH are more readable, but i understand it's a personal feeling. We don't have high frequency scraping, so the performances of CSS vs XPATH were not considered. It's an interesting point i'd like to write more about, thanks for sharing.
lancebeet 4y ago
In my experience, XPath selectors are easier to write but usually result in selectors that are less robust to DOM changes. It is possible to write reliable XPath selectors as well, but I often see XPath selectors breaking because of implicit assumptions about the DOM structure. I don't see this as often for CSS selectors since they encourage you to make more explicit assumptions. This is in the context of test automation of modern web apps with a virtual DOM. I'm sure things might be different in other areas.

I was reading another thread about webscraping, someone mentioned CSS selectors being way quicker than xpath. I'm easy either way but apart from a more powerful syntax what other benefits are there?

byteface 4y ago

I'm not sure about quicker. Doesn't scrapy use elementpath?. which converts a css query to an xpath under the hood as there is no complete CSSOM available for python. Likely as there is no modern standards based python dom to operate on so doing it on lxml tree is probably the best option. I find the main difference is xpath can return an attribute value where as css returns the node. You can use either from the terminal in my lib... https://github.com/byteface/domonic https://github.com/byteface/domonic (as it uses elementpath like scrapy)

byteface 4y ago
sorry i meant cssselect... https://pypi.org/project/cssselect/ https://pypi.org/project/cssselect/ which converts to xpath.

mdaniel 4y ago

In my experience, it's not that CSS selectors are "more powerful," but rather "more legible." XPath is for sure more powerful, but also usually lower signal to noise ratio response.css("#the-id") # vs response.xpath("//*[@id='the-id']") Thankfully, Scrapy (well, pedantically "parsel") allows mixing and matching, using the one which makes the most sense response.css(".someClass").xpath(".//*[starts-with(text(), 'Price')]")

showerst 4y ago

CSS is nice because it's more readable than XPATH for longer queries, and is friendlier to newer programmers who didn't come up when XML was big. XPATH is generally more powerful for really gnarly things and for backtracking. "Show me the 3rd paragraph that's a sibling of the fourth div id="subhed" and contains the text "starting".

dotancohen 4y ago
> XPATH is generally more powerful... That is a convincing argument is you can back it up with an XPATH expression.
- mdaniel 4y ago
  Well, the rest of their sentence summed it up pretty well; try and implement that example using CSS selectors Hell, even "find id=subhead and _go up one element_" isn't possible in CSS because that's not a problem it was designed to solve
- showerst 4y ago
  Here’s an example of parsing some particularly annoying old school html. I’m not claiming it’s the _best_ way to do it, just that you can, and I’m not sure this one is doable with selectors. https://github.com/openstates/openstates-scrapers/blob/40246156dcd56ecaeb6130d6b610813992d90093/scrapers/mo/bills.py#L578 https://github.com/openstates/openstates-scrapers/blob/40246...

PigiVinci83 4y ago

Having a large codebase like ours, we find out that XPATH are more readable, but i understand it's a personal feeling. We don't have high frequency scraping, so the performances of CSS vs XPATH were not considered. It's an interesting point i'd like to write more about, thanks for sharing.

lancebeet 4y ago

In my experience, XPath selectors are easier to write but usually result in selectors that are less robust to DOM changes. It is possible to write reliable XPath selectors as well, but I often see XPath selectors breaking because of implicit assumptions about the DOM structure. I don't see this as often for CSS selectors since they encourage you to make more explicit assumptions. This is in the context of test automation of modern web apps with a virtual DOM. I'm sure things might be different in other areas.