4 ms·

I love requests+lxml, use it fairly regularly, just a few quick notes: 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for

by drej 9y ago

I love requests+lxml, use it fairly regularly, just a few quick notes: 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. 2. Don't forget to check the status code of r (r.status_code or less generally r.ok) 3. Those with a background in coding might prefer the .cssselect method available in whatever object the parsed document results in. That's obviously a tad slower than find/findall/xpath, but it's oftentimes too convenient to pass upon. 4. Kind of automatic, but I'll say it anyway - scraping is a gray area, always make sure that what you're doing is legitimate.

jacobush 9y ago
I had reason to gather news articles and extract keywords and authors - can't remember why I didn't use BeatifulSoup, because that was my first choice. In the end I used lxml with its html5parser: https://github.com/vonj/scraping https://github.com/vonj/scraping Regarding legality - something frowned upon is putting load on servers. You may get blocked, rate limited or worse if you put too much strain on servers. Especially when you as I did, experimented with different query options to get the data I needed, and had to rerun scraping a number of times. A neat trick I found, was to configure a web proxy locally (squid, in my case) to aggressively cache EVERYTHING. This was, new runs only went out to the news sites for new queries I had never run before. Very helpful, it also speeded up the development to access files locally (cached in squid) instead of having to go out to the internet all the time.
- jacobush 9y ago
  In that github repo there is an example squid config for caching permanently.
masklinn 9y ago
> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building. html5lib fixes that (and can construct both lxml and bs trees, and both libraries have html5lib integration), however it's slow. I don't know that there is a native compatible parser (there are plenty of native HTML5 parsers e.g. gumbo or html5ever but I don't remember them being able to generate lxml or bs trees). > 2. Don't forget to check the status code of r (r.status_code or less generally r.ok) Alternatively (depending on use case) `r.raise_for_status()`. I'm still annoyed that there's no way to ask requests to just check it outright. > Those with a background in coding might prefer the .cssselect method available in whatever object the parsed document results in. That's obviously a tad slower than find/findall/xpath, but it's oftentimes too convenient to pass upon. FWIW cssselect simply translates CSS selectors to XPath, and while I don't know for sure I'm guessing it has an expression cache, so it should not be noticeably slower than XPath (CSS selectors are not a hugely complex language anyway)
- aumerle 9y ago
  https://github.com/kovidgoyal/html5-parser https://github.com/kovidgoyal/html5-parser
  - masklinn 9y ago
    Nice. Seems to only do lxml tree building?
    - aumerle 9y ago
      http://html5-parser.readthedocs.io/en/latest/#html5_parser.parse http://html5-parser.readthedocs.io/en/latest/#html5_parser.p...
      - masklinn 9y ago
        Damn. Are you the author? If so, well done, that looks like a great package.
- dmn001 9y ago
  On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.
  - masklinn 9y ago
    LXML itself is, the problem is that its HTML parser (libxml's really) is an ad-hoc "HTML4" parser which means the tree it builds routinely diverges from a proper HTML5 tree as you'd find in e.g. your browser's developer tools and the way it fixes (or whether it fixes it at all) markup is completely ad-hoc and hard to predict.
- staticautomatic 9y ago
  Are you talking about etree.HTML() being garbage? And what are your thoughts on parsing it as xml (e.g. etree.fromstring(), etree.parse() )?
sonofgod 9y ago
Yep. requests.session fixes the vast majority of cookie / login / session / problems. Replaying the same headers is also a powerful technique, so long as they aren't constantly changing. ASPX is still horrid, but just about doable if you pull all the hidden form variables out of the HTML and put them back into the header verbatim. But you're probably best off going down a headless browser route at that point. If anyone's wanting someone with scraping experience (UK/remote), I'm currently available... (dave.mckee@gmail.com)

by drej 9y ago

jacobush 9y ago

I had reason to gather news articles and extract keywords and authors - can't remember why I didn't use BeatifulSoup, because that was my first choice. In the end I used lxml with its html5parser: https://github.com/vonj/scraping https://github.com/vonj/scraping Regarding legality - something frowned upon is putting load on servers. You may get blocked, rate limited or worse if you put too much strain on servers. Especially when you as I did, experimented with different query options to get the data I needed, and had to rerun scraping a number of times. A neat trick I found, was to configure a web proxy locally (squid, in my case) to aggressively cache EVERYTHING. This was, new runs only went out to the news sites for new queries I had never run before. Very helpful, it also speeded up the development to access files locally (cached in squid) instead of having to go out to the internet all the time.

jacobush 9y ago
In that github repo there is an example squid config for caching permanently.

masklinn 9y ago

> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building. html5lib fixes that (and can construct both lxml and bs trees, and both libraries have html5lib integration), however it's slow. I don't know that there is a native compatible parser (there are plenty of native HTML5 parsers e.g. gumbo or html5ever but I don't remember them being able to generate lxml or bs trees). > 2. Don't forget to check the status code of r (r.status_code or less generally r.ok) Alternatively (depending on use case) `r.raise_for_status()`. I'm still annoyed that there's no way to ask requests to just check it outright. > Those with a background in coding might prefer the .cssselect method available in whatever object the parsed document results in. That's obviously a tad slower than find/findall/xpath, but it's oftentimes too convenient to pass upon. FWIW cssselect simply translates CSS selectors to XPath, and while I don't know for sure I'm guessing it has an expression cache, so it should not be noticeably slower than XPath (CSS selectors are not a hugely complex language anyway)

aumerle 9y ago
https://github.com/kovidgoyal/html5-parser https://github.com/kovidgoyal/html5-parser
- masklinn 9y ago
  Nice. Seems to only do lxml tree building?
  - aumerle 9y ago
    http://html5-parser.readthedocs.io/en/latest/#html5_parser.parse http://html5-parser.readthedocs.io/en/latest/#html5_parser.p...
    - masklinn 9y ago
      Damn. Are you the author? If so, well done, that looks like a great package.
dmn001 9y ago
On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.
- masklinn 9y ago
  LXML itself is, the problem is that its HTML parser (libxml's really) is an ad-hoc "HTML4" parser which means the tree it builds routinely diverges from a proper HTML5 tree as you'd find in e.g. your browser's developer tools and the way it fixes (or whether it fixes it at all) markup is completely ad-hoc and hard to predict.
staticautomatic 9y ago
Are you talking about etree.HTML() being garbage? And what are your thoughts on parsing it as xml (e.g. etree.fromstring(), etree.parse() )?

sonofgod 9y ago

Yep. requests.session fixes the vast majority of cookie / login / session / problems. Replaying the same headers is also a powerful technique, so long as they aren't constantly changing. ASPX is still horrid, but just about doable if you pull all the hidden form variables out of the HTML and put them back into the header verbatim. But you're probably best off going down a headless browser route at that point. If anyone's wanting someone with scraping experience (UK/remote), I'm currently available... (dave.mckee@gmail.com)