4 ms·

I wrote this for a friend who is just getting started out in python/web stuff. It might be a little n00bly for lots of people here, but it could be interesting

by Mizza 15y ago

I wrote this for a friend who is just getting started out in python/web stuff. It might be a little n00bly for lots of people here, but it could be interesting for people who aren't pythonic who are thinking about making the switch!

sixtofour 15y ago
Nicely done, very clear and concise. A question: is lxml what we would use today instead of Beautiful Soup?
- waterside81 15y ago
  lxml is definitely faster, but I've found BSoup to be more forgiving with poorly formatted DOMs
  - kenneth_reitz 15y ago
    BeautifulSoup is poorly maintained — you have to be very specific with which version you're using. Note: Lxml has a number of repair modes that allows it to parse virtually anything. Cpu cycles and memory go up quite a bit when they're activated, but it's still better than BeautifulSoup.
  - ericflo 15y ago
    Thankfully lxml has a slower-but-more-forgiving mode that you can use when interacting with poorly formatted HTML, which takes advantage of BeautifulSoup http://lxml.de/elementsoup.html http://lxml.de/elementsoup.html
  - sudont 15y ago
    I’ve found the exact opposite. BSoup will choke on invalid tags in the DOM, such as: <div id=“content”><content>…</content></div> If I try to return the innerHTML of #content, I get '<div id=“content”><content>’ as a string, nothing else. While I know that’s inexcusable markup, it’s nothing I have control over. lxml (if it builds on the target system) has been much better for my scripts.
- Tobu 15y ago
  lxml's default parsers are good for xml, atom and xhtml; for html5 and html tag soup, lxml.html.html5parser (which depends on html5lib) is the way to go. For feed tag soup, feedparser still uses BeautifulSoup internally.
michaelchisari 15y ago
Thank you for using a large font. Makes it much more readable.
d0mine 15y ago
Great that you promote requests, lxml, json (just json, not simplejson since Python 2.6), django to use by a beginner in Python. Pages from http://wiki.python.org/moin/WebProgramming http://wiki.python.org/moin/WebProgramming might provide too much choice for a beginner. You might add a tiny example of a microframework such as Bottle, Flask or an example from Pyramid written in the microframework-ish style. Recommending `sudo` with `pip install` to a novice is not a good idea. System package manager or `pip install --user` could be used as alternatives. `virtualenv` might be out of topic for a short tutorial.
- pjscott 15y ago
  About JSON: Python 2.6 added a "json" module to the standard library that was forked from simplejson, but in that time, simplejson has improved; the interface is exactly the same as the built-in json module, but simplejson tends to be quite a bit faster. On my informal benchmark of JSON parsing speed, simplejson was 27x faster than json. The idiom I've been using is: try: import simplejson as json except ImportError: import json
  - d0mine 15y ago
    Documentation for simplejson mentions that it should be faster. Though it contains reference to Python 2.5, 2.6 so It is unclear whether the claim is outdated: It is the externally maintained version of the json library contained in Python 2.6, but maintains compatibility with Python 2.5 and (currently) has significant performance advantages, even without using the optional C extension for speedups. </quote> http://simplejson.readthedocs.org/en/latest/index.html http://simplejson.readthedocs.org/en/latest/index.html I see there are performance patches to stdlib's json since 2.6: http://hg.python.org/cpython/search/?rev=_json.c&revcount=80 http://hg.python.org/cpython/search/?rev=_json.c&revcoun... An order of magnitude difference might be an issue with your python installation. On my machine version from 2.6 (1.9) is 15 times slower than from 2.7 (2.0.9). Though simplejson (2.2.1) is still faster on both 40 and 2 times correspondingly for .loads() timeline.js from the example as measured by: % timeit json.loads(timeline_text) it seems that C speedups available on both versions: python -v -c'import json' 2>&1 | grep _json.so python -c'import json.decoder as d; print d.scanstring' Is your python version old? Otherwise it is worth to open issues at http://bugs.python.org http://bugs.python.org to merge speed improvements and at https://github.com/simplejson/simplejson/issues https://github.com/simplejson/simplejson/issues to mention in the docs and on http://pypi.python.org/pypi/simplejson http://pypi.python.org/pypi/simplejson more clearly that Python 2.7+ stdlib's version is also slower.
carlio 15y ago
I'm a big fan of pyquery[1] - it uses lxml under the hood, but is much nicer to interact with, especially if you're familiar with jQuery already. [1] http://packages.python.org/pyquery/api.html http://packages.python.org/pyquery/api.html

by Mizza 15y ago

sixtofour 15y ago

Nicely done, very clear and concise. A question: is lxml what we would use today instead of Beautiful Soup?

waterside81 15y ago
lxml is definitely faster, but I've found BSoup to be more forgiving with poorly formatted DOMs
- kenneth_reitz 15y ago
  BeautifulSoup is poorly maintained — you have to be very specific with which version you're using. Note: Lxml has a number of repair modes that allows it to parse virtually anything. Cpu cycles and memory go up quite a bit when they're activated, but it's still better than BeautifulSoup.
- ericflo 15y ago
  Thankfully lxml has a slower-but-more-forgiving mode that you can use when interacting with poorly formatted HTML, which takes advantage of BeautifulSoup http://lxml.de/elementsoup.html http://lxml.de/elementsoup.html
- sudont 15y ago
  I’ve found the exact opposite. BSoup will choke on invalid tags in the DOM, such as: <div id=“content”><content>…</content></div> If I try to return the innerHTML of #content, I get '<div id=“content”><content>’ as a string, nothing else. While I know that’s inexcusable markup, it’s nothing I have control over. lxml (if it builds on the target system) has been much better for my scripts.
Tobu 15y ago
lxml's default parsers are good for xml, atom and xhtml; for html5 and html tag soup, lxml.html.html5parser (which depends on html5lib) is the way to go. For feed tag soup, feedparser still uses BeautifulSoup internally.

michaelchisari 15y ago

Thank you for using a large font. Makes it much more readable.

d0mine 15y ago

Great that you promote requests, lxml, json (just json, not simplejson since Python 2.6), django to use by a beginner in Python. Pages from http://wiki.python.org/moin/WebProgramming http://wiki.python.org/moin/WebProgramming might provide too much choice for a beginner. You might add a tiny example of a microframework such as Bottle, Flask or an example from Pyramid written in the microframework-ish style. Recommending `sudo` with `pip install` to a novice is not a good idea. System package manager or `pip install --user` could be used as alternatives. `virtualenv` might be out of topic for a short tutorial.

pjscott 15y ago
About JSON: Python 2.6 added a "json" module to the standard library that was forked from simplejson, but in that time, simplejson has improved; the interface is exactly the same as the built-in json module, but simplejson tends to be quite a bit faster. On my informal benchmark of JSON parsing speed, simplejson was 27x faster than json. The idiom I've been using is: try: import simplejson as json except ImportError: import json
- d0mine 15y ago
  Documentation for simplejson mentions that it should be faster. Though it contains reference to Python 2.5, 2.6 so It is unclear whether the claim is outdated: It is the externally maintained version of the json library contained in Python 2.6, but maintains compatibility with Python 2.5 and (currently) has significant performance advantages, even without using the optional C extension for speedups. </quote> http://simplejson.readthedocs.org/en/latest/index.html http://simplejson.readthedocs.org/en/latest/index.html I see there are performance patches to stdlib's json since 2.6: http://hg.python.org/cpython/search/?rev=_json.c&revcount=80 http://hg.python.org/cpython/search/?rev=_json.c&revcoun... An order of magnitude difference might be an issue with your python installation. On my machine version from 2.6 (1.9) is 15 times slower than from 2.7 (2.0.9). Though simplejson (2.2.1) is still faster on both 40 and 2 times correspondingly for .loads() timeline.js from the example as measured by: % timeit json.loads(timeline_text) it seems that C speedups available on both versions: python -v -c'import json' 2>&1 | grep _json.so python -c'import json.decoder as d; print d.scanstring' Is your python version old? Otherwise it is worth to open issues at http://bugs.python.org http://bugs.python.org to merge speed improvements and at https://github.com/simplejson/simplejson/issues https://github.com/simplejson/simplejson/issues to mention in the docs and on http://pypi.python.org/pypi/simplejson http://pypi.python.org/pypi/simplejson more clearly that Python 2.7+ stdlib's version is also slower.

carlio 15y ago

I'm a big fan of pyquery[1] - it uses lxml under the hood, but is much nicer to interact with, especially if you're familiar with jQuery already. [1] http://packages.python.org/pyquery/api.html http://packages.python.org/pyquery/api.html