3 ms·
Not necessarily. Let's think of some operations a crawler would need to do: 1. Spawn some threads (Python uses native threading) 2. Connect to and load some UR
by jmtulloss 18y ago
Not necessarily. Let's think of some operations a crawler would need to do:
1. Spawn some threads (Python uses native threading)
2. Connect to and load some URLs (This isn't going to be slow anywhere.)
3. Run some regular expressions (Python's regexp engine is all C)
4. Write to the disk. (Python uses the standard system libraries for this)
It wouldn't be as fast, but it might be surprisingly close. There would be fewer lines of code to boot.
The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time. It sounds like the man only has 1 processor, so that wouldn't be the end of the world, but if he has more, you can just swap in the process module for the threading module. Of course, then you might have some RAM issues.
- silentbicycle 18y agoOr, briefly: Certain classes of Python programs can actually be faster than average C, because they consist of small amounts of Python code connecting libraries that are already written in very thoroughly tuned C.
- jrockway 18y agoWhy use threads when you can use an event-based IO system? (I think Twisted is the Python way of doing this.)
- jmtulloss 18y agoI was just going off of what he had already done. I agree that async is a better way to do this. Twisted is one way, but Python ships with asynchronous libraries http://docs.python.org/lib/module-asyncore.html http://docs.python.org/lib/module-asyncore.html.
- llimllib 18y ago> The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time. Not actually true; socket calls release the GIL when they're not active, as do many of python's C libraries. Especially the better-written standard ones.
- jmtulloss 18y agoTrue, although I would argue that my statement is still correct. You cannot have more than 1 thread running simultaneously. The GIL is not as big of a problem as most people see it, it only interferes with CPU bound, highly parallelizable tasks.