3 ms·

Congrats on your big speedup; successful optimization like that is always a rush. I wonder what the result would be if you did everything you describe, but wro

Congrats on your big speedup; successful optimization like that is always a rush. I wonder what the result would be if you did everything you describe, but wrote the code that's now in C, in Python instead. I suspect the speed would be very similar. (I like C, for what it's worth.)

thomasmallen 18y ago
I wonder what the result would be if you ... wrote the code that's now in C, in Python instead. I suspect the speed would be very similar. Is that a joke?
- jmtulloss 18y ago
  Not necessarily. Let's think of some operations a crawler would need to do: 1. Spawn some threads (Python uses native threading) 2. Connect to and load some URLs (This isn't going to be slow anywhere.) 3. Run some regular expressions (Python's regexp engine is all C) 4. Write to the disk. (Python uses the standard system libraries for this) It wouldn't be as fast, but it might be surprisingly close. There would be fewer lines of code to boot. The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time. It sounds like the man only has 1 processor, so that wouldn't be the end of the world, but if he has more, you can just swap in the process module for the threading module. Of course, then you might have some RAM issues.
  - silentbicycle 18y ago
    Or, briefly: Certain classes of Python programs can actually be faster than average C, because they consist of small amounts of Python code connecting libraries that are already written in very thoroughly tuned C.
  - jrockway 18y ago
    Why use threads when you can use an event-based IO system? (I think Twisted is the Python way of doing this.)
    - jmtulloss 18y ago
      I was just going off of what he had already done. I agree that async is a better way to do this. Twisted is one way, but Python ships with asynchronous libraries http://docs.python.org/lib/module-asyncore.html http://docs.python.org/lib/module-asyncore.html.
  - llimllib 18y ago
    > The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time. Not actually true; socket calls release the GIL when they're not active, as do many of python's C libraries. Especially the better-written standard ones.
    - jmtulloss 18y ago
      True, although I would argue that my statement is still correct. You cannot have more than 1 thread running simultaneously. The GIL is not as big of a problem as most people see it, it only interferes with CPU bound, highly parallelizable tasks.
- ConradHex 18y ago
  The "..." above had some important bits in it. My point was, how much of the speedup came from switching to C, and how much came from all the other stuff he did (much of which could also be done in Python). Switching from a scripting language to C doesn't mean an automatic huge speed boost. It depends where your time is being spent.
  - thomasmallen 18y ago
    "everything [he] described" regarding code was porting code over to C.

thomasmallen 18y ago

I wonder what the result would be if you ... wrote the code that's now in C, in Python instead. I suspect the speed would be very similar. Is that a joke?

jmtulloss 18y ago
Not necessarily. Let's think of some operations a crawler would need to do: 1. Spawn some threads (Python uses native threading) 2. Connect to and load some URLs (This isn't going to be slow anywhere.) 3. Run some regular expressions (Python's regexp engine is all C) 4. Write to the disk. (Python uses the standard system libraries for this) It wouldn't be as fast, but it might be surprisingly close. There would be fewer lines of code to boot. The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time. It sounds like the man only has 1 processor, so that wouldn't be the end of the world, but if he has more, you can just swap in the process module for the threading module. Of course, then you might have some RAM issues.
- silentbicycle 18y ago
  Or, briefly: Certain classes of Python programs can actually be faster than average C, because they consist of small amounts of Python code connecting libraries that are already written in very thoroughly tuned C.
- jrockway 18y ago
  Why use threads when you can use an event-based IO system? (I think Twisted is the Python way of doing this.)
  - jmtulloss 18y ago
    I was just going off of what he had already done. I agree that async is a better way to do this. Twisted is one way, but Python ships with asynchronous libraries http://docs.python.org/lib/module-asyncore.html http://docs.python.org/lib/module-asyncore.html.
- llimllib 18y ago
  > The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time. Not actually true; socket calls release the GIL when they're not active, as do many of python's C libraries. Especially the better-written standard ones.
  - jmtulloss 18y ago
    True, although I would argue that my statement is still correct. You cannot have more than 1 thread running simultaneously. The GIL is not as big of a problem as most people see it, it only interferes with CPU bound, highly parallelizable tasks.
ConradHex 18y ago
The "..." above had some important bits in it. My point was, how much of the speedup came from switching to C, and how much came from all the other stuff he did (much of which could also be done in Python). Switching from a scripting language to C doesn't mean an automatic huge speed boost. It depends where your time is being spent.
- thomasmallen 18y ago
  "everything [he] described" regarding code was porting code over to C.