4 ms·

Being implemented in C does not make things fast. Integers are implemented in C in python, but they are really slow, because of the boxing/unboxing thing (you n

by cdavid 16y ago

Being implemented in C does not make things fast. Integers are implemented in C in python, but they are really slow, because of the boxing/unboxing thing (you need to chase one pointer pointer to get the actual C int from the Python object PyObject_Int, and that cost alone is high). Numpy is much faster for this kind of thing because it does not have this cost, and run the core loops + function calls in pure C, without going through the python runtime at all.

jrockway 16y ago
Yes, exactly. As you context switch between the high-level and low-level language, you sacrifice performance. If you want to write a fast extension, you don't do this; you let the high-level language call your pointer an object, and when you invoke a function, you invoke it on that pointer. This way, you don't marshal back and forth (except perhaps incidental parameters and results or exceptions).
- cdavid 16y ago
  What I am saying is that the whole concept of doing the heavy logic in the high level language and do the work in the C implementation cannot work when your logic depends on handling many objects. For example, I wrote a few years ago a small audio app to track frequencies lines in audio. Each track was a list of positions (integers), generally a few tens items max, and I needed to handle millions of such lists. Now, just optimizing the track object to be in C is not that efficient because I needed to access the content of each list in the high level logic. Basically, If your logic needs to be able to handle those millions small objects, you end up writing almost everything in the compiled language. Abstracting this becomes very difficult.
  - jrockway 16y ago
    I see. Incidentally, Haskell has a nice solution to this problem: http://hackage.haskell.org/packages/archive/vector/0.7/doc/html/Data-Vector-Storable.html http://hackage.haskell.org/packages/archive/vector/0.7/doc/h...
    - cdavid 16y ago
      Oh, python has a nice solution in this problem space - NumPy. But again, this only works in some cases. Unfortunately, the model fails performance-wise once you use non native type - think array of arbitrary precision floats. Haskell being much more powerful and expressive, I would expect it to be able to express those concepts in a more elegant manner, but I could be wrong (I know next to nothing about haskell).

by cdavid 16y ago

jrockway 16y ago

Yes, exactly. As you context switch between the high-level and low-level language, you sacrifice performance. If you want to write a fast extension, you don't do this; you let the high-level language call your pointer an object, and when you invoke a function, you invoke it on that pointer. This way, you don't marshal back and forth (except perhaps incidental parameters and results or exceptions).

cdavid 16y ago
What I am saying is that the whole concept of doing the heavy logic in the high level language and do the work in the C implementation cannot work when your logic depends on handling many objects. For example, I wrote a few years ago a small audio app to track frequencies lines in audio. Each track was a list of positions (integers), generally a few tens items max, and I needed to handle millions of such lists. Now, just optimizing the track object to be in C is not that efficient because I needed to access the content of each list in the high level logic. Basically, If your logic needs to be able to handle those millions small objects, you end up writing almost everything in the compiled language. Abstracting this becomes very difficult.
- jrockway 16y ago
  I see. Incidentally, Haskell has a nice solution to this problem: http://hackage.haskell.org/packages/archive/vector/0.7/doc/html/Data-Vector-Storable.html http://hackage.haskell.org/packages/archive/vector/0.7/doc/h...
  - cdavid 16y ago
    Oh, python has a nice solution in this problem space - NumPy. But again, this only works in some cases. Unfortunately, the model fails performance-wise once you use non native type - think array of arbitrary precision floats. Haskell being much more powerful and expressive, I would expect it to be able to express those concepts in a more elegant manner, but I could be wrong (I know next to nothing about haskell).