8 ms·

Python Internals: PyObject

169 points by kercker 10y ago

dfox 10y ago
Contrary to what the article says, PyVarObject is still PyObject at the same time and PyVarObjects have nothing to do with mutability (certainly, it's trivialy obvious that not all PyVarObjects are mutable as immutable tuples are PyVarObjects, while mutable lists are not) The thing that is handled by PyVarObject are types whose instances can differ in their size, with tuple and bytes objects being probably most common cases when it is used.
harrisi 10y ago
Although it's been posted here (only twice, though!) before, Philip Guo put up a 10-hour series of lectures on CPython internals[0]. I am nearly done with them, and have found them very useful and well-paced. Posting here because the author of this article seems to have stopped the series, and anyone interested in this stuff should definitely check out the Guo lectures. [0]: http://pgbovine.net/cpython-internals.htm http://pgbovine.net/cpython-internals.htm
- denfromufa 10y ago
  Also this blog post about modifying CPython internals is very interesting: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/ https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow...
- giis 10y ago
  +1 Guo series is excellent. I completed them twice & very confident about my python skills more than ever before :)
  - pgbovine 10y ago
    Thanks everyone! Recording lectures has been great since it doesn't take much extra work than not recording; just takes some time to render the final videos and upload. Ideally I'd have time to split the videos up into smaller chunks, but alas free time is diminishing nowadays.
    - harrisi 10y ago
      Would you have any interest in providing someone (I would volunteer) the videos so that they can package them in smaller, topic-specific pieces for easier consumption? I don't know anything about editing videos, but if your writings and lectures have taught me anything, it's that I'm capable of learning things and using that knowledge to do interesting and useful work. As an aside, you are one of my main sources of inspiration. I was a high school dropout with several terms of failing out of community college courses, but after reading your PhD Grind series/book, I'm now a (nearly) straight A student and am very determined to go as far with my education as I can. I would be honored to help you in any small way I can to express my gratitude! Thanks again for unknowingly improving my life.
    - giis 10y ago
      Thank you for your effort and sharing the knowledge. pythontutor.com is excellent project! While watching the series for second time - I started taking some notes/hints. Here its: http://giis.co.in/c_py_notes.txt http://giis.co.in/c_py_notes.txt Its more like hints. one need to watch video to understand this small note. (sorry for typos/mistakes) thanks again! EDIT: with youtube offline mode, I saved some video on smartphone & then while going for evening walk, I was listening to it, Got better in health & python skills at the same time :D
- bogomipz 10y ago
  Wow, I had none seen this before. What a fantastic resource. Thanks for sharing, also thanks to Phillip for making this available to everyone.
denfromufa 10y ago
Cython can be used to auto-generate optimized Python C-API code that gets source-mapped line by line to original Cython code in the form of HTML annotations. This stuff is very hard to write by hand.
denfromufa 10y ago
Here is blog post about python list implementation, one of 2 main data structures used internally: http://www.laurentluce.com/posts/python-list-implementation/ http://www.laurentluce.com/posts/python-list-implementation/
partycoder 10y ago
If you try to embed or extend Python in C you will be exposed to PyObject very quickly. The official Python documentation can walk you through that process (visit the embedding or extending parts). The PyObject structure mentioned in the article also comes with many functions to manage them programatically, cast them to other types, etc.
- p4wnc6 10y ago
  I'm curious if you know much about jobs doing this kind of thing. I've worked a ton with Python, and know a lot about using Cython to write extension modules, wrap C++ code, etc. I really want a job where I find performance bottlenecks in research or business analytics or financial analytics code, migrate parts to robust Cython implementations, and then build and maintain the broader Python libraries and APIs surrounding that. But what I'm unfortunately finding is that basically nobody uses this stuff anywhere in practice. I've interviewed at a ton of places doing everything in C++ entirely, and I just see their code and I think, my god, why would anyone choose this? You can write the critical stuff in C++ just as you are, and then so easily wrap it for Python, then write the other 99% of the code in super easy, dynamic Python, even with various kinds of validating systems, or using typing and type annotations to avoid certain kinds of bugs, and just everyone's life is easier. But they are just so entrenched in the old way of doing it that nobody is willing to try. I've been consistently amazed at how few jobs in the monthly Who Is Hiring thread are tagged even just with NumPy. I think I've seen maybe one job in 6 months that mentions Cython, and then I found two others outside of HN and the interviews with those places didn't work out. How to find a job doing this stuff is like my main vexing work problem at the moment. It's so useful, but it's sort of locked outside of some kind of energy barrier that quant teams seem unwilling to cross.
  - partycoder 10y ago
    Well, people have found success embedding Lua or JavaScript, since they're easier to sandbox than Python. Now, Python extensions can be a bit hard to get right. Especially since Python has a specific threading model and if you are trying to wrap an existing C++ code base that is multithreaded in Python... it can be challenging. Anyways, best of luck with your quest.

169 points by kercker 10y ago

dfox 10y ago

Contrary to what the article says, PyVarObject is still PyObject at the same time and PyVarObjects have nothing to do with mutability (certainly, it's trivialy obvious that not all PyVarObjects are mutable as immutable tuples are PyVarObjects, while mutable lists are not) The thing that is handled by PyVarObject are types whose instances can differ in their size, with tuple and bytes objects being probably most common cases when it is used.

harrisi 10y ago

Although it's been posted here (only twice, though!) before, Philip Guo put up a 10-hour series of lectures on CPython internals[0]. I am nearly done with them, and have found them very useful and well-paced. Posting here because the author of this article seems to have stopped the series, and anyone interested in this stuff should definitely check out the Guo lectures. [0]: http://pgbovine.net/cpython-internals.htm http://pgbovine.net/cpython-internals.htm

denfromufa 10y ago
Also this blog post about modifying CPython internals is very interesting: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/ https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow...
giis 10y ago
+1 Guo series is excellent. I completed them twice & very confident about my python skills more than ever before :)
- pgbovine 10y ago
  Thanks everyone! Recording lectures has been great since it doesn't take much extra work than not recording; just takes some time to render the final videos and upload. Ideally I'd have time to split the videos up into smaller chunks, but alas free time is diminishing nowadays.
  - harrisi 10y ago
    Would you have any interest in providing someone (I would volunteer) the videos so that they can package them in smaller, topic-specific pieces for easier consumption? I don't know anything about editing videos, but if your writings and lectures have taught me anything, it's that I'm capable of learning things and using that knowledge to do interesting and useful work. As an aside, you are one of my main sources of inspiration. I was a high school dropout with several terms of failing out of community college courses, but after reading your PhD Grind series/book, I'm now a (nearly) straight A student and am very determined to go as far with my education as I can. I would be honored to help you in any small way I can to express my gratitude! Thanks again for unknowingly improving my life.
  - giis 10y ago
    Thank you for your effort and sharing the knowledge. pythontutor.com is excellent project! While watching the series for second time - I started taking some notes/hints. Here its: http://giis.co.in/c_py_notes.txt http://giis.co.in/c_py_notes.txt Its more like hints. one need to watch video to understand this small note. (sorry for typos/mistakes) thanks again! EDIT: with youtube offline mode, I saved some video on smartphone & then while going for evening walk, I was listening to it, Got better in health & python skills at the same time :D
bogomipz 10y ago
Wow, I had none seen this before. What a fantastic resource. Thanks for sharing, also thanks to Phillip for making this available to everyone.

denfromufa 10y ago

Cython can be used to auto-generate optimized Python C-API code that gets source-mapped line by line to original Cython code in the form of HTML annotations. This stuff is very hard to write by hand.

denfromufa 10y ago

Here is blog post about python list implementation, one of 2 main data structures used internally: http://www.laurentluce.com/posts/python-list-implementation/ http://www.laurentluce.com/posts/python-list-implementation/

partycoder 10y ago

If you try to embed or extend Python in C you will be exposed to PyObject very quickly. The official Python documentation can walk you through that process (visit the embedding or extending parts). The PyObject structure mentioned in the article also comes with many functions to manage them programatically, cast them to other types, etc.

p4wnc6 10y ago
I'm curious if you know much about jobs doing this kind of thing. I've worked a ton with Python, and know a lot about using Cython to write extension modules, wrap C++ code, etc. I really want a job where I find performance bottlenecks in research or business analytics or financial analytics code, migrate parts to robust Cython implementations, and then build and maintain the broader Python libraries and APIs surrounding that. But what I'm unfortunately finding is that basically nobody uses this stuff anywhere in practice. I've interviewed at a ton of places doing everything in C++ entirely, and I just see their code and I think, my god, why would anyone choose this? You can write the critical stuff in C++ just as you are, and then so easily wrap it for Python, then write the other 99% of the code in super easy, dynamic Python, even with various kinds of validating systems, or using typing and type annotations to avoid certain kinds of bugs, and just everyone's life is easier. But they are just so entrenched in the old way of doing it that nobody is willing to try. I've been consistently amazed at how few jobs in the monthly Who Is Hiring thread are tagged even just with NumPy. I think I've seen maybe one job in 6 months that mentions Cython, and then I found two others outside of HN and the interviews with those places didn't work out. How to find a job doing this stuff is like my main vexing work problem at the moment. It's so useful, but it's sort of locked outside of some kind of energy barrier that quant teams seem unwilling to cross.
- partycoder 10y ago
  Well, people have found success embedding Lua or JavaScript, since they're easier to sandbox than Python. Now, Python extensions can be a bit hard to get right. Especially since Python has a specific threading model and if you are trying to wrap an existing C++ code base that is multithreaded in Python... it can be challenging. Anyways, best of luck with your quest.