11 ms·

Convert macros to functions in the Python C API

RcouF1uZ4gsC 5y ago
> _Py_NewReference() Actually just the name is undefined behavior per C and C++. The standards reserve all identifiers starting with underscore and followed by a capital letter to the implementation.
- zabzonk 5y ago
  Quite right. I have never understood this obsession that people have with underscores. Here's something I wrote about them some time ago: https://punchlet.wordpress.com/2009/12/01/letter-the-second/ https://punchlet.wordpress.com/2009/12/01/letter-the-second/
  - mayli 5y ago
    Agreed, the obsession pretty much true, but without really practical reason behind it.
  - kevin_thibedeau 5y ago
    They make up for C's lack of namespaces. If you're not writing a compiler or OS you shouldn't be profligate with them.
- david2ndaccount 5y ago
  That being UB is more of trivia than anything else. It’s UB because it allows freedom for compilers and standard library implementations to change their implementation without it being a “breaking” change, not because the compiler will produce nasal demons if you commit the crime of naming anything _Py_NewReference.
latenightcoding 5y ago
This is great, if you want to see what the other extreme looks like check the source code of the perl5 interpreter, it's all weird macros.
matzf 5y ago
This is not the point of the article, and maybe I'm just tired, but I'm confused; multiple paragraphs mention increased stack size with inlining: > When a C compiler decides to not inline, there is likely a good reason. For example, inlining would reuse a register which require to save/restore the register value on the stack and so increase the stack memory usage or be less efficient. > On the other side, the Py_NO_INLINE macro can be used to disable inlining. It is useful to reduce the stack memory usage. This seems completely backwards, what are they talking about?
- stefanos82 5y ago
  I guess `flatten` attribute could be used here to help the situation? flatten Generally, inlining into a function is limited. For a function marked with this attribute, every call inside this function is inlined, if possible. Functions declared with attribute noinline and similar are not inlined. Whether the function itself is considered for inlining depends on its size and the current inlining parameters.
- kevin_thibedeau 5y ago
  Reading between the lines, they're probably pointing out that inlining non-trivial functions increases register pressure which leads to larger stack frames as variables spill onto the stack. That's a bit of a strawman since calling the non-inlined function will need to build an entire new stack frame anyway. Which one is most beneficial depends on other circumstances like cache behavior and how often the code is called.
- rstuart4133 5y ago
  I think their reasoning appears earlier: > When a C compiler decides to not inline, there is likely a good reason. For example, inlining would reuse a register which require to save/restore the register value on the stack and so increase the stack memory usage or be less efficient. And ... that removes all doubt. They are wrong. If a calculation requires an extra register, doing a function call won't conjure it out of thin air. It has to spill the register too, and it will push the PC along with it. It's still possible a doing function call rather than inlining will speed up code on modern CPU's. The repeated code the inlining generates extra demands on the caches.
omnicognate 5y ago
My selfish desire is better support for programs that use the "C API" from some other language/runtime via an FFI (foreign function interface). In my case it's .NET and P/Invoke, where I'm facing challenges similar to those faced by Python.NET, but the same things apply to other FFIs, (although they are ameliorated if you assume a common, system-wide C compiler as is usually the case with linux, which is probably why this case seems rarely to be considered). Macros are zero use in these situations. There should be exported functions corresponding to these. This PEP sounds like a move in the right direction, but I don't see it mentioned that these macro-replacing inline functions will also be exported from the library so they're accessible with GetProcAddress (windows) or dlsym (linux). If they will be, it will make me very happy. There's an asssumption in cpython "C API" development that clients are only written in C/C++. Even the Stable ABI and its improvements (which have been a great boon) are only discussed in terms of the ABI not changing, never in terms of actually defining an ABI. It would be great to have an interface you could interact with via an FFI without having to make assumptions about how it was compiled, even if it involves stuff like calling a bunch of functions to discover how big a ssize_t is, etc.
seiferteric 5y ago
This is probably the right solution, but I recently began digging into the python C API and found these in-lined functions in the header files and am a bit annoyed by having actual code in header files. The actual justification is that since these are shared libraries commonly called functions like GC calls will be inlined code instead of function calls, which makes sense but I wish there was a better way of doing that without putting code in headers. On a side note, my use of the API is a bit odd in that I am trying to create a python->ruby C API compatibility layer, and the ability to just #undef macros and redefine them in my header was nice, but oh well.

> _Py_NewReference() Actually just the name is undefined behavior per C and C++. The standards reserve all identifiers starting with underscore and followed by a capital letter to the implementation.

zabzonk 5y ago
Quite right. I have never understood this obsession that people have with underscores. Here's something I wrote about them some time ago: https://punchlet.wordpress.com/2009/12/01/letter-the-second/ https://punchlet.wordpress.com/2009/12/01/letter-the-second/
- mayli 5y ago
  Agreed, the obsession pretty much true, but without really practical reason behind it.
- kevin_thibedeau 5y ago
  They make up for C's lack of namespaces. If you're not writing a compiler or OS you shouldn't be profligate with them.
david2ndaccount 5y ago
That being UB is more of trivia than anything else. It’s UB because it allows freedom for compilers and standard library implementations to change their implementation without it being a “breaking” change, not because the compiler will produce nasal demons if you commit the crime of naming anything _Py_NewReference.

latenightcoding 5y ago

This is great, if you want to see what the other extreme looks like check the source code of the perl5 interpreter, it's all weird macros.

matzf 5y ago

This is not the point of the article, and maybe I'm just tired, but I'm confused; multiple paragraphs mention increased stack size with inlining: > When a C compiler decides to not inline, there is likely a good reason. For example, inlining would reuse a register which require to save/restore the register value on the stack and so increase the stack memory usage or be less efficient. > On the other side, the Py_NO_INLINE macro can be used to disable inlining. It is useful to reduce the stack memory usage. This seems completely backwards, what are they talking about?

stefanos82 5y ago
I guess `flatten` attribute could be used here to help the situation? flatten Generally, inlining into a function is limited. For a function marked with this attribute, every call inside this function is inlined, if possible. Functions declared with attribute noinline and similar are not inlined. Whether the function itself is considered for inlining depends on its size and the current inlining parameters.
kevin_thibedeau 5y ago
Reading between the lines, they're probably pointing out that inlining non-trivial functions increases register pressure which leads to larger stack frames as variables spill onto the stack. That's a bit of a strawman since calling the non-inlined function will need to build an entire new stack frame anyway. Which one is most beneficial depends on other circumstances like cache behavior and how often the code is called.
rstuart4133 5y ago
I think their reasoning appears earlier: > When a C compiler decides to not inline, there is likely a good reason. For example, inlining would reuse a register which require to save/restore the register value on the stack and so increase the stack memory usage or be less efficient. And ... that removes all doubt. They are wrong. If a calculation requires an extra register, doing a function call won't conjure it out of thin air. It has to spill the register too, and it will push the PC along with it. It's still possible a doing function call rather than inlining will speed up code on modern CPU's. The repeated code the inlining generates extra demands on the caches.

omnicognate 5y ago

My selfish desire is better support for programs that use the "C API" from some other language/runtime via an FFI (foreign function interface). In my case it's .NET and P/Invoke, where I'm facing challenges similar to those faced by Python.NET, but the same things apply to other FFIs, (although they are ameliorated if you assume a common, system-wide C compiler as is usually the case with linux, which is probably why this case seems rarely to be considered). Macros are zero use in these situations. There should be exported functions corresponding to these. This PEP sounds like a move in the right direction, but I don't see it mentioned that these macro-replacing inline functions will also be exported from the library so they're accessible with GetProcAddress (windows) or dlsym (linux). If they will be, it will make me very happy. There's an asssumption in cpython "C API" development that clients are only written in C/C++. Even the Stable ABI and its improvements (which have been a great boon) are only discussed in terms of the ABI not changing, never in terms of actually defining an ABI. It would be great to have an interface you could interact with via an FFI without having to make assumptions about how it was compiled, even if it involves stuff like calling a bunch of functions to discover how big a ssize_t is, etc.

seiferteric 5y ago

This is probably the right solution, but I recently began digging into the python C API and found these in-lined functions in the header files and am a bit annoyed by having actual code in header files. The actual justification is that since these are shared libraries commonly called functions like GC calls will be inlined code instead of function calls, which makes sense but I wish there was a better way of doing that without putting code in headers. On a side note, my use of the API is a bit odd in that I am trying to create a python->ruby C API compatibility layer, and the ability to just #undef macros and redefine them in my header was nice, but oh well.