7 ms·

Is writing extensions a lost art? I read a few blog posts about speeding up Python and Ruby with Rust extensions. This should enable rewriting only the slow par

• rgperkins 9y ago
  It is not a lost art, but it is a thankless job. If an extension just works, people take it, the author gets two or three positive remarks and is ignored from then on because the extension now has utility status. Much better to write an application in Python using 30 slow Python module dependencies, so there are always some fires to fight and the application always gets publicity.
• jerf 9y ago
  Extensions aren't a total solution, though, which people often sell them as. You have an impedance mismatch between Python and C code, because Python has all of its objects packed in a way that is very strange to C, so you end up essentially deserializing all objects into C, then back out into Python, in a very expensive and allocation-heavy (on both sides) conversion. If you can set up your computation in Python and run it in C, as with a lot of NumPy code, you can have your entire program basically run at C speeds. But if you have a complicated algorithm in Python, perhaps implementing business logic, you can very easily see a slowdown if you try to move bits of that logic into C piecemeal, as you end up paying more in cross-language serialization and overhead than you can win back. In addition, writing extensions can be hard. First you've got a maze of choices nowadays, and while many of them are quite good at what they do, it can be difficult to figure out whether you're going to do something they aren't good at and have to switch options later, and it's really hard to figure out how to even analyze what they are and are not good at when you're not already familiar with the space. Then, if you do end up having to delve into the raw C, it's very tedious code, very tricky code to deal with the PyObjs, and code that can segfault the interpreter instantly if you don't get it right, which is not what you want to read about your multi-hour processing code. And for a greenfield or very young product, this maze of options is competing against other ecosystems where you can simply implement your code and get it to run 20-50x faster while writing code that is easier to write than a Python extension. They are a solution to some problems. I don't deny this. NumPy is an existence proof of that statement. But I'd write this post because I'd say "if it's slow, just write the slow bit in C!" has been oversold in the dynamic language community now for at least the 15 years I've been paying attention, and it still seems to be going strong. In 2003, it may still have been a good choice; in 2018, my recommendation to anybody writing the sort of code where this matters is to pick up one of the several languages that are simply faster to start with, and are much more convenient (even when statically typed) to work with than the competition was in 2003. And I also want to say that Python is still good for many things; I still whip it out every couple of months for something; it's still on my very short list of best languages. But the ground on which it is the best choice is definitely getting squeezed by a lot of very good competition and the changing nature of computer hardware, and a wise engineer pays attention to that and adjusts as needed.
  • bjourne 9y ago
    I agree with that. But for the OPs specific problem, a function returning the genetic diversity as a float between two samples, rewriting it in C or Pyrex would have been an ideal solution.
  • fafhrd91 9y ago
    Python extension doesn’t mean C. Rust works perfectly for extensions, it covers a lot of low level c-api integration and it is fast. You can write whole application in rust and use python as a glue language https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3 Pyo3 library gives you ability to work both diractions. Call python code from rust and call rust code from python.
    • jsmthrowaway 9y ago
      Cython interops just fine with Rust, too.
    • jerf 9y ago
      That sounds like one of the "maze of choices" I mentioned, no? And if you're "writing the whole application in Rust and using Python as a glue language", you don't have the problem that this entire discussion is about, which is when you have Python code that is slow. Python as an extension language is a completely different world. Performance problems there are a much less big deal, because you've already got the option to simply use the fast language with only modestly more complexity, if indeed even that given how nice Rust is once you get used to it. It's when your whole app is in Python that these issues emerge, and "Just write extensions" is an option far less often than portrayed.
      • fafhrd91 9y ago
        My point is, you are not limited with using extensions only for optimizing hot loops, in rust you can write application logic as well. I doubt you should do this in C for example
    • deathanatos 9y ago
• fireflash38 9y ago
  Not even extensions are required -- you could have C libraries that you hook into rather easily. You don't have to learn to navigate the PyObj or the Python extension docs, just know how to write & expose C functions as a shared library.
• aldanor 9y ago
  Most definitely not. You can write C++ extensions inside Jupyter notebook these days ([1]) -- thanks to libraries like pybind11 [2]. [1] https://github.com/aldanor/ipybind https://github.com/aldanor/ipybind [2] https://github.com/pybind/pybind11 https://github.com/pybind/pybind11