23 ms·
Want to call C from Python? Use D
- michelpp 7y agoInteresting article, but I wouldn't say introducing a 3rd language is the easiest way to call C code from Python, that distinction goes to the CFFI library by Armin Rigo et al. I've used it to wrap several C libraries and it's amazingly powerful. Bonus it works with CPython and PyPy. https://cffi.readthedocs.io/en/latest/ https://cffi.readthedocs.io/en/latest/
- oefrha 7y agoDon’t forget the builtin options. First there’s PSL ctypes,[1] which is supposedly inferior to cffi to some extent, but hey it comes with your CPython installation, no third-party dep required. Or you can write a C/C++ extension module.[2] A bit more heavy-handed and there’s a learning curve, but once you’ve done it once or twice it’s really not hard, and it affords you great flexibility. [1] https://docs.python.org/3/library/ctypes.html https://docs.python.org/3/library/ctypes.html [2] https://docs.python.org/3/extending/extending.html https://docs.python.org/3/extending/extending.html
- atilaneves 7y agoThe solution in the blog post is an extension module. Written for you, by the compiler. Instead of a learning curve and "it's really not hard", it's "no learning curve" and "I struggle to find how this could be easier".
- oefrha 7y agoI would take a tested and true solution sanctioned by CPython core devs any day over a novelty third party library that gets another language toolchain involved. Also, the solution seems to only cover what’s possible with ctypes already. If don’t need the flexibility there’s little reason to hand roll an extension module. (Not saying ctypes doesn’t have its problems; e.g. when you pass in a wrong number of arguments instead of throwing a TypeError it segfaults.)
- loeg 7y agoctypes does not parse C headers and doesn't know about enums or function parameter types — only that they have a symbol in the .dynsym section of a .so. Maybe you're thinking of cffi? Cffi does basically what this blog post covers, except without D. It requires a C compiler at some stage in the process, but you can pregenerate during 'build' phase and ship a header snapshot to avoid requiring the C compiler at runtime.
- oefrha 7y agoI didn’t say ctypes takes the same approach — I even mentioned ctypes’ shortcomings. I only said ctypes makes what this does possible already.
- atilaneves 7y agoNo, ctypes does not. It can't handle macros, and it requires users to define the structures in Python themselves. It's miles away from what's in the blog post.
- SQueeeeeL 7y agoMy problem with these is that in my use case, it completely broke on Windows...
- oefrha 7y agoI mentioned two completely different solutions, it’s not clear what “these” is referring to (both?). Also, both are clearly supported on Windows and used by tons of successful libraries and applications, so without details about your use case it’s not a very useful comment.
- yellowapple 7y agoI'm curious about what ctypes' supposed inferiorities are. Last time I used it was to implement native printing in Windows (long story short: needed to print to a ZPL printer, and Qt doesn't seem to offer any ability to just send raw data to a printer, so I implemented it myself with the Winspool API), and it was a breeze (comparatively speaking, i.e. as much of a breeze as Windows API programming can be). If ctypes can handle the Windows APIs, then I'm convinced it can handle just about anything.
- loeg 7y agoctypes allows you to trivially mistype C objects, send the wrong types and number of arguments to C functions, etc. Cffi just imports C headers and gives you more or less what TFA illustrates (without D).
- yellowapple 7y ago> ctypes allows you to trivially mistype C objects I mean, so does C. Pretty sure that's a "feature". ;) But yeah, fair point; it's definitely nice when an FFI wrapper is able to actually read the header files and figure out function/type signatures.
- 0xff00ffee 7y agoI don't understand how the OP could write this article and not talk about FFI? What a glaring oversight. (FWIW: The FFI module works extremely well in NodeJS as well.)
- atilaneves 7y agoAfter a week, I've finally had time to look at cffi. It's not even remotely as easy. I tried using it just now to call nanomsg and gave up immediately. I can't just give it the headers, I have to pass it a string which is valid C code. I can't read the headers (separately!) and pass that in either because it can't handle preprocessor directives. It's a mess. In the article, I do this: ``` #include "nanomsg/nn.h" #include "nanomsg/pipeline.h" ``` That's it. I don't think the approaches can be compared.
- fizixer 7y agoWow, I guess I should just stop everything and start learning D. I was dying to find a programming language that will let me interface two other programming languages.
- davidguda 7y agoYou'r being sarcastic but having convenient interop between languages is not a small thing.
- fizixer 7y agoOn second thought, I'll wait for two new programming languages to appear that make it even easier to do the interface, one making the Python-to-D interface easier, and the other making the D-to-C interface easier. Maybe those two languages already exist, but someone hasn't posted the solution on HN yet. Using three new awesome programming languages to seamlessly interface Python and C would solve an enormous number of interop problems.
- mkborregaard 7y agoIt's funny because it is true
- hatsunearu 7y agoOr you know, learn any of the native options that let you call C things relatively painlessly.
- atilaneves 7y agoI describe how easy it is to call C code from Python by using D as the glue layer. AMA.
- cannam 7y agoSo behind the scenes is a utility that scans the #included headers, runs the C preprocessor on them, then uses Clang code to pick out the declarations, wraps them for calling from D, and redefines any C macro values as D constants? Do I understand that correctly? (The other half of the puzzle being a tool called autowrap that wraps the resulting D for consumption by Python.) I'm not sure whether I personally would ever use this, but it's a very interesting exploration of the subject.
- atilaneves 7y ago> Do I understand that correctly? Yep! > (The other half of the puzzle being a tool called autowrap that wraps the resulting D for consumption by Python.) Correct. > I'm not sure whether I personally would ever use this, but it's a very interesting exploration of the subject. Python was an example. Want to call it from C# instead? Same code. Excel? Same code.
- jcelerier 7y agocould you explain the weird fascination of the D community with Excel ? there are so many "D" things for Excel and it seems really weird as an outsider - I don't remember the last time I've seen an actual Excel in the wild and not libreoffice or openoffice
- skocznymroczny 7y ago"Envious of C++’s and Objective C’s credible claim to be the only languages that can seamlessly interoperate with C (due to header inclusion and compatible syntax)" Wouldn't Nim and Zig also fit the bill?
- atilaneves 7y agoThe last time I looked at the Nim documentation, one had to manually declare C and C++ functions. I don't see how that's different from pretty much any other language that can call C (i.e. pretty much all of them). Then there's preprocessor macros. I don't know how Zig does it, I'd have to look. I expand on why dpp is the way it is in my DConf 2019 talk: https://www.youtube.com/watch?v=79COPHF3TnE&t=8s https://www.youtube.com/watch?v=79COPHF3TnE&t=8s
- nimmer 7y agoNim has an automated generator for C/C++ wrappers or translating from C. Also there are very easy ways to create Python modules in Nim: https://robert-mcdermott.gitlab.io/posts/speeding-up-python-with-nim/ https://robert-mcdermott.gitlab.io/posts/speeding-up-python-... https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy
- atilaneves 7y agoThanks for the links. That's nice, but not the same - users have to intend to write a Python extension in Nim. With autowrap, you can call unmodified D code (and as shown in the blog, C code as well with dpp). That means code that was never meant for consumption from another language can be made available.
- nimmer 7y agoYou don't need to modify the code, just add a wrapper. Nothing prevents generating a wrapper automatically but 99% of the time that is not very useful. The large majority of Python extension contain Python-specific code to access or return Python objects or act in a pythonic way.
- jaked89 7y agoD & Nim (especially Nim) are very appealing from a language perspective, but the lack an IDE undermines the advantages they bring. I personally won't consider using a language for a meaningful project unless it has a decent debugger, and reasonable refactoring; ie. at least find references & rename. Beef is building on an IDE, which is a refreshing take, but it's still too early.
- atilaneves 7y agoI'm not an IDE user, but I know that certain D programmers use Visual Studio (with VisualD) and seem happy. Did you try that option?
- EvenThisAcronym 7y agoVisual Studio with Visual D is a top notch D experience.
- giancarlostoro 7y agoD works fantastic with VS Code. See here: https://wiki.dlang.org/Editors https://wiki.dlang.org/Editors Also had really good luck doing D with Spacemacs, but Spacemacs is a hit or miss to setup for me. Other times I just use Sublime Text for the syntax highlighting.
- kbd 7y agoNim also works very well in vscode. Inline compiler errors as you edit, and so on.
- giancarlostoro 7y agoI can't remember what I used for Nim, I think I either used Nano or vim just because I've only tested it once or twice. I havent used it heavily yet, though I do have an interest in Nim, since I do enjoy Python programming.
- vmchale 7y ago
- jononor 7y agoI often use C++ and pybind11 to do this. pybind11 takes an approach very similar to boost::python (template magic). It works rather well. Though most of the C code I write can compile as C++ also. Why write in C then? It's still the lingua franca in microcontroller land... Just want to reuse the code on PC and Embedded Linux, and Python makes high-level testing much nicer.
- atilaneves 7y agoBoth pybind11 and boost::python involve writing code. This solution doesn't, save for listing the names of the files to be made available to Python (or C#, or Excel, or...).
- oefrha 7y agoNeither ctypes nor cffi involves writing glue code. In fact you don’t even need another source file, let alone a whole other language toolchain (wait, you actually need two other source files plus your third-party dpp library).
- bachmeier 7y agoI'm not sure I understand your comment. Someone said they use pybind11, someone responded to that by saying pybind11 requires writing code, and you responded by saying ctypes and cffi don't. How is that relevant to a discussion of pybind11?
- oefrha 7y agoOP presents their solution as an “easy” solution that doesn’t require glue code. Except there are already established native solutions that actually don’t require glue code.
- loeg 7y agoIn what sense does ctypes not require writing glue code?
- jakogut 7y agoI'm currently working on a project using Cython to wrap and expose a large and complex C API (CEF) to Python. One of the most time-consuming, tedious, and error prone tasks is translating functions and structures to Cython syntax, including managing the GIL. Also, keeping the translated API up to date with new releases. I'm curious to see how this D approach compares, both in workload required to expose and use an API, and the flexibility of the end result. There are a lot of quirks and special syntaxes in Cython to allow for not just interop between C and Python, but also writing C in Python-like syntax mixed with actual Python. This is an area I assume this approach would be lacking. Regardless, as someone who loves mixing C and Python, this is an encouraging reason to look more at D.
- jessermeyer 7y agoIs there a reason you chose not to build / refactor the project as a library to call from Python?
- jakogut 7y agoCan you clarify your question? I'm using Cython to write extension modules that directly interface with CEF's C API. CEF is about 40k lines of C++ that programmatically controls Chromium, which itself is written in C and C++.
- ratboy666 7y agoEmail me at fred(underscore)weigel(at)hotmail(dot)com I interfaced CEF to Java in 2000 lines of C++ Simple, easy API. I run CEF in a separate process, and use the Window manager (Linux and Windows) to cause the Chrome windows to overlay the application windows. Standard pipes are used between the controlling process and CEF. Email me if you are interested. As a ps -- I can use CEF from shell scripts as well. Anything with a C FFI works. FredW
- jessermeyer 7y agoI see. Thanks, that does change my understanding of your situation. My revised question with your clarifications in mind is, is there a reason you're not calling CEF's C api directly from Python? Are your extension modules doing some heavy lifting themselves, or are you using Cython just to handle the glue between the language interface?
- epicgiga 7y agoHow many languages do we need again? The amount of time we spend playing around with special case languages and frameworks. Would be nice if machine languages followed the lead of natural languages, with a single one slowly pushing the rest to the sidelines.
- abrax3141 7y agoIndeed lisp is slowly (very very slowly) doing this. Witness how it seems to get reinvented or embedded in every other language.
- mratsim 7y agoThere were some studies about the impact of the loss of some natural languages. Some represent a complete different ways of handling the world that are worth keeping. And there is no one size fits all.
- macintux 7y agoWhat a terrible world that would be. For a while, 20 years ago, it looked like Java was going to be the one language to rule them all; dodged that bullet. Kitchen sink languages are my least favorite. Give me something with a tight focus on a problem space.
- terminaljunkid 7y ago> with a single one slowly pushing the rest to the sidelines. That didn't quite happen. > How many languages do we need again? Till you eliminate all tradeoffs in a single language, there would be new ones.
- abrax3141 7y agoOkay, but what if I want to call python libraries from C (or any other language)? What we need is some sort of .net-like global interop in the Unix world.
- 2fast4you 7y agoIf I want to make an application composed of C, C#, and Visual Basic, there nothing stopping me in Windows. You try to pull that shit in Unix though, and you get a whole mess of problems. This is why Unix is clearly just so inferior. /s
- dralley 7y agoYou can't really do that without including the entire Python runtime in your application, and cpython is (so I've heard) absolutely horrid at being embedded in things.
- abrax3141 7y agoWhy can’t we create a --server mode standard where every language can be started up in that mode and serves a standard API with standard linker types (where possible)?
- srean 7y agoExtending is ok, embedding is indeed quite horrid. Tcl got that right. Yu want to embed multiple interpreters in the same address space with no sharing -- no problem. The mantle has been passed to Lua I think, but Guile is pretty good and not as minimalistic as Lua (that's bit of an understatement though).
- vmchale 7y agoLinker types? I think I recall Amal Ahmed gave a talk on them. http://www.ccs.neu.edu/home/amal/papers/linking-types.pdf http://www.ccs.neu.edu/home/amal/papers/linking-types.pdf
- bepvte 7y agoYou might be interested in GraalVM, which can do this between certain languages implemented on top of it.
- andrewprock 7y agoIs swig no longer a thing in the 2020s? I've always done this using the bundled Python/C integration path, or when there was too much stuff, use swig. http://swig.org/ http://swig.org/
- atilaneves 7y agoI used swig once. Never again.
- andrewprock 7y agoI'm curious, what was your negative experience?
- atilaneves 7y agoLiterally all of it.
- dekhn 7y agoIt's really crazy how many ways there are to implement FFI in Python. I started in the "old days" (Makefile, a bunch of CPython C code to implement argument parsing, etc), switched to swig (experienced both the good and bad sides), hoped that GCC-XML would make it easier to convert complicated C++ classes to Python interfaces (big mistake), then hoped that LLVM 'would solve the problem', saw Google release CLIF (still can't compile it from scratch on a modern linux system), and the large number of "late-binding" approaches. Of all of them, I was most skeptical of Cython but have slowly come to appreciate its sublime cleverness.
- WalterBright 7y ago> I wrote a little project called dpp because I’m lazy I can confirm that Atila is a very lazy man. He's always figuring out ways to have his computer do the work for him. He's so successful at avoiding work, people keep giving him more work to do, making him a very hard working lazy man.
- canada_dry 7y ago> always figuring out ways to have his computer do the work for him One of my earliest programming mentors mused that the reason I might well become a decent programmer was because I was both extremely stubborn and horrifically lazy. He'd observed that I would re-write code until it was the most efficient possible to automate a needed task that could save me time/effort in the future.
- ugexe 7y agoIt’s a common observation, and indeed known as one of the Three Virtues of a great programmer according to Larry Wall http://threevirtues.com http://threevirtues.com
- MaxBarraclough 7y agoThe developer of that site could have exhibited more Laziness and Hubris. Not sure I see a need for JavaScript there. Also, the virtue of Impatience seems to be at odds with the principle of YAGNI.
- taneq 7y agoImpatience is about writing code that is responsive, potentially by using some form of prediction to pre-load things. It's not about a programmer anticipating potential new features and implementing those.
- FeepingCreature 7y agoI think arrogance should replace impatience, which has redundancy with laziness. (Or possibly hubris, which is the weakest.) Arrogance makes you start projects that you are not skilled enough for, and grow in the doing.
- marmaduke 7y agohttps://github.com/davidjamesca/ctypesgen/wiki https://github.com/davidjamesca/ctypesgen/wiki does this without needing a compiler, by parsing the header files and generating the ctypes wrappers
- atilaneves 7y agoThere's no way to parse headers without a compiler.
- justlexi93 7y agoScripting languages by nature will always be slower than C. There's a tradeoff between program speed and development speed
- fithisux 7y agoHopefully the dream to compile a D library to another language (something like swigd) it will be interesting
- atilaneves 7y agoThat already works with Python, Excel and C# now. There's a link in the blog post to another blog post to that effect.