6 ms·

Show HN: PyCParser, C parser and interpreter in Python

112 points by albertzeyer 7y ago

fwip 7y ago
Looks awesome - the interpreter seems really handy for learning C, too. It'll be a neat milestone once it's complete enough to interpret Cython. :)
- jonjacky 7y ago
  He's already started on it: https://github.com/albertz/PyCPython https://github.com/albertz/PyCPython
pfdietz 7y ago
A problem with tools like these is dealing with the C preprocessor. You can either parse preprocessed code (in which case you can't deal with the source file as written, for example to do rewrites or searches on it), or you hack on it to somehow accept "common" preprocessor idioms (but that's fragile and imperfect).
- rightbyte 7y ago
  Of all the pitfalls how can the text replace preprocessor be the hard part? Just apply the preprocessor on any input before the rest of the processing. The project looks kinda cool. I wonder if I could hack it for "real time feedback" of c code in Eclipse some how.
  - pfdietz 7y ago
    That's fine if you want to compile or interpret the program for a particular set of definitions. It's not so good if you want to manipulate the program text as a structured object.
    - userbinator 7y ago
      That's fine if you want to compile or interpret the program for a particular set of definitions. That's what this does. No one said anything about your second point.
      - pfdietz 7y ago
        I was pointing out a problem with C parsers. This is not really the tool's fault; it's a problem with the design of C itself. One would not have this problem with, say, Java. I don't much like preprocessors for this reason.
- userbinator 7y ago
  It has a preprocessor. You'll start to see it at around 20% of the way through cparser.py . I don't know how standards-compliant it is, because I just quickly skimmed over the code, but fortunately the correct algorithm has been documented: https://www.spinellis.gr/blog/20060626/cpp.algo.pdf https://www.spinellis.gr/blog/20060626/cpp.algo.pdf
tehsauce 7y ago
Can it run cpython?
- detaro 7y ago
  from the readme: https://github.com/albertz/PyCPython https://github.com/albertz/PyCPython
ColanR 7y ago
Is python the optimal language for writing a C parser? It seems like text manipulation is not one of python's strong points (not that that is a strong point of that many langugages anyway).
- sgillen 7y ago
  I certainly like text manipulation (well string manipulation at least) in python much more than other languages I use (C, C++, Matlab). It might just be the case that python is the best tool in the authors tool belt for this sort of work. I'm curious what language you would consider optimal for writing a parser in?
  - ColanR 7y ago
    > I'm curious what language you would consider optimal for writing a parser in? I've been researching that question, and was hoping someone would reply and point me in a useful direction.
    - anaphor 7y ago
      I've written a couple parsers in D using https://github.com/PhilippeSigaud/Pegged https://github.com/PhilippeSigaud/Pegged and they seem pretty fast and easy to verify. Of course since it generates a lot of code at compile time, you can end up with somewhat slow builds.
- userbinator 7y ago
  Text manipulation is far easier in Python and a lot of other high-level languages than it is in C. Incidentally, this ease (such as string concatenation with '+') also makes it easy to unknowingly/accidentally create a quadratic or higher complexity algorithm.
matmann2001 7y ago
All for projects in this space, but you're going to generate so much confusion naming it the same thing as Eli Bendersky's pycparser (which you even acknowledge the existence of in your Similar Projects section). The alternate capitalization makes no difference to a search engine.
- albertzeyer 7y ago
  I know, that's unfortunate. I think I learned only later about Eli Bendersky's pycparser and then I was too lazy to change the name, although you are right, I should have. But as my project is mostly a toy project at the moment, and not sure if this will every get serious usage, I'm not sure if this is too critical. Also, whoever searches for pycparser, and ends up at this homepage, will find the link to the real pycparser project right there in the readme. Both projects are quite old btw. I just checked. Eli Bendersky's pycparser is from May 2010 (or that was the first Git commit, which seems to be an import from an SVN repo), and I started mine in June 2011. I never really got any complaints about confusions.
  - chubot 7y ago
    FWIW I was confused by this title because I know of Bendersky’s pycparser project.
  - eliben 7y ago
    pycparser is actually from ~2008 :-) [http://eli.thegreenplace.net/2008/11/15/pycparser-v10-is-out http://eli.thegreenplace.net/2008/11/15/pycparser-v10-is-out], but it was probably less easily discoverable in those days - it was initially in SVN and the first release was a tarball drop. pycparser doesn't do anything about interpreting C, it ends its job at creating the AST. By the way, pycparser has no external dependencies. It uses PLY but vendors it. I had to port it to Python 3 before PLY did itself, and stuck to the vendoring because it's so convenient.

112 points by albertzeyer 7y ago

fwip 7y ago

Looks awesome - the interpreter seems really handy for learning C, too. It'll be a neat milestone once it's complete enough to interpret Cython. :)

jonjacky 7y ago
He's already started on it: https://github.com/albertz/PyCPython https://github.com/albertz/PyCPython

pfdietz 7y ago

A problem with tools like these is dealing with the C preprocessor. You can either parse preprocessed code (in which case you can't deal with the source file as written, for example to do rewrites or searches on it), or you hack on it to somehow accept "common" preprocessor idioms (but that's fragile and imperfect).

rightbyte 7y ago
Of all the pitfalls how can the text replace preprocessor be the hard part? Just apply the preprocessor on any input before the rest of the processing. The project looks kinda cool. I wonder if I could hack it for "real time feedback" of c code in Eclipse some how.
- pfdietz 7y ago
  That's fine if you want to compile or interpret the program for a particular set of definitions. It's not so good if you want to manipulate the program text as a structured object.
  - userbinator 7y ago
    That's fine if you want to compile or interpret the program for a particular set of definitions. That's what this does. No one said anything about your second point.
    - pfdietz 7y ago
      I was pointing out a problem with C parsers. This is not really the tool's fault; it's a problem with the design of C itself. One would not have this problem with, say, Java. I don't much like preprocessors for this reason.
userbinator 7y ago
It has a preprocessor. You'll start to see it at around 20% of the way through cparser.py . I don't know how standards-compliant it is, because I just quickly skimmed over the code, but fortunately the correct algorithm has been documented: https://www.spinellis.gr/blog/20060626/cpp.algo.pdf https://www.spinellis.gr/blog/20060626/cpp.algo.pdf

tehsauce 7y ago

Can it run cpython?

detaro 7y ago
from the readme: https://github.com/albertz/PyCPython https://github.com/albertz/PyCPython

ColanR 7y ago

Is python the optimal language for writing a C parser? It seems like text manipulation is not one of python's strong points (not that that is a strong point of that many langugages anyway).

sgillen 7y ago
I certainly like text manipulation (well string manipulation at least) in python much more than other languages I use (C, C++, Matlab). It might just be the case that python is the best tool in the authors tool belt for this sort of work. I'm curious what language you would consider optimal for writing a parser in?
- ColanR 7y ago
  > I'm curious what language you would consider optimal for writing a parser in? I've been researching that question, and was hoping someone would reply and point me in a useful direction.
  - anaphor 7y ago
    I've written a couple parsers in D using https://github.com/PhilippeSigaud/Pegged https://github.com/PhilippeSigaud/Pegged and they seem pretty fast and easy to verify. Of course since it generates a lot of code at compile time, you can end up with somewhat slow builds.
userbinator 7y ago
Text manipulation is far easier in Python and a lot of other high-level languages than it is in C. Incidentally, this ease (such as string concatenation with '+') also makes it easy to unknowingly/accidentally create a quadratic or higher complexity algorithm.

matmann2001 7y ago

All for projects in this space, but you're going to generate so much confusion naming it the same thing as Eli Bendersky's pycparser (which you even acknowledge the existence of in your Similar Projects section). The alternate capitalization makes no difference to a search engine.

albertzeyer 7y ago
I know, that's unfortunate. I think I learned only later about Eli Bendersky's pycparser and then I was too lazy to change the name, although you are right, I should have. But as my project is mostly a toy project at the moment, and not sure if this will every get serious usage, I'm not sure if this is too critical. Also, whoever searches for pycparser, and ends up at this homepage, will find the link to the real pycparser project right there in the readme. Both projects are quite old btw. I just checked. Eli Bendersky's pycparser is from May 2010 (or that was the first Git commit, which seems to be an import from an SVN repo), and I started mine in June 2011. I never really got any complaints about confusions.
- chubot 7y ago
  FWIW I was confused by this title because I know of Bendersky’s pycparser project.
- eliben 7y ago
  pycparser is actually from ~2008 :-) [http://eli.thegreenplace.net/2008/11/15/pycparser-v10-is-out http://eli.thegreenplace.net/2008/11/15/pycparser-v10-is-out], but it was probably less easily discoverable in those days - it was initially in SVN and the first release was a tarball drop. pycparser doesn't do anything about interpreting C, it ends its job at creating the AST. By the way, pycparser has no external dependencies. It uses PLY but vendors it. I had to port it to Python 3 before PLY did itself, and stuck to the vendoring because it's so convenient.