7 ms·

> Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. Basically, we needed to support a large existing Python 2.7 c

> Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. Basically, we needed to support a large existing Python 2.7 codebase. See discussion here: https://github.com/google/grumpy/issues/1 https://github.com/google/grumpy/issues/1 > It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it. There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable. > If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what? It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated.

AnkhMorporkian 10y ago
> There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable. I'm guessing pretty much the entire AST module is a no-go?
- trotterdylan 10y ago
  I think the CPython AST module is written as a C extension module so currently it's a no-go. I don't think there's a fundamental reason Grumpy couldn't run a pure Python AST module, though.
  - brettcannon 10y ago
    The ast module itself is in Python, but it imports the _ast module which is an extension module. This actually isn't that big of a deal, though, as the entire AST is defined in a DSL (see https://cpython-devguide.readthedocs.io/en/latest/compiler.html#abstract-syntax-trees-ast https://cpython-devguide.readthedocs.io/en/latest/compiler.h... for some details), so you just have to write some code to generate _ast in Python instead of C (which PyPy may have already done).
  - gok 10y ago
    So I take that means Grumpy can't run itself?
    - trotterdylan 10y ago
      Correct, Grumpy cannot yet run Grumpy :)
throwathrowaway 10y ago
> Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable. What about stuff like literal_eval? Or even just monkeypatching with name.__dict__[param] = value ? > It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated. Would there be a succinct theoretical description of exactly how that's implemented anywhere? What about things like numpy arrays.
- trotterdylan 10y ago
  > > Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable. > What about stuff like literal_eval? Or even just monkeypatching with name.__dict__[param] = value ? literal_eval could in principle be supported I think. name.__dict__[param] = value works as you'd expect: $ make run class A(object): pass a = A() a.__dict__['foo'] = 'bar' print a.foo bar EDIT: fixed formatting
- ehsankia 10y ago
  Hmm, numpy isn't pure python, is it? If I read correctly this only works with pure python.
  - genericpseudo 10y ago
    By volume numpy is mostly assembler written to the Fortran ABI (it's a LAPACK/BLAS-etc wrapper).
    - onalark 10y ago
      NumPy is a library that provides typed multidimensional arrays and functions that run atop them. It does provide a built-in LAPACK/BLAS or can link externally to LAPACK/BLAS, but that's a side effect of providing typed arrays and is nowhere near the central purpose of the library. Also, NumPy is implemented completely in C and Python, and makes extensive use of CPython extension hooks and knowledge of the CPython reference counting implementation, which is part of the reason why it is so hard to port to other implementations of Python.
Walkman 10y ago
Is there not a single namedtuple in the entire Google codebase? That's strange :o
- brianwawok 10y ago
  Are namedtuples that popular? They always felt awkward to me. If some temp variable with multiple values inside a loop, I either use normal tuple or a dict. If passing data around a dict or a real class. I never got the huge win from namedtuple?
  - jonasvp 10y ago
    Attrs (https://attrs.readthedocs.io/ https://attrs.readthedocs.io/) replaced namedtuple for us (and many others). It's slightly more verbose but allows all class goodness such as methods, attribute validation, etc.
    - justinsaccount 10y ago
      Doesn't work for everything, but you can subclass a namedtuple: from collections import namedtuple class Foo(namedtuple("Foo", "a b c")): @property def sum(self): return self.a + self.b + self.c f = Foo(1,2,3) print f.sum
      - brianwawok 10y ago
        That doesn't look super awesome to me. I.e. classes or attrs both seem better.
        deleted 10y ago
        [deleted]
    - foxylad 10y ago
      Aaaaaarrrrrgggggh! I've had that particular itch for every one of my ten years with python, and at last I get to scratch it! Thanks so much for bringing it up.
justinsaccount 10y ago
> I'll update the README to make note of them. I managed to run into 2 trying to build a 5 line program :-) $ cat t.py; ./tools/grumpc t.py > t.go;go build t.go;echo '----';./t import sys print sys.stdin.readline() ---- AttributeError: 'module' object has no attribute 'stdin' $ $ cat t.py ;./tools/grumpc t.py c = {} top = sorted(c.items(), key=lambda (k,v): v) Traceback (most recent call last): File "./tools/grumpc", line 102, in <module> sys.exit(main(parser.parse_args())) File "./tools/grumpc", line 60, in main visitor.visit(mod) File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit return visitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 302, in visit_Module self._visit_each(node.body) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 632, in _visit_each self.visit(node) File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit return visitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stin visit_Assign with self.expr_visitor.visit(node.value) as value: File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit return visitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 101, in visit_Call values.append((util.go_str(k.arg), self.visit(k.value))) File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit return visitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 246, in visit_Lambda return self.visit_function_inline(func_node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 388, in visit_function_inline func_visitor = block.FunctionBlockVisitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/block.py", line 432, in __init__ args = [a.id for a in node_args.args] AttributeError: 'Tuple' object has no attribute 'id'
- trotterdylan 10y ago
  Ugh, sorry about that. There's a couple issues here: 1. Lambda tuple args are not yet supported -- I actually didn't know that was a thing :\ -- https://github.com/google/grumpy/issues/17 https://github.com/google/grumpy/issues/17 2. Even if that worked properly, sorted() is not yet implemented: https://github.com/google/grumpy/issues/16 https://github.com/google/grumpy/issues/16
merb 10y ago
> Basically, exec and eval don't work. Couldn't they supported with a slower runtime implementation? I mean I still love the idea and actually like the idea.

AnkhMorporkian 10y ago

> There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable. I'm guessing pretty much the entire AST module is a no-go?

trotterdylan 10y ago
I think the CPython AST module is written as a C extension module so currently it's a no-go. I don't think there's a fundamental reason Grumpy couldn't run a pure Python AST module, though.
- brettcannon 10y ago
  The ast module itself is in Python, but it imports the _ast module which is an extension module. This actually isn't that big of a deal, though, as the entire AST is defined in a DSL (see https://cpython-devguide.readthedocs.io/en/latest/compiler.html#abstract-syntax-trees-ast https://cpython-devguide.readthedocs.io/en/latest/compiler.h... for some details), so you just have to write some code to generate _ast in Python instead of C (which PyPy may have already done).
- gok 10y ago
  So I take that means Grumpy can't run itself?
  - trotterdylan 10y ago
    Correct, Grumpy cannot yet run Grumpy :)

throwathrowaway 10y ago

> Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable. What about stuff like literal_eval? Or even just monkeypatching with name.__dict__[param] = value ? > It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated. Would there be a succinct theoretical description of exactly how that's implemented anywhere? What about things like numpy arrays.

trotterdylan 10y ago
> > Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable. > What about stuff like literal_eval? Or even just monkeypatching with name.__dict__[param] = value ? literal_eval could in principle be supported I think. name.__dict__[param] = value works as you'd expect: $ make run class A(object): pass a = A() a.__dict__['foo'] = 'bar' print a.foo bar EDIT: fixed formatting
ehsankia 10y ago
Hmm, numpy isn't pure python, is it? If I read correctly this only works with pure python.
- genericpseudo 10y ago
  By volume numpy is mostly assembler written to the Fortran ABI (it's a LAPACK/BLAS-etc wrapper).
  - onalark 10y ago
    NumPy is a library that provides typed multidimensional arrays and functions that run atop them. It does provide a built-in LAPACK/BLAS or can link externally to LAPACK/BLAS, but that's a side effect of providing typed arrays and is nowhere near the central purpose of the library. Also, NumPy is implemented completely in C and Python, and makes extensive use of CPython extension hooks and knowledge of the CPython reference counting implementation, which is part of the reason why it is so hard to port to other implementations of Python.

Walkman 10y ago

Is there not a single namedtuple in the entire Google codebase? That's strange :o

brianwawok 10y ago
Are namedtuples that popular? They always felt awkward to me. If some temp variable with multiple values inside a loop, I either use normal tuple or a dict. If passing data around a dict or a real class. I never got the huge win from namedtuple?
- jonasvp 10y ago
  Attrs (https://attrs.readthedocs.io/ https://attrs.readthedocs.io/) replaced namedtuple for us (and many others). It's slightly more verbose but allows all class goodness such as methods, attribute validation, etc.
  - justinsaccount 10y ago
    Doesn't work for everything, but you can subclass a namedtuple: from collections import namedtuple class Foo(namedtuple("Foo", "a b c")): @property def sum(self): return self.a + self.b + self.c f = Foo(1,2,3) print f.sum
    - brianwawok 10y ago
      That doesn't look super awesome to me. I.e. classes or attrs both seem better.
      - deleted 10y ago
        [deleted]
  - foxylad 10y ago
    Aaaaaarrrrrgggggh! I've had that particular itch for every one of my ten years with python, and at last I get to scratch it! Thanks so much for bringing it up.

justinsaccount 10y ago

> I'll update the README to make note of them. I managed to run into 2 trying to build a 5 line program :-) $ cat t.py; ./tools/grumpc t.py > t.go;go build t.go;echo '----';./t import sys print sys.stdin.readline() ---- AttributeError: 'module' object has no attribute 'stdin' $ $ cat t.py ;./tools/grumpc t.py c = {} top = sorted(c.items(), key=lambda (k,v): v) Traceback (most recent call last): File "./tools/grumpc", line 102, in <module> sys.exit(main(parser.parse_args())) File "./tools/grumpc", line 60, in main visitor.visit(mod) File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit return visitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 302, in visit_Module self._visit_each(node.body) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 632, in _visit_each self.visit(node) File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit return visitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stin visit_Assign with self.expr_visitor.visit(node.value) as value: File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit return visitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 101, in visit_Call values.append((util.go_str(k.arg), self.visit(k.value))) File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit return visitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 246, in visit_Lambda return self.visit_function_inline(func_node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 388, in visit_function_inline func_visitor = block.FunctionBlockVisitor(node) File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/block.py", line 432, in __init__ args = [a.id for a in node_args.args] AttributeError: 'Tuple' object has no attribute 'id'

trotterdylan 10y ago
Ugh, sorry about that. There's a couple issues here: 1. Lambda tuple args are not yet supported -- I actually didn't know that was a thing :\ -- https://github.com/google/grumpy/issues/17 https://github.com/google/grumpy/issues/17 2. Even if that worked properly, sorted() is not yet implemented: https://github.com/google/grumpy/issues/16 https://github.com/google/grumpy/issues/16

merb 10y ago

> Basically, exec and eval don't work. Couldn't they supported with a slower runtime implementation? I mean I still love the idea and actually like the idea.