43 ms·

Grumpy: Go running Python

1411 points by trotterdylan 10y ago

brian_herman 10y ago
No C extensions boo.
- MBCook 10y ago
  But isn't that really what lets them do this? In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions. As soon as something can see into the guts of the interpreter you have to maintain compatibility which is a pain/waste. Worse than that is that view wasn't designed for multi-threading which is why the GIL exists. The C extensions were t designed to be multi-threaded because that wasn't a thing in Python so they're not safe. You either have to drop them, define a new interface layer that would be safe, or I suppose somehow sandbox their little view of the world but keep it coherent between threads. If you have a codebase where you can make the choice to drop C extensions and you're trying to accelerate Python it seems like a very smart choice.
  - guitarbill 10y ago
    The irony is that some C extensions exist because they have better performance than Python. On the other hand, there are many C extensions which are just interop/wrappers for existing C code. I think no language is naive enough to think they can get away without C interop. C extensions are actually pretty nice because you can wrap the C code into idiomatic Python. With ctypes, you have to e.g. maintain two structure definitions, which is very problematic for some codebases. > In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions. AFAIK, many projects initially struggle to even achieve performance parity with CPython. The GIL isn't evil, it's a simple solution to a hard problem. But at this point, a complex solution to a hard problem is better if it's faster, and some people care more about speed than C interop (and vice versa). What would be awesome is a Python runtime that could run with fine-grain locking until a C extension is loaded, and then continue with coarse locking. But the problem is still there's no upgrade path for C interop. The only thing I can think of is using type annotations and something like Cython's cdef to write Python-implementation-independent C interop that doesn't suck as bad as ctypes. Then the Python rumtime could also lock the arguments at a very fine level while the function is being called.
deleted 10y ago
[deleted]
naasking 10y ago
I wonder why the Grumpy Fibonacci is so much slower than CPython for 1 thread. Seems weird given Grumpy is compiled.
- sgift 10y ago
  From the post it looks like it is optimized for multithreading from the start, so they possibly used the multithreaded version with thread count = 1. When you care for parallel workloads its not so important if your single-threaded code is the fastest.
- deleted 10y ago
  [deleted]
- deleted 10y ago
  [deleted]
- machbio 10y ago
  Guido insists that single threaded performance of Python has the highest priority - http://www.artima.com/weblogs/viewpost.jsp?thread=214235 http://www.artima.com/weblogs/viewpost.jsp?thread=214235 I am not sure, any implementation of Python will beat the single threaded performance of Cpython..
  - Veedrac 10y ago
    CPython is still a naïve interpreter. PyPy beats it in most cases, sometimes by a lot. Of course, PyPy still has a GIL for the same reasons CPython does.
  - dekhn 10y ago
    IIRC, IronPython single-threaded performance was competitive with CPython.
  - throwaway91111 10y ago
    Multiple ones already do.
  - dom0 10y ago
Twirrim 10y ago
This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?
- matt_wulfeck 10y ago
  That's a question I have as well. I'm guessing they have a very massive cpython codebase, and the trade-off was worth it.
  - MBCook 10y ago
    I'd assume that's true, but they also didn't know how well this would work. A 'little' project to make Grumpy is probably much easier to sell to management/throw away if to slow than "Let's rewrite all our code in language X to see if it goes faster".
- greenhatman 10y ago
  Maybe there is too much to rewrite.
- ericfrederich 10y ago
  It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.
  - martincmartin 10y ago
    I think he's talking about moving their Python codebase to Go, so they don't have to run Python anymore.
    - ericfrederich 10y ago
      You're probably right. He seemed to be questioning the translator so I thought he was suggesting replacing CPython piece by piece with a Go implementation.
oblio 10y ago
It seems interesting but my concern is that it's just another Unladen Swallow.
- the_duke 10y ago
  Well, if Google is pushing it and actually using it internally... They also have the benefit of being able to push features into the Go core that they might need for this.
  - ProAm 10y ago
    > Well, if Google is pushing it and actually using it internally... It will be silently abandoned in 18 months....
    - ehsankia 10y ago
      Not if it turns out to be usable and have a significant impact on the performance. You need to realize that at the scale Youtube works, even a small performance boost translates to huge savings in server cost. So the worst case here is if they never improve this past their own needs (which is a pretty limited subset). But if it's successful for them, and it's opensource, I could very easily see other people who run heavy stuff on Python contributing to it and helping it grow. That's the thing with opensource, even if Google doesn't actively work on it, others can (if it has actual value and is useful to people).
  - vorg 10y ago
    Perhaps Golang will get a VM or something similar to enable stuff like exec/eval to be implemented with rapid response times in Grumpy. The only way I've ever seen a REPL implemented for Go is by recompiling the source code via a call to "os/exec".Command for each statement or declaration entered, which gives a 1-second delay on many computers.
  - tedmielczarek 10y ago
    Google has pretty specific needs, so this project may be very successful internally even if it gets zero traction externally. A lot of big software companies do this nowadays. Google and Facebook both have a lot of purpose-built software, some of which gets released as open source, that meets their needs well but is hard to use for other purposes. I guess it's still strictly better than them not open-sourcing the code, but it's definitely an existence proof that just making something open source doesn't make magic happen.
nodivbyzero 10y ago
Facebook: PHP -> HipHop Google: Python -> Go
scrollaway 10y ago
That looks like a super interesting runtime. Seems to target 2.7 only, I hope they're open to supporting 3.x as well.
- MBCook 10y ago
  I wonder if they intend to maintain it long term or if they're just going to use it as a bridge while they rewrite things natively in Go.
  - ericfrederich 10y ago
    Seems like a huge effort for the interim... unless they really need interoperability between Python and Go during the switch.
- trotterdylan 10y ago
  I'd like to support 3.x at some point. See https://github.com/google/grumpy/issues/1 https://github.com/google/grumpy/issues/1
  - statsmatscats 10y ago
    Have you thought about using type hints to help type inference? Some work on that is discussed here. I would love a dropbox google colab (though also targeting 3.x :) ) https://github.com/python/mypy/issues/1862 https://github.com/python/mypy/issues/1862
    - trotterdylan 10y ago
      Yes, leveraging type hints for optimization purposes is a long term goal. Thanks for pointing me to that issue, I'll keep an eye on it. One of the goals of open sourcing was to get feedback and work with outside folks so I'm definitely open to collaboration!
      - statsmatscats 10y ago
aibottle 10y ago
This actually pretty great. Even if you will never run your code in grumpy, it underlines the status of Python. Having large corps investing in Python will help pulling in new talent into the Python environment also it strengthens the ecosystem of Python.
- the_duke 10y ago
  Actually, I would interpret this as "CPython is broken for scalability. It's not worth trying to fork/fix it, we'd rather just roll our own runtime". Not exactly an endorsement. Sadly, the code was just dumped into a new Git repo, so no way to tell how many people contributed internally so far.
  - ericfrederich 10y ago
    I agree and disagree ... on one side this may signal a shift away from Python. Something to help the migration effort and the ultimate goal is to write everything in Go. ... but then I don't see Python going anywhere anytime soon. Didn't Microsoft just start a project to get Python's runtime to use CoreCLR's JIT? There was an article, can't find it now, about an upcoming Python renaissance saying there may be an influx of new interpreters. There's PyPy, Microsoft's CoreCLR thing, now this, etc. It seems people really want to program in Python so there is an effort to make it faster. *edit: found the article: https://lwn.net/Articles/691070/ https://lwn.net/Articles/691070/
    - the_duke 10y ago
      Devs love Python and will continue to do so, no question there. I'm just afraid the surge of different compilers and interpreters will bring up plenty of issues in the medium term. There is no formal spec of Python like there is for, eg, Javascript. (which is of course driven by multiple, VERY engaged adopters). How long until subtle and not so subtle differences creep in between different implementations, leading to incompatabilities and a continuous fragmentation of the ecosystem?
    - dagss 10y ago
ohitsdom 10y ago
Very interesting project and technical solution. Is this in use at Google? The described example problem is Youtube with millions of requests per second, but the post doesn't say if it Grumpy was put in production (and what performance gains were then achieved).
- Zikes 10y ago
  I'm thinking it isn't in production yet, based on wording like "we're excited about the prospects" and "although it's still alpha software".
- hacker_9 10y ago
  Yeah without performance metrics I'm not sure what to think about this. I'm surprised they even used Python to begin with for their front-end server, for such high volumes of traffic.
- trotterdylan 10y ago
  As has been said by others, Grumpy is not used in production at Google currently. There's still a lot of work to do -- especially on the standard library -- to support large real world codebases.
gamesbrainiac 10y ago
In case anyone wanted the github repo: https://github.com/google/grumpy https://github.com/google/grumpy
ericfrederich 10y ago
Python needs a new runtime. This talk shows how bad of shape it's really in. https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seLRqszBqVDF342RMlCWgOTm6q&index=11 https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seL... Basically, the language doesn't have a "spec" per-se. The language is whatever the defacto CPython implementation happens to do within it's giant eval loop. Another great talk about CPython internals: http://pgbovine.net/cpython-internals.htm http://pgbovine.net/cpython-internals.htm
- dijit 10y ago
  cpython is the reference implementation, so it makes sense that it's; A) Not well optimised. B) Touting features before the spec/standard. EDIT: people really dislike that I said this, and I'm having trouble finding my original citation- it was on one of the many python books I own. Most likely "Learn Python The Hard Way" but I'll dig out the exact chapter where they compare pypy to cpython and mention that because cpython is the reference implementation it values code clarity over performance optimisation.
  - the_duke 10y ago
    It makes sense that the official, primary and by far most popular implementation of one of the most used languages in existance is not well optimized? (edit: I'm just being polemic about your statement here. CPython is reasonably optimized within the constraints it currently has).
    - nodja 10y ago
      It only makes sense because it's python. A language where style is part of the syntax, and readability is one of the things that many libraries focus on , the so called being "pythonic". It makes sense that the reference implementation mirrors the same patterns than the language itself.
      - the_duke 10y ago
        Someone else already posted this link: https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seLRqszBqVDF342RMlCWgOTm6q&index=11 https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seL... It explains why CPython can't improve on many things.
sfifs 10y ago
The main reason I moved from coding in Python to Go as my main language many years back is because concurrency was such a pain in standard Python (the other was compile time error checking). It's interesting to see the same pain has now made caused the runtime itself to be implemented in Go. It's a pity C extensions (often used in scientific computing) are not supported but Go does have support via CGO, so maybe some approach can be worked out to access C routines in the future.
- vram22 10y ago
  Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? Is it to do with Go's being a GC'd language? I would have thought that should not be a reason per se, since Python also has GC, but has plenty of extensions written in C. But I'm not a language internals expert. Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libraries somewhat easily (not sure about all cases or how far the ease goes).
  - moosingin3space 10y ago
    It's because Go uses a different stack structure, called "segmented stacks", in order to enable cheap goroutines. Basically, Go stacks start tiny (8 KiB, as opposed to much larger C stacks), then it grows them in small segments. Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches, and ordinary C function calls conflict with this event loop.
    - vram22 10y ago
      >Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches Interesting, didn't know this (that Go code runs in an event loop). Is the reason something to do with goroutines and channels? something like, a routine gets info that data is available for it to read (on a channel, sent by another goroutine), via an event it receives? Also, can you explain this point: "which enables excellent I/O performance without kernel context-switches" ?
the_duke 10y ago
Haha, now that's what I call an interesting project. The biggest surprise for me is that the Go runtime would be a good fit for Python, performance wise, considering the very different object and dispatch model. The post also mentions runtime reflection, which used to be painfully slow last time I used it. (Go 1.5, i think). Has this improved in the latest releases?
- jerf 10y ago
  A huge part of the reason Python (and Perl, PHP, etc. in their original interpreters) is so slow is that it is like running a Go program that does everything through runtime reflection, even just to add two integers. If your code is already in Python, this level of performance is apparently not a problem for you. If you know Go or are willing to learn about Go and reflection, you can learn a lot about how dynamic languages work under the hood by implementing: func Add(interface{}, interface{}) interface{} { ... } using the reflect module to accept all types of numbers, including for a bit of extra fun the math.Big* number types, and returning upgraded numbers as appropriate, or panicking on types you can't Add with. That's not all there is to writing a dynamic language interpreter, but I'd say you can learn the core idea this way, shorn away from a lot of accidental complexity and with a lot of the grunt work plumbing of setting up (type, value) pairs already done for you.
- trotterdylan 10y ago
  Reflection in Go is used minimally in Grumpy because it is slow. Currently importing Go packages into Python code is accomplished via the reflect module, but I think this will have to change to make such integration useful.
0xdeadbeefbabe 10y ago
> Once we started going down the rabbit hole, Go seemed like an obvious choice... Good? It's funny that a rabbit hole is the place you go to make obvious choices. This culture must seem strange to outsiders.
epanastasi 10y ago
As someone who works on both python and go day to day, I find this to be quite interesting. Just tried this out on a reasonably complex project to see what it outputs. Looks like it only handles individual files and not any python imports in those files. So for now you have to manually convert each file in the project and put them into the correct location within the build/src/grumpy/lib directory to get your dependencies imported. Unless I missed something somewhere.. The documentation is a bit sparse. Overall I think the project has a lot of potential and I'm hoping it continues to be actively developed to smooth out some of the rough edges.
- ericfrederich 10y ago
  I hope this is a well thought out solution that can evolve into something great... and not just something built for a single purpose. I question the transpiler. I think I'd much rather prefer a solution like Jython.
  - Floegipoky 10y ago
    I'm confused because Jython runs on the JVM, but Go is a compiled language. Can you clarify?
    - bb88 10y ago
      Jython is a python interpreter written in Java. Grumpy is a python transpiler that converts python to navtive go object code. Edited to add: The difference is that Jython doesn't covert python to JVM bytecode.
      - weberc2 10y ago
        What's the advantage of writing an interpreter? Go already has an excellent runtime (scheduler, GC, etc)--why should this project reimplement it?
        solidsnack9000 10y ago
rpedela 10y ago
How does Grumpy handle Decimal in Python? As far as I am aware there isn't an equivalent in Go.
- brettcannon 10y ago
  The decimal module in Python 2.7 is implemented in pure Python: https://github.com/python/cpython/blob/2.7/Lib/decimal.py https://github.com/python/cpython/blob/2.7/Lib/decimal.py (in Python 3 there is an accelerated extension module that's used when available, falling back on the pure Python version otherwise).
  - webmaven 10y ago
    So, does Decimal work with Grumpy (you're implying it does)? For that matter, does Grumpy match CPython's Float behavior exactly? ie.: >>> 2.2 + 3.1 5.300000000000001 >>>
    - brettcannon 10y ago
      I'm implying that they can get the decimal module into Grumpy by implementing enough of Python to support the module (i.e. they don't need to implement the decimal module from scratch because the module isn't implemented only in C).
      - webmaven 10y ago
        Got it. Thanks for the clarification, Brett.
matthewrudy 10y ago
I wonder if future work might support C Extensions in the same was as the jruby truffle / graal implementation plans to do. Here's a great article from a couple of years ago by Chris Seaton on this topic. http://chrisseaton.com/rubytruffle/cext/ http://chrisseaton.com/rubytruffle/cext/
deleted 10y ago
[deleted]
zedpm 10y ago
I would love to see this support Python 3.5+, specifically for asyncio. Concurrency via threads in Python is far less appealing.
bigato 10y ago
I see some parallel with the work made to automatically convert Go compiler C code to Go code. Strategically wise, I'd rather have the software do the transpilation to the new language and then after extensive testing, ditch the old python codebase. But they probably don't agree with my preference of Go over Python.
quotemstr 10y ago
I'm a bit disappointed that the blog post doesn't explain why the authors didn't choose PyPy instead.
- ot 10y ago
  The objective of the authors is to efficiently exploit multi-core parallelism. PyPy still has a GIL. They have been doing some experiments with transactional memory, but performance is quite bad.
- theandrewbailey 10y ago
  > These efforts have borne a lot of fruit over the years, but we always run up against the same issue: it's very difficult to make concurrent workloads perform well on CPython. > To solve this problem, we investigated a number of other Python runtimes. Each had trade-offs and none solved the concurrency problem without introducing other issues.
jerf 10y ago
For those who are interested, I've used grumpy to compile the following Python code and placed it at https://play.golang.org/p/YP1SP7WsdR https://play.golang.org/p/YP1SP7WsdR . (Note the playground can't run this, it just had convenient formatting support for Go; the generated source wasn't 100% gofmt compliant.) class Test(object): def __init__(self, value): self.value = value def method(self): print(self.value) class Test2(Test): pass t = Test("hello") t.method() Pythonistas, note I had to have "class Test(object):" and not just "class Test:". The former compiled successfully into a Go program but that program then failed at runtime with "TypeError: class must have base classes".
- trotterdylan 10y ago
  Thanks for trying it out! Yeah, Grumpy does not currently support old-style classes. Since all of our code internally requires new-style classes, this was not a high priority feature. It is something that we'll get to.
  - jerf 10y ago
    I did not mean that as a complaint against your very young codebase, I meant that as a defense against Python people complaining about my code. :)
    - kstrauser 10y ago
      I don't think they'd do that. All Python 2 code I've seen uses `class Foo(object)`, at least since 2.2 came out.
- Animats 10y ago
  That's fascinating. It's creating run-time data structures similar to CPython's for data, and manipulating them with very general code. There seems to be a type comparable to Python's internal CObject, and it's used for most (all?) data. It's not generating Go that looks anything like human-written Go. There's no sign of type inference, although it's hard to tell from such a simple example. It's a lot like a Python run time environment, where everything is a CObject. Still, once you can do that, you can start optimizing, such as inferring that something is an integer and using ordinary Go arithmetic types. All that stuff with "switch" seems to be to handle Python exceptions in a language that doesn't have exceptions. Maybe later, analysis can tell that some function can't raise an exception, and translated calls for such functions can be simpler.
thedjinn 10y ago
I'm curious to why they started this project when there already is low hanging fruit that can speed up Python (e.g. pypy). What makes this better other than to satisfy the inner go-fanboy?
- zardeh 10y ago
  Pypy still has a GIL. GIL-free seems to be one of the explicit goals of this project.
  - poooogles 10y ago
    >Pypy still has a GIL. Not strictly true, http://doc.pypy.org/en/latest/stm.html http://doc.pypy.org/en/latest/stm.html. In general for the main project however, this is true.
    - zardeh 10y ago
      Pypy still has a gil. Pypy is experimenting with non-GILed versions, and in fact there are people actively working to remove the GIL from pypy. But the same can be said of cpython (the gilectomy), and yet I don't think that you'd say "not strictly true" to "cpython still has a GIL".
- nemith 10y ago
  GIL still exists in pypy
- alrs 10y ago
  Because they want to get off of Python completely, and forever. They're not looking for speed, they're looking to ditch the liability of supporting Python and its ecosystem.
  - JPKab 10y ago
    Yeah, they didn't at all say that.
Animats 10y ago
- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. - It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it. - If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?
- ionforce 10y ago
  What does "hard-code compiler" mean?
  - zardeh 10y ago
    It seems to be that it pesudo-transpiles python to go and compiles that down using a normal go toolchain.
    - omaranto 10y ago
      What's the difference between transpiling and pesudo-transpiling? (Even if you meant pseudo-transpiling, I still don't know what the difference between that and transpiling is.)
      - zardeh 10y ago
        That was a typo. I meant pseudo. I don't actually know that there is any difference in this case. I made that statement hastily. It transpiles, nothing pseudo about it.
  - eva1984 10y ago
    Transpilers with no runtime.
- 10y ago
ericfrederich 10y ago
There was also this effort: https://blog.heroku.com/see_python_see_python_go_go_python_go https://blog.heroku.com/see_python_see_python_go_go_python_g...
sea6ear 10y ago
Reminds me of the (old I guess?) ShedSkin project to automatically translate Python to C++. I suppose the easy concurrency and ability to inter-operate with Go libraries is really the driver for Go over that.
- Arkanosis 10y ago
  Shed Skin is still living at https://shedskin.github.io/ https://shedskin.github.io/ , but hasn't changed much since the last release in 2013.
ot 10y ago
There's no mention of whether Grumpy passes the CPython test suite. Until it doesn't, it's not a Python runtime, it's a compiler for a language with Python-like syntax. Compilers like this, from almost-Python to say C/C++, have existed for a while: Cython, Shedskin, Nuitka are some examples.
- trotterdylan 10y ago
  It does not, yet. The standard library is very incomplete at this point.
  - gigatexal 10y ago
    experimental ~= alpha, give the team a break
    - weberc2 10y ago
      You're responding to the principle developer. :)
      - gigatexal 10y ago
        I know, but i figure people will read his comments first and then see this.
  - ot 10y ago
    I'm not particularly concerned about the standard library, but about the more dynamic features of the language. Not just exec/eval, but for example the complex dispatching logic behind magic methods, especially the various __getattr__, __getattribute__. They are where alternative runtime implementations usually stumble, as if they were an intrinsic bottleneck to Python runtime performance.
    - trotterdylan 10y ago
      exec/eval do not work, but all of the other dispatching and magic methods are supposed to work exactly like CPython (and if they don't, it's a bug/not yet implemented).
gigatexal 10y ago
on a related note: given my affection for python and having, until just now, no idea that YouTube runs python (albeit 2.7) at Google scale (a language not known for being able to scale very well given the GIL and such)-- I now more than ever would love to work on something like that. Welp, back to learning and hacking with python.
deleted 10y ago
[deleted]
stiff 10y ago
It's actually a transpiler written in Python that generates Go code.
- kodablah 10y ago
  Nice. I've been toying with doing similar for the JVM [0]. I think people are coming to realize that while Go is not a great/powerful language (IMO), the runtime is great. I would not be surprised if more and more people start targetting Go if they want a GC and cross platform static compilation w/out LLVM complications and get a nice stdlib for free. 0 - https://github.com/cretz/goahead https://github.com/cretz/goahead
  - bsaul 10y ago
    First to write a language that is basically go + generics wins
cbhl 10y ago
The blog post says that YouTube is the inspiration for this runtime, but does YouTube run on Grumpy yet?
- trotterdylan 10y ago
  YouTube does not run on Grumpy. There is a lot of work left to do before Grumpy can run a large existing codebase.
zigzigzag 10y ago
How does this compare to Jython? The blog post says they looked at alternative runtimes but didn't like that they had tradeoffs (implying that Grumpy has no tradeoffs??). But Jython has been around for quite a while and also has no GIL: http://www.jython.org/jythonbook/en/1.0/Concurrency.html http://www.jython.org/jythonbook/en/1.0/Concurrency.html It can also handle Python's dynamic aspects.
- Twirrim 10y ago
  Every time I've tried to use Jython, I've found it won't work with pre-existing code all that well. In part it has exposed CPython "implementation quirks" that people were wittingly or otherwise taking advantage of. In other cases there doesn't seem to be obvious reasons for the differences and has required special-casing the python code to handle it. It has been great with code written from scratch, specifically for it.
  - zigzigzag 10y ago
    That sounds like the kind of issue you'd have with any reimplementation of CPython though. It sounds like Grumpy doesn't support quite a few things, so I still wonder how they compare and why developing a new runtime was considered easier than reusing Jython.
    - toyg 10y ago
      There might be a strategic element: Jython development has been erratic throughout the years.
      - zigzigzag 10y ago
        Google's own projects have been developed erratically throughout the years. They could have simply forked it.
BuckRogers 10y ago
Is this the final nail in the coffin for Python3? Seems like it. Who would use 3 if you could have CPython2 for existing code and write new code in Grumpy Python? This is the dream language for me. Python on the Go runtime.
- int_19h 10y ago
  Everyone who likes the numerous new language features, or new syntax for old features, in Python 3?
  - BuckRogers 10y ago
    Those can be built into Grumpy if anyone cares to do so. You don't need Python3 for that stuff. Someone, one developer, recently released a Python 2.8 that backported almost every new Python3 feature. I'm hoping it becomes a permanent fork of Python2 that uses the Go runtime. That would really be great and exactly what they've got now.
    - int_19h 10y ago
      If you backport every new Python3 feature to Python2, it becomes Python3, by definition.
      - dismantlethesun 10y ago
        It could become Python 2.8, wherein the standard library has both the legacy versions and the new versions, and any cross-compatible syntax is allowed. This leaves us with more than 1 way to do things, like meta-class declaration. There's a lot in Python 3 where changes were made to the syntax for 'clarity', but those worts weren't removed for any technical reason but because of the thought that since backwards compatability was being broken anyways, then we might as well get the most bang for our buck.
        int_19h 10y ago
        > It could become Python 2.8, wherein the standard library has both the legacy versions and the new versions, and any cross-compatible syntax is allowed. So two `builtin` modules, then? What if someone does `sys.modules['builtin']`? Or any other kind of explicit string-based lookup? How does pickle figure out which types to instantiate? There'd be a lot of types with same qualified names but different implementations with this approach... It feels like the only way this would work reliably, is if you completely isolate the Py2 and Py3 universes. So if you e.g. pickle from Py2 code, it only looks at modules and types that Py2 universe knows, and vice versa. But then what happens when code using the old library interacts with the new one (e.g. tries to pass objects around)? If that is prohibited, then you effectively still have two different languages, just with a single shared implementation - but no ability to gradually replace bits and pieces of code, for example, which would seem to be the biggest motivation for such a thing.
waitingkuo 10y ago
Will Grumpy be a good fit for go's scientific packages such like gonum?
- trotterdylan 10y ago
  I don't know. I hope so. I think this is an area where Go can succeed and integration with existing Python libraries could be useful.
  - waitingkuo 10y ago
    This is amazing work. Look forward to the integration between go & python
curtis 10y ago
One of my complaints about Go is that it seems to be designed with the (mistaken, in my opinion) notion that what we really need is just a better C. One way to leverage C's widespread availability and high-performance while side-stepping some of its deficiencies was simply to use it as a target for a different language. The Cfront C++ compiler which generated C code is probably the most famous example, but I recall that there were many others. Maybe Go, like C, will make a fruitful target for other language implementations.
- coldtea 10y ago
  >One of my complaints about Go is that it seems to be designed with the (mistaken, in my opinion) notion that what we really need is just a better C. It was designed with the notion that all some people really need is just a better C. For other people there's Swift, Rust, etc.
  - curtis 10y ago
    For better or worse, I think Go sucked up a lot of the metaphorical oxygen for similar languages. Swift and Rust are in different niches and don't compete head to head with Go, so this isn't a real problem for them. What I've long wanted is a better C++, not a better C. It would be hard for any such language to succeed today, because it will have to compete with Go, and Go has a fairly mature toolchain and widespread adoption and a lot of mindshare. D literally is designed to be a better C++, and it hasn't really been a big success. Maybe that's because there really wasn't as big a demand for a better C++ as there was for a better C. On the other hand it may have been because Go sucked up a lot of the oxygen that D was going to need to succeed, and maybe that was because Go was a product of Google and D wasn't. (If this was my primary point, I'd make a more nuanced argument, though.) I'm now wondering if the best way to get to a better C++ might be by piggy-backing on the Go toolchain. To get back to my original point, I'm wondering if Go might in fact be a good target for all sorts of better (for some definition) programming languages.
    - wrsh07 10y ago
      For my money, a better C++ is in development: it's called C++20.
agentgt 10y ago
* I would love to know how big the codebase is. * It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces. * Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)? Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)? Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7). With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C?
- trotterdylan 10y ago
  > I would love to know how big the codebase is. Sorry, can't be very specific, but rewriting all the frontend code would take a lot more effort than writing a new Python runtime :) > It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces. We do iteratively rewrite components as well. We are pursuing multiple strategies. > Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)? Most async operations are performed out-of-process by other servers. > Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)? I'd love to support async and await at some point. > Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7). We did also do an evaluation of Jython but there were a number of technical issues that made it unsuitable for our codebase and workload. One such example is this longstanding issue: http://bugs.jython.org/issue527524 http://bugs.jython.org/issue527524. I just noticed the very recent update on that thread that implemented the workaround outlined in 2010 by Jim Baker. We tried that workaround and found we got a huge performance hit on affected code. There were a few other general performance problems as well but I can't recall all the details. Please note I'm not at all bashing Jython, I think it's a great project with a sound design, it just wasn't right for us. > With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C? You can interface with Go code directly, e.g. from the blog post: from __go__.net.http import ListenAndServe, RedirectHandler handler = RedirectHandler('http://github.com/google/grumpy', 303) ListenAndServe('127.0.0.1:8080', handler)
agounaris 10y ago
Whatever the haters say here, this is really cool!
weberc2 10y ago
I tried a simple `http.Get()` example, but the resultant `Response` object appears to not allow access to any of the struct fields as attributes. How does one access a Go object's fields from Python? For example, I want to `io.Copy(os.Stdout, rsp.Body)`.
- trotterdylan 10y ago
  Unfortunately the native interface is still pretty immature so I can't guarantee things will work as they should. In this case, I think the problem is that you have a *Response object which is not itself a struct. I've rewritten this part of the code a couple times and I think this functionality was lost. I've filed: https://github.com/google/grumpy/issues/13 https://github.com/google/grumpy/issues/13
  - trotterdylan 10y ago
    Whoops, just noticed you (or somebody) had already filed it: https://github.com/google/grumpy/issues/12 https://github.com/google/grumpy/issues/12
    - weberc2 10y ago
      Yeah, that was me. I should have updated my post here accordingly. Apologies for the inconvenience!
      - trotterdylan 10y ago
        It's all good. Thanks for filing the issue. I'll get that fixed.
senthilnayagam 10y ago
wish there was one for ruby as well
Antwan 10y ago
"So we asked ourselves a crazy question: What if we were to implement an alternative ?" As always with Google... What if you humbly contribute to open source projects rather than creating new stuffs, labelled with your own brand, controlled by your own engineers, and with your own design choices, however good they may be ?
- Sphax 10y ago
  How entitled can you be ? Why are they required to do anything of the sort ?
- tedmielczarek 10y ago
  They tried that with CPython, it didn't work: https://www.python.org/dev/peps/pep-3146/ https://www.python.org/dev/peps/pep-3146/
  - teddyh 10y ago
    But it was no fault of Python: https://qinsb.blogspot.nl/2011/03/unladen-swallow-retrospective.html https://qinsb.blogspot.nl/2011/03/unladen-swallow-retrospect...
    - tedmielczarek 10y ago
      Interesting, I hadn't seen that post! I wonder if there are any posts that elaborate on the internal Google bits more. I'd love to know what "they found other ways to solve their performance problems" wound up meaning in practice.
      - webmaven 10y ago
        > I'd love to know what "they found other ways to solve their performance problems" wound up meaning in practice. Me too, especially as this was already several years after the YouTube acquisition. Also interesting to note that potential internal customers were put off by having to upgrade to 2.6.1 in order to use unladen-swallow (presumably they got over that reticence at some point, as 2.7 is now standard).
jamestimmins 10y ago
Does anyone know if there is an easy way to get involved in open source projects like this? The readme mentions adding PRs, but as someone that doesn't have much experience working with open source projects like this, I don't even know where to begin. It sounds like an incredible learning opportunity though.
- steveklabnik 10y ago
  The first stop is usually a CONTRIBUTING.md Here's theirs https://github.com/google/grumpy/blob/master/CONTRIBUTING.md https://github.com/google/grumpy/blob/master/CONTRIBUTING.md There's not a ton there. The next place I'd go is open issues: https://github.com/google/grumpy/issues https://github.com/google/grumpy/issues It looks like at the moment they're mostly random bug reports from people who have tried Grumpy since this announcement, rather than ones filed by people working on Grumpy since before it was made public. So that's a bit trickier. The last place I'd look is then, the README: https://github.com/google/grumpy https://github.com/google/grumpy not a ton there about ways they wish to have people contribute. At this point, what I'd do personally is open an issue asking how you can get involved; possibly by improving this documentation on how to get involved! Anyway, that's what I'd do. Hope that helps!
  - jamestimmins 10y ago
    That's super helpful; much appreciated!
    - steveklabnik 10y ago
      No worries. Open source can be tricky to get into, but it's largely about just keeping at it: A story I like to tell is how my first ever PR to Rust was actually rejected based on a procedural issue. Now I'm on its core team. It might take you a while, but if you keep at it, I'm sure you'll figure it out.
- yakubin 10y ago
  You can probably mail a maintainer or a developer for hints of a project you'd be interested in. Or better you can send a question to the project's mailing list. People there often are very helpful. I've done that. It worked. (I haven't tried contributing to any Google-managed projects though.)
sheerun 10y ago
I'd love something like this for JavaScript
theandrewbailey 10y ago
This seems like a win for Go, at least to me. I wrote a random sentence generator in Python several years ago. A bit later, I wrote my blog using Java EE. Early on, I had an idea: put that generator in Jython, and spit out a random sentence on every request. It's probably the one feature that I was OK to let go, should I switch platforms. Since Oracle has only gotten more evil and Java more stagnant over the years (especially in light of TLS features), I've been thinking of possible alternatives. I've been intrigued by newer compiled languages, and it's come down to either Go or Rust, but I've yet to dig too far into them. I might have a winner.
sandGorgon 10y ago
any chance you guys have tested this with numba? that would be awesome.
teabee89 10y ago
From looking in the code, this is my favorite part so far: https://github.com/google/grumpy/blob/75a35b4b30afb049b9cfff756bfd6ad6f805a5a7/runtime/core.go#L666 https://github.com/google/grumpy/blob/75a35b4b30afb049b9cfff... making python threads as lightweight as goroutines must be a breeze!
vegabook 10y ago
2.7 haha. First Tensorflow, now Grumpy. I can't stop laughing as all the 3.x zealots try to squirm out of this. Talk about awkward. yes yes I know tf now "does" 3 but we all know what Google really cares about. "I'd like to support 3.x at some point" - trotterdylan. Read: nice to have but when it gets down to brass tacks, 2.7 is where it's at for Google.
BuckRogers 10y ago
This is the best news to come out of the Python space in a very long time. Python on the Go runtime, no GIL, more performance, is exactly what people wanted out of a new Python.
Pxtl 10y ago
I can't help but see this balkanization of Python as a sign that the core language is falling apart. How many interpreters are there now? And how many of them have even close to 100% compatibility with Python 2.7 or 3.N? Guido has lost control of the language, but has he's still officially the BDFL there's no real standardization body. His stubborn view on functional mechanisms have held the language back syntactically, breaking BC with Python 3 without fixing the language's fundamental problems... it really feels like Python is lost in the desert. Which doesn't mean the language is dead, but it's rudderless. I think we were all hopeful when Guido joined Google that we'd see real direction for Python, but that obviously didn't happen. Not that Python is dead, obviously - still lots of great projects are written in Python. But I don't like the language's future.
- skybrian 10y ago
  It's not that different from what's happening with Java (via Android) or Go (with GopherJS, for example), or even C (many, many extensions). A popular language attracts implementors, even if they don't implement the whole thing on their target platform.
- viraptor 10y ago
  Why do you think multiple implementations are a problem? We've got multiple ruby runtimes, multiple basics, multiple JavaScript engines, multiple C compilers, etc. None of those languages are falling apart because of it.
- BuckRogers 10y ago
  GVR, the PSF and the core dev team overestimated their influence. They still truly believe the majority will come around to Python3. I agree with your sentiments and balkanization is the right word. Guido won't even read these comments. He thinks it's all some unjust slander and nonsense that will be forgotten in 3 years. :) It is a good time to jump off the Python train in general, and I say that as someone invested in Python who loves it. If possible I'd recommend people reach for Go or Elixir depending on their needs or requirements. I will admit I'm a little shocked how much of a failure Python3 adoption has been. I think if it had been Grumpy from the start it would've been a huge success. This is exactly what people want and Google should be commended for sharing this. Here's to hoping Grumpy takes on a life of its own and is the new de facto Python.
acd 10y ago
May I suggest GoPy as name
- vorg 10y ago
  Maybe you don't like Grumpy as a name. I think it was named after NumPy (pr. ˈnʌmpaɪ) so I guess it's pronounced ˈgrʌmpaɪ rather than ˈgrʌmpi: Around 2011 the beta releases of the static typing additions to Apache Groovy were called "grumpy" but the name was dropped after objections from the Grails crowd. I think the quality of the product is more important than the name, so Grumpy should do OK regardless if it's built and maintained properly.
philip1209 10y ago
With click.py, this could make amazing command line utilities
drej 10y ago
It's all about performance and concurrency here, but I'm quite happy about this for another reason: static compilation. If I needed really performant code, I'd write it in Go/Cython/[insert your fav language] in the first place. But if I have some existing code that I just want to run without worrying about the Python runtime being installed/correct, this is a good solution. (Sure, none of my code will work, because it's all Python3 and usually uses C/asm, but it's still early and I'm hopefull.)
tiffanyh 10y ago
>> "The front-end server that drives youtube.com and YouTube’s APIs is primarily written in Python, and it serves millions of requests per second! YouTube’s front-end runs on CPython 2.7, so we’ve put a ton of work into improving the runtime and adapting our application to work optimally within it." So has YouTube already migrated over to using Grumpy (and no longer running python in production)?
bootload 10y ago
"The downside is less development and deployment flexibility, but it offers several advantages. For one, it creates optimisation opportunities at compile time via static program analysis." Optimisation. This is a smart move, hard though. A compiler, written well allows the back end to improve the code. So the whole code base can improve with improved analysis. "The biggest advantage is that interoperability with Go code becomes very powerful and straightforward: Grumpy programs can import Go packages just like Python modules!" Extending Python (youtube codebase) with Go modules. That's interesting.
Sir_Cmpwn 10y ago
God, will Python 2 just die already?
jsmith_dev 10y ago
Win for crypto library inclusion https://twitter.com/jsmith_dev/status/816894517844418561 https://twitter.com/jsmith_dev/status/816894517844418561
MrBra 10y ago
Sometimes I really wonder what Google has against Ruby.
walrus1066 10y ago
Question: better threading performance seems to be the main motivation for this project. Could they not just use multiprocessing instead of threading for the cpu-heavy parts of the YouTube codebase, and threading or asyncio for the io-bound parts of it? Edit: for example, the fib benchmark they cite is cpu-bound. If the python code used multiprocessing, the performance would scale almost linearly with the number of processes.
- webmaven 10y ago
  > Could they not just use multiprocessing instead of threading for the cpu-heavy parts of the YouTube codebase, and threading or asyncio for the io-bound parts of it? Probably, but consider how much bare iron Google has lying around, much of it likely with few cores-per-CPU. Using it efficiently and not accelerating it's obsolescence is probably a priority for them. BTW, does anyone have data that would suggest how long it does take an org at the scale of an Amazon, Google, or Facebook to entirely replace their HW? I assume that it isn't only through attrition, and that Google for example currently has no servers running that date back to Y2k, but I have no idea what the "half-life" of a server is at their scale.
codebungl 10y ago
+1 to cooperative multitasking
statsmatscats 10y ago
Will you continue maintenance on this indefinitely (barring orders from higher up) or is this meant for a one shot translate and dump?
kristianp 10y ago
Insightful discussion in the comments here about performance and JITs https://lwn.net/Articles/710634/ https://lwn.net/Articles/710634/
webmaven 10y ago
The code initially landed on Github 16 days ago[0]. How long was it developed internally before that? [0] https://github.com/google/grumpy/commit/f60ee257db9d7996e3f8da148260e113516e3a90 https://github.com/google/grumpy/commit/f60ee257db9d7996e3f8...
deleted 10y ago
[deleted]

1411 points by trotterdylan 10y ago

brian_herman 10y ago

No C extensions boo.

MBCook 10y ago
But isn't that really what lets them do this? In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions. As soon as something can see into the guts of the interpreter you have to maintain compatibility which is a pain/waste. Worse than that is that view wasn't designed for multi-threading which is why the GIL exists. The C extensions were t designed to be multi-threaded because that wasn't a thing in Python so they're not safe. You either have to drop them, define a new interface layer that would be safe, or I suppose somehow sandbox their little view of the world but keep it coherent between threads. If you have a codebase where you can make the choice to drop C extensions and you're trying to accelerate Python it seems like a very smart choice.
- guitarbill 10y ago
  The irony is that some C extensions exist because they have better performance than Python. On the other hand, there are many C extensions which are just interop/wrappers for existing C code. I think no language is naive enough to think they can get away without C interop. C extensions are actually pretty nice because you can wrap the C code into idiomatic Python. With ctypes, you have to e.g. maintain two structure definitions, which is very problematic for some codebases. > In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions. AFAIK, many projects initially struggle to even achieve performance parity with CPython. The GIL isn't evil, it's a simple solution to a hard problem. But at this point, a complex solution to a hard problem is better if it's faster, and some people care more about speed than C interop (and vice versa). What would be awesome is a Python runtime that could run with fine-grain locking until a C extension is loaded, and then continue with coarse locking. But the problem is still there's no upgrade path for C interop. The only thing I can think of is using type annotations and something like Cython's cdef to write Python-implementation-independent C interop that doesn't suck as bad as ctypes. Then the Python rumtime could also lock the arguments at a very fine level while the function is being called.

deleted 10y ago

[deleted]

naasking 10y ago

I wonder why the Grumpy Fibonacci is so much slower than CPython for 1 thread. Seems weird given Grumpy is compiled.

sgift 10y ago
From the post it looks like it is optimized for multithreading from the start, so they possibly used the multithreaded version with thread count = 1. When you care for parallel workloads its not so important if your single-threaded code is the fastest.
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
machbio 10y ago
Guido insists that single threaded performance of Python has the highest priority - http://www.artima.com/weblogs/viewpost.jsp?thread=214235 http://www.artima.com/weblogs/viewpost.jsp?thread=214235 I am not sure, any implementation of Python will beat the single threaded performance of Cpython..
- Veedrac 10y ago
  CPython is still a naïve interpreter. PyPy beats it in most cases, sometimes by a lot. Of course, PyPy still has a GIL for the same reasons CPython does.
- dekhn 10y ago
  IIRC, IronPython single-threaded performance was competitive with CPython.
- throwaway91111 10y ago
  Multiple ones already do.
- dom0 10y ago

Twirrim 10y ago

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

matt_wulfeck 10y ago
That's a question I have as well. I'm guessing they have a very massive cpython codebase, and the trade-off was worth it.
- MBCook 10y ago
  I'd assume that's true, but they also didn't know how well this would work. A 'little' project to make Grumpy is probably much easier to sell to management/throw away if to slow than "Let's rewrite all our code in language X to see if it goes faster".
greenhatman 10y ago
Maybe there is too much to rewrite.
ericfrederich 10y ago
It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.
- martincmartin 10y ago
  I think he's talking about moving their Python codebase to Go, so they don't have to run Python anymore.
  - ericfrederich 10y ago
    You're probably right. He seemed to be questioning the translator so I thought he was suggesting replacing CPython piece by piece with a Go implementation.

oblio 10y ago

It seems interesting but my concern is that it's just another Unladen Swallow.

the_duke 10y ago
Well, if Google is pushing it and actually using it internally... They also have the benefit of being able to push features into the Go core that they might need for this.
- ProAm 10y ago
  > Well, if Google is pushing it and actually using it internally... It will be silently abandoned in 18 months....
  - ehsankia 10y ago
    Not if it turns out to be usable and have a significant impact on the performance. You need to realize that at the scale Youtube works, even a small performance boost translates to huge savings in server cost. So the worst case here is if they never improve this past their own needs (which is a pretty limited subset). But if it's successful for them, and it's opensource, I could very easily see other people who run heavy stuff on Python contributing to it and helping it grow. That's the thing with opensource, even if Google doesn't actively work on it, others can (if it has actual value and is useful to people).
- vorg 10y ago
  Perhaps Golang will get a VM or something similar to enable stuff like exec/eval to be implemented with rapid response times in Grumpy. The only way I've ever seen a REPL implemented for Go is by recompiling the source code via a call to "os/exec".Command for each statement or declaration entered, which gives a 1-second delay on many computers.
- tedmielczarek 10y ago
  Google has pretty specific needs, so this project may be very successful internally even if it gets zero traction externally. A lot of big software companies do this nowadays. Google and Facebook both have a lot of purpose-built software, some of which gets released as open source, that meets their needs well but is hard to use for other purposes. I guess it's still strictly better than them not open-sourcing the code, but it's definitely an existence proof that just making something open source doesn't make magic happen.

nodivbyzero 10y ago

Facebook: PHP -> HipHop Google: Python -> Go

scrollaway 10y ago

That looks like a super interesting runtime. Seems to target 2.7 only, I hope they're open to supporting 3.x as well.

MBCook 10y ago
I wonder if they intend to maintain it long term or if they're just going to use it as a bridge while they rewrite things natively in Go.
- ericfrederich 10y ago
  Seems like a huge effort for the interim... unless they really need interoperability between Python and Go during the switch.
trotterdylan 10y ago
I'd like to support 3.x at some point. See https://github.com/google/grumpy/issues/1 https://github.com/google/grumpy/issues/1
- statsmatscats 10y ago
  Have you thought about using type hints to help type inference? Some work on that is discussed here. I would love a dropbox google colab (though also targeting 3.x :) ) https://github.com/python/mypy/issues/1862 https://github.com/python/mypy/issues/1862
  - trotterdylan 10y ago
    Yes, leveraging type hints for optimization purposes is a long term goal. Thanks for pointing me to that issue, I'll keep an eye on it. One of the goals of open sourcing was to get feedback and work with outside folks so I'm definitely open to collaboration!
    - statsmatscats 10y ago

aibottle 10y ago

This actually pretty great. Even if you will never run your code in grumpy, it underlines the status of Python. Having large corps investing in Python will help pulling in new talent into the Python environment also it strengthens the ecosystem of Python.

the_duke 10y ago
Actually, I would interpret this as "CPython is broken for scalability. It's not worth trying to fork/fix it, we'd rather just roll our own runtime". Not exactly an endorsement. Sadly, the code was just dumped into a new Git repo, so no way to tell how many people contributed internally so far.
- ericfrederich 10y ago
  I agree and disagree ... on one side this may signal a shift away from Python. Something to help the migration effort and the ultimate goal is to write everything in Go. ... but then I don't see Python going anywhere anytime soon. Didn't Microsoft just start a project to get Python's runtime to use CoreCLR's JIT? There was an article, can't find it now, about an upcoming Python renaissance saying there may be an influx of new interpreters. There's PyPy, Microsoft's CoreCLR thing, now this, etc. It seems people really want to program in Python so there is an effort to make it faster. *edit: found the article: https://lwn.net/Articles/691070/ https://lwn.net/Articles/691070/
  - the_duke 10y ago
    Devs love Python and will continue to do so, no question there. I'm just afraid the surge of different compilers and interpreters will bring up plenty of issues in the medium term. There is no formal spec of Python like there is for, eg, Javascript. (which is of course driven by multiple, VERY engaged adopters). How long until subtle and not so subtle differences creep in between different implementations, leading to incompatabilities and a continuous fragmentation of the ecosystem?
  - dagss 10y ago

ohitsdom 10y ago

Very interesting project and technical solution. Is this in use at Google? The described example problem is Youtube with millions of requests per second, but the post doesn't say if it Grumpy was put in production (and what performance gains were then achieved).

Zikes 10y ago
I'm thinking it isn't in production yet, based on wording like "we're excited about the prospects" and "although it's still alpha software".
hacker_9 10y ago
Yeah without performance metrics I'm not sure what to think about this. I'm surprised they even used Python to begin with for their front-end server, for such high volumes of traffic.
trotterdylan 10y ago
As has been said by others, Grumpy is not used in production at Google currently. There's still a lot of work to do -- especially on the standard library -- to support large real world codebases.

gamesbrainiac 10y ago

In case anyone wanted the github repo: https://github.com/google/grumpy https://github.com/google/grumpy

ericfrederich 10y ago

Python needs a new runtime. This talk shows how bad of shape it's really in. https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seLRqszBqVDF342RMlCWgOTm6q&index=11 https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seL... Basically, the language doesn't have a "spec" per-se. The language is whatever the defacto CPython implementation happens to do within it's giant eval loop. Another great talk about CPython internals: http://pgbovine.net/cpython-internals.htm http://pgbovine.net/cpython-internals.htm

dijit 10y ago
cpython is the reference implementation, so it makes sense that it's; A) Not well optimised. B) Touting features before the spec/standard. EDIT: people really dislike that I said this, and I'm having trouble finding my original citation- it was on one of the many python books I own. Most likely "Learn Python The Hard Way" but I'll dig out the exact chapter where they compare pypy to cpython and mention that because cpython is the reference implementation it values code clarity over performance optimisation.
- the_duke 10y ago
  It makes sense that the official, primary and by far most popular implementation of one of the most used languages in existance is not well optimized? (edit: I'm just being polemic about your statement here. CPython is reasonably optimized within the constraints it currently has).
  - nodja 10y ago
    It only makes sense because it's python. A language where style is part of the syntax, and readability is one of the things that many libraries focus on , the so called being "pythonic". It makes sense that the reference implementation mirrors the same patterns than the language itself.
    - the_duke 10y ago
      Someone else already posted this link: https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seLRqszBqVDF342RMlCWgOTm6q&index=11 https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seL... It explains why CPython can't improve on many things.

sfifs 10y ago

The main reason I moved from coding in Python to Go as my main language many years back is because concurrency was such a pain in standard Python (the other was compile time error checking). It's interesting to see the same pain has now made caused the runtime itself to be implemented in Go. It's a pity C extensions (often used in scientific computing) are not supported but Go does have support via CGO, so maybe some approach can be worked out to access C routines in the future.

vram22 10y ago
Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? Is it to do with Go's being a GC'd language? I would have thought that should not be a reason per se, since Python also has GC, but has plenty of extensions written in C. But I'm not a language internals expert. Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libraries somewhat easily (not sure about all cases or how far the ease goes).
- moosingin3space 10y ago
  It's because Go uses a different stack structure, called "segmented stacks", in order to enable cheap goroutines. Basically, Go stacks start tiny (8 KiB, as opposed to much larger C stacks), then it grows them in small segments. Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches, and ordinary C function calls conflict with this event loop.
  - vram22 10y ago
    >Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches Interesting, didn't know this (that Go code runs in an event loop). Is the reason something to do with goroutines and channels? something like, a routine gets info that data is available for it to read (on a channel, sent by another goroutine), via an event it receives? Also, can you explain this point: "which enables excellent I/O performance without kernel context-switches" ?

the_duke 10y ago

Haha, now that's what I call an interesting project. The biggest surprise for me is that the Go runtime would be a good fit for Python, performance wise, considering the very different object and dispatch model. The post also mentions runtime reflection, which used to be painfully slow last time I used it. (Go 1.5, i think). Has this improved in the latest releases?

jerf 10y ago
A huge part of the reason Python (and Perl, PHP, etc. in their original interpreters) is so slow is that it is like running a Go program that does everything through runtime reflection, even just to add two integers. If your code is already in Python, this level of performance is apparently not a problem for you. If you know Go or are willing to learn about Go and reflection, you can learn a lot about how dynamic languages work under the hood by implementing: func Add(interface{}, interface{}) interface{} { ... } using the reflect module to accept all types of numbers, including for a bit of extra fun the math.Big* number types, and returning upgraded numbers as appropriate, or panicking on types you can't Add with. That's not all there is to writing a dynamic language interpreter, but I'd say you can learn the core idea this way, shorn away from a lot of accidental complexity and with a lot of the grunt work plumbing of setting up (type, value) pairs already done for you.
trotterdylan 10y ago
Reflection in Go is used minimally in Grumpy because it is slow. Currently importing Go packages into Python code is accomplished via the reflect module, but I think this will have to change to make such integration useful.

0xdeadbeefbabe 10y ago

> Once we started going down the rabbit hole, Go seemed like an obvious choice... Good? It's funny that a rabbit hole is the place you go to make obvious choices. This culture must seem strange to outsiders.

epanastasi 10y ago

As someone who works on both python and go day to day, I find this to be quite interesting. Just tried this out on a reasonably complex project to see what it outputs. Looks like it only handles individual files and not any python imports in those files. So for now you have to manually convert each file in the project and put them into the correct location within the build/src/grumpy/lib directory to get your dependencies imported. Unless I missed something somewhere.. The documentation is a bit sparse. Overall I think the project has a lot of potential and I'm hoping it continues to be actively developed to smooth out some of the rough edges.

ericfrederich 10y ago
I hope this is a well thought out solution that can evolve into something great... and not just something built for a single purpose. I question the transpiler. I think I'd much rather prefer a solution like Jython.
- Floegipoky 10y ago
  I'm confused because Jython runs on the JVM, but Go is a compiled language. Can you clarify?
  - bb88 10y ago
    Jython is a python interpreter written in Java. Grumpy is a python transpiler that converts python to navtive go object code. Edited to add: The difference is that Jython doesn't covert python to JVM bytecode.
    - weberc2 10y ago
      What's the advantage of writing an interpreter? Go already has an excellent runtime (scheduler, GC, etc)--why should this project reimplement it?
      - solidsnack9000 10y ago

rpedela 10y ago

How does Grumpy handle Decimal in Python? As far as I am aware there isn't an equivalent in Go.

brettcannon 10y ago
The decimal module in Python 2.7 is implemented in pure Python: https://github.com/python/cpython/blob/2.7/Lib/decimal.py https://github.com/python/cpython/blob/2.7/Lib/decimal.py (in Python 3 there is an accelerated extension module that's used when available, falling back on the pure Python version otherwise).
- webmaven 10y ago
  So, does Decimal work with Grumpy (you're implying it does)? For that matter, does Grumpy match CPython's Float behavior exactly? ie.: >>> 2.2 + 3.1 5.300000000000001 >>>
  - brettcannon 10y ago
    I'm implying that they can get the decimal module into Grumpy by implementing enough of Python to support the module (i.e. they don't need to implement the decimal module from scratch because the module isn't implemented only in C).
    - webmaven 10y ago
      Got it. Thanks for the clarification, Brett.

matthewrudy 10y ago

I wonder if future work might support C Extensions in the same was as the jruby truffle / graal implementation plans to do. Here's a great article from a couple of years ago by Chris Seaton on this topic. http://chrisseaton.com/rubytruffle/cext/ http://chrisseaton.com/rubytruffle/cext/

deleted 10y ago

[deleted]

zedpm 10y ago

I would love to see this support Python 3.5+, specifically for asyncio. Concurrency via threads in Python is far less appealing.

bigato 10y ago

I see some parallel with the work made to automatically convert Go compiler C code to Go code. Strategically wise, I'd rather have the software do the transpilation to the new language and then after extensive testing, ditch the old python codebase. But they probably don't agree with my preference of Go over Python.

quotemstr 10y ago

I'm a bit disappointed that the blog post doesn't explain why the authors didn't choose PyPy instead.

ot 10y ago
The objective of the authors is to efficiently exploit multi-core parallelism. PyPy still has a GIL. They have been doing some experiments with transactional memory, but performance is quite bad.
theandrewbailey 10y ago
> These efforts have borne a lot of fruit over the years, but we always run up against the same issue: it's very difficult to make concurrent workloads perform well on CPython. > To solve this problem, we investigated a number of other Python runtimes. Each had trade-offs and none solved the concurrency problem without introducing other issues.

jerf 10y ago

For those who are interested, I've used grumpy to compile the following Python code and placed it at https://play.golang.org/p/YP1SP7WsdR https://play.golang.org/p/YP1SP7WsdR . (Note the playground can't run this, it just had convenient formatting support for Go; the generated source wasn't 100% gofmt compliant.) class Test(object): def __init__(self, value): self.value = value def method(self): print(self.value) class Test2(Test): pass t = Test("hello") t.method() Pythonistas, note I had to have "class Test(object):" and not just "class Test:". The former compiled successfully into a Go program but that program then failed at runtime with "TypeError: class must have base classes".

trotterdylan 10y ago
Thanks for trying it out! Yeah, Grumpy does not currently support old-style classes. Since all of our code internally requires new-style classes, this was not a high priority feature. It is something that we'll get to.
- jerf 10y ago
  I did not mean that as a complaint against your very young codebase, I meant that as a defense against Python people complaining about my code. :)
  - kstrauser 10y ago
    I don't think they'd do that. All Python 2 code I've seen uses `class Foo(object)`, at least since 2.2 came out.
Animats 10y ago
That's fascinating. It's creating run-time data structures similar to CPython's for data, and manipulating them with very general code. There seems to be a type comparable to Python's internal CObject, and it's used for most (all?) data. It's not generating Go that looks anything like human-written Go. There's no sign of type inference, although it's hard to tell from such a simple example. It's a lot like a Python run time environment, where everything is a CObject. Still, once you can do that, you can start optimizing, such as inferring that something is an integer and using ordinary Go arithmetic types. All that stuff with "switch" seems to be to handle Python exceptions in a language that doesn't have exceptions. Maybe later, analysis can tell that some function can't raise an exception, and translated calls for such functions can be simpler.

thedjinn 10y ago

I'm curious to why they started this project when there already is low hanging fruit that can speed up Python (e.g. pypy). What makes this better other than to satisfy the inner go-fanboy?

zardeh 10y ago
Pypy still has a GIL. GIL-free seems to be one of the explicit goals of this project.
- poooogles 10y ago
  >Pypy still has a GIL. Not strictly true, http://doc.pypy.org/en/latest/stm.html http://doc.pypy.org/en/latest/stm.html. In general for the main project however, this is true.
  - zardeh 10y ago
    Pypy still has a gil. Pypy is experimenting with non-GILed versions, and in fact there are people actively working to remove the GIL from pypy. But the same can be said of cpython (the gilectomy), and yet I don't think that you'd say "not strictly true" to "cpython still has a GIL".
nemith 10y ago
GIL still exists in pypy
alrs 10y ago
Because they want to get off of Python completely, and forever. They're not looking for speed, they're looking to ditch the liability of supporting Python and its ecosystem.
- JPKab 10y ago
  Yeah, they didn't at all say that.

Animats 10y ago

- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. - It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it. - If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?

ionforce 10y ago
What does "hard-code compiler" mean?
- zardeh 10y ago
  It seems to be that it pesudo-transpiles python to go and compiles that down using a normal go toolchain.
  - omaranto 10y ago
    What's the difference between transpiling and pesudo-transpiling? (Even if you meant pseudo-transpiling, I still don't know what the difference between that and transpiling is.)
    - zardeh 10y ago
      That was a typo. I meant pseudo. I don't actually know that there is any difference in this case. I made that statement hastily. It transpiles, nothing pseudo about it.
- eva1984 10y ago
  Transpilers with no runtime.
10y ago

ericfrederich 10y ago

There was also this effort: https://blog.heroku.com/see_python_see_python_go_go_python_go https://blog.heroku.com/see_python_see_python_go_go_python_g...

sea6ear 10y ago

Reminds me of the (old I guess?) ShedSkin project to automatically translate Python to C++. I suppose the easy concurrency and ability to inter-operate with Go libraries is really the driver for Go over that.

Arkanosis 10y ago
Shed Skin is still living at https://shedskin.github.io/ https://shedskin.github.io/ , but hasn't changed much since the last release in 2013.

ot 10y ago

There's no mention of whether Grumpy passes the CPython test suite. Until it doesn't, it's not a Python runtime, it's a compiler for a language with Python-like syntax. Compilers like this, from almost-Python to say C/C++, have existed for a while: Cython, Shedskin, Nuitka are some examples.

trotterdylan 10y ago
It does not, yet. The standard library is very incomplete at this point.
- gigatexal 10y ago
  experimental ~= alpha, give the team a break
  - weberc2 10y ago
    You're responding to the principle developer. :)
    - gigatexal 10y ago
      I know, but i figure people will read his comments first and then see this.
- ot 10y ago
  I'm not particularly concerned about the standard library, but about the more dynamic features of the language. Not just exec/eval, but for example the complex dispatching logic behind magic methods, especially the various __getattr__, __getattribute__. They are where alternative runtime implementations usually stumble, as if they were an intrinsic bottleneck to Python runtime performance.
  - trotterdylan 10y ago
    exec/eval do not work, but all of the other dispatching and magic methods are supposed to work exactly like CPython (and if they don't, it's a bug/not yet implemented).

gigatexal 10y ago

on a related note: given my affection for python and having, until just now, no idea that YouTube runs python (albeit 2.7) at Google scale (a language not known for being able to scale very well given the GIL and such)-- I now more than ever would love to work on something like that. Welp, back to learning and hacking with python.

deleted 10y ago

[deleted]

stiff 10y ago

It's actually a transpiler written in Python that generates Go code.

kodablah 10y ago
Nice. I've been toying with doing similar for the JVM [0]. I think people are coming to realize that while Go is not a great/powerful language (IMO), the runtime is great. I would not be surprised if more and more people start targetting Go if they want a GC and cross platform static compilation w/out LLVM complications and get a nice stdlib for free. 0 - https://github.com/cretz/goahead https://github.com/cretz/goahead
- bsaul 10y ago
  First to write a language that is basically go + generics wins

cbhl 10y ago

The blog post says that YouTube is the inspiration for this runtime, but does YouTube run on Grumpy yet?

trotterdylan 10y ago
YouTube does not run on Grumpy. There is a lot of work left to do before Grumpy can run a large existing codebase.

zigzigzag 10y ago

How does this compare to Jython? The blog post says they looked at alternative runtimes but didn't like that they had tradeoffs (implying that Grumpy has no tradeoffs??). But Jython has been around for quite a while and also has no GIL: http://www.jython.org/jythonbook/en/1.0/Concurrency.html http://www.jython.org/jythonbook/en/1.0/Concurrency.html It can also handle Python's dynamic aspects.

Twirrim 10y ago
Every time I've tried to use Jython, I've found it won't work with pre-existing code all that well. In part it has exposed CPython "implementation quirks" that people were wittingly or otherwise taking advantage of. In other cases there doesn't seem to be obvious reasons for the differences and has required special-casing the python code to handle it. It has been great with code written from scratch, specifically for it.
- zigzigzag 10y ago
  That sounds like the kind of issue you'd have with any reimplementation of CPython though. It sounds like Grumpy doesn't support quite a few things, so I still wonder how they compare and why developing a new runtime was considered easier than reusing Jython.
  - toyg 10y ago
    There might be a strategic element: Jython development has been erratic throughout the years.
    - zigzigzag 10y ago
      Google's own projects have been developed erratically throughout the years. They could have simply forked it.

BuckRogers 10y ago

Is this the final nail in the coffin for Python3? Seems like it. Who would use 3 if you could have CPython2 for existing code and write new code in Grumpy Python? This is the dream language for me. Python on the Go runtime.

int_19h 10y ago
Everyone who likes the numerous new language features, or new syntax for old features, in Python 3?
- BuckRogers 10y ago
  Those can be built into Grumpy if anyone cares to do so. You don't need Python3 for that stuff. Someone, one developer, recently released a Python 2.8 that backported almost every new Python3 feature. I'm hoping it becomes a permanent fork of Python2 that uses the Go runtime. That would really be great and exactly what they've got now.
  - int_19h 10y ago
    If you backport every new Python3 feature to Python2, it becomes Python3, by definition.
    - dismantlethesun 10y ago
      It could become Python 2.8, wherein the standard library has both the legacy versions and the new versions, and any cross-compatible syntax is allowed. This leaves us with more than 1 way to do things, like meta-class declaration. There's a lot in Python 3 where changes were made to the syntax for 'clarity', but those worts weren't removed for any technical reason but because of the thought that since backwards compatability was being broken anyways, then we might as well get the most bang for our buck.
      - int_19h 10y ago
        > It could become Python 2.8, wherein the standard library has both the legacy versions and the new versions, and any cross-compatible syntax is allowed. So two `builtin` modules, then? What if someone does `sys.modules['builtin']`? Or any other kind of explicit string-based lookup? How does pickle figure out which types to instantiate? There'd be a lot of types with same qualified names but different implementations with this approach... It feels like the only way this would work reliably, is if you completely isolate the Py2 and Py3 universes. So if you e.g. pickle from Py2 code, it only looks at modules and types that Py2 universe knows, and vice versa. But then what happens when code using the old library interacts with the new one (e.g. tries to pass objects around)? If that is prohibited, then you effectively still have two different languages, just with a single shared implementation - but no ability to gradually replace bits and pieces of code, for example, which would seem to be the biggest motivation for such a thing.

waitingkuo 10y ago

Will Grumpy be a good fit for go's scientific packages such like gonum?

trotterdylan 10y ago
I don't know. I hope so. I think this is an area where Go can succeed and integration with existing Python libraries could be useful.
- waitingkuo 10y ago
  This is amazing work. Look forward to the integration between go & python

curtis 10y ago

One of my complaints about Go is that it seems to be designed with the (mistaken, in my opinion) notion that what we really need is just a better C. One way to leverage C's widespread availability and high-performance while side-stepping some of its deficiencies was simply to use it as a target for a different language. The Cfront C++ compiler which generated C code is probably the most famous example, but I recall that there were many others. Maybe Go, like C, will make a fruitful target for other language implementations.

coldtea 10y ago
>One of my complaints about Go is that it seems to be designed with the (mistaken, in my opinion) notion that what we really need is just a better C. It was designed with the notion that all some people really need is just a better C. For other people there's Swift, Rust, etc.
- curtis 10y ago
  For better or worse, I think Go sucked up a lot of the metaphorical oxygen for similar languages. Swift and Rust are in different niches and don't compete head to head with Go, so this isn't a real problem for them. What I've long wanted is a better C++, not a better C. It would be hard for any such language to succeed today, because it will have to compete with Go, and Go has a fairly mature toolchain and widespread adoption and a lot of mindshare. D literally is designed to be a better C++, and it hasn't really been a big success. Maybe that's because there really wasn't as big a demand for a better C++ as there was for a better C. On the other hand it may have been because Go sucked up a lot of the oxygen that D was going to need to succeed, and maybe that was because Go was a product of Google and D wasn't. (If this was my primary point, I'd make a more nuanced argument, though.) I'm now wondering if the best way to get to a better C++ might be by piggy-backing on the Go toolchain. To get back to my original point, I'm wondering if Go might in fact be a good target for all sorts of better (for some definition) programming languages.
  - wrsh07 10y ago
    For my money, a better C++ is in development: it's called C++20.

agentgt 10y ago

* I would love to know how big the codebase is. * It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces. * Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)? Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)? Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7). With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C?

trotterdylan 10y ago
> I would love to know how big the codebase is. Sorry, can't be very specific, but rewriting all the frontend code would take a lot more effort than writing a new Python runtime :) > It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces. We do iteratively rewrite components as well. We are pursuing multiple strategies. > Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)? Most async operations are performed out-of-process by other servers. > Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)? I'd love to support async and await at some point. > Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7). We did also do an evaluation of Jython but there were a number of technical issues that made it unsuitable for our codebase and workload. One such example is this longstanding issue: http://bugs.jython.org/issue527524 http://bugs.jython.org/issue527524. I just noticed the very recent update on that thread that implemented the workaround outlined in 2010 by Jim Baker. We tried that workaround and found we got a huge performance hit on affected code. There were a few other general performance problems as well but I can't recall all the details. Please note I'm not at all bashing Jython, I think it's a great project with a sound design, it just wasn't right for us. > With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C? You can interface with Go code directly, e.g. from the blog post: from __go__.net.http import ListenAndServe, RedirectHandler handler = RedirectHandler('http://github.com/google/grumpy', 303) ListenAndServe('127.0.0.1:8080', handler)

agounaris 10y ago

Whatever the haters say here, this is really cool!

weberc2 10y ago

I tried a simple `http.Get()` example, but the resultant `Response` object appears to not allow access to any of the struct fields as attributes. How does one access a Go object's fields from Python? For example, I want to `io.Copy(os.Stdout, rsp.Body)`.

trotterdylan 10y ago
Unfortunately the native interface is still pretty immature so I can't guarantee things will work as they should. In this case, I think the problem is that you have a *Response object which is not itself a struct. I've rewritten this part of the code a couple times and I think this functionality was lost. I've filed: https://github.com/google/grumpy/issues/13 https://github.com/google/grumpy/issues/13
- trotterdylan 10y ago
  Whoops, just noticed you (or somebody) had already filed it: https://github.com/google/grumpy/issues/12 https://github.com/google/grumpy/issues/12
  - weberc2 10y ago
    Yeah, that was me. I should have updated my post here accordingly. Apologies for the inconvenience!
    - trotterdylan 10y ago
      It's all good. Thanks for filing the issue. I'll get that fixed.

senthilnayagam 10y ago

wish there was one for ruby as well

Antwan 10y ago

"So we asked ourselves a crazy question: What if we were to implement an alternative ?" As always with Google... What if you humbly contribute to open source projects rather than creating new stuffs, labelled with your own brand, controlled by your own engineers, and with your own design choices, however good they may be ?

Sphax 10y ago
How entitled can you be ? Why are they required to do anything of the sort ?
tedmielczarek 10y ago
They tried that with CPython, it didn't work: https://www.python.org/dev/peps/pep-3146/ https://www.python.org/dev/peps/pep-3146/
- teddyh 10y ago
  But it was no fault of Python: https://qinsb.blogspot.nl/2011/03/unladen-swallow-retrospective.html https://qinsb.blogspot.nl/2011/03/unladen-swallow-retrospect...
  - tedmielczarek 10y ago
    Interesting, I hadn't seen that post! I wonder if there are any posts that elaborate on the internal Google bits more. I'd love to know what "they found other ways to solve their performance problems" wound up meaning in practice.
    - webmaven 10y ago
      > I'd love to know what "they found other ways to solve their performance problems" wound up meaning in practice. Me too, especially as this was already several years after the YouTube acquisition. Also interesting to note that potential internal customers were put off by having to upgrade to 2.6.1 in order to use unladen-swallow (presumably they got over that reticence at some point, as 2.7 is now standard).

jamestimmins 10y ago

Does anyone know if there is an easy way to get involved in open source projects like this? The readme mentions adding PRs, but as someone that doesn't have much experience working with open source projects like this, I don't even know where to begin. It sounds like an incredible learning opportunity though.

steveklabnik 10y ago
The first stop is usually a CONTRIBUTING.md Here's theirs https://github.com/google/grumpy/blob/master/CONTRIBUTING.md https://github.com/google/grumpy/blob/master/CONTRIBUTING.md There's not a ton there. The next place I'd go is open issues: https://github.com/google/grumpy/issues https://github.com/google/grumpy/issues It looks like at the moment they're mostly random bug reports from people who have tried Grumpy since this announcement, rather than ones filed by people working on Grumpy since before it was made public. So that's a bit trickier. The last place I'd look is then, the README: https://github.com/google/grumpy https://github.com/google/grumpy not a ton there about ways they wish to have people contribute. At this point, what I'd do personally is open an issue asking how you can get involved; possibly by improving this documentation on how to get involved! Anyway, that's what I'd do. Hope that helps!
- jamestimmins 10y ago
  That's super helpful; much appreciated!
  - steveklabnik 10y ago
    No worries. Open source can be tricky to get into, but it's largely about just keeping at it: A story I like to tell is how my first ever PR to Rust was actually rejected based on a procedural issue. Now I'm on its core team. It might take you a while, but if you keep at it, I'm sure you'll figure it out.
yakubin 10y ago
You can probably mail a maintainer or a developer for hints of a project you'd be interested in. Or better you can send a question to the project's mailing list. People there often are very helpful. I've done that. It worked. (I haven't tried contributing to any Google-managed projects though.)

sheerun 10y ago

I'd love something like this for JavaScript

theandrewbailey 10y ago

This seems like a win for Go, at least to me. I wrote a random sentence generator in Python several years ago. A bit later, I wrote my blog using Java EE. Early on, I had an idea: put that generator in Jython, and spit out a random sentence on every request. It's probably the one feature that I was OK to let go, should I switch platforms. Since Oracle has only gotten more evil and Java more stagnant over the years (especially in light of TLS features), I've been thinking of possible alternatives. I've been intrigued by newer compiled languages, and it's come down to either Go or Rust, but I've yet to dig too far into them. I might have a winner.

sandGorgon 10y ago

any chance you guys have tested this with numba? that would be awesome.

teabee89 10y ago

From looking in the code, this is my favorite part so far: https://github.com/google/grumpy/blob/75a35b4b30afb049b9cfff756bfd6ad6f805a5a7/runtime/core.go#L666 https://github.com/google/grumpy/blob/75a35b4b30afb049b9cfff... making python threads as lightweight as goroutines must be a breeze!

vegabook 10y ago

2.7 haha. First Tensorflow, now Grumpy. I can't stop laughing as all the 3.x zealots try to squirm out of this. Talk about awkward. yes yes I know tf now "does" 3 but we all know what Google really cares about. "I'd like to support 3.x at some point" - trotterdylan. Read: nice to have but when it gets down to brass tacks, 2.7 is where it's at for Google.

BuckRogers 10y ago

This is the best news to come out of the Python space in a very long time. Python on the Go runtime, no GIL, more performance, is exactly what people wanted out of a new Python.

Pxtl 10y ago

I can't help but see this balkanization of Python as a sign that the core language is falling apart. How many interpreters are there now? And how many of them have even close to 100% compatibility with Python 2.7 or 3.N? Guido has lost control of the language, but has he's still officially the BDFL there's no real standardization body. His stubborn view on functional mechanisms have held the language back syntactically, breaking BC with Python 3 without fixing the language's fundamental problems... it really feels like Python is lost in the desert. Which doesn't mean the language is dead, but it's rudderless. I think we were all hopeful when Guido joined Google that we'd see real direction for Python, but that obviously didn't happen. Not that Python is dead, obviously - still lots of great projects are written in Python. But I don't like the language's future.

skybrian 10y ago
It's not that different from what's happening with Java (via Android) or Go (with GopherJS, for example), or even C (many, many extensions). A popular language attracts implementors, even if they don't implement the whole thing on their target platform.
viraptor 10y ago
Why do you think multiple implementations are a problem? We've got multiple ruby runtimes, multiple basics, multiple JavaScript engines, multiple C compilers, etc. None of those languages are falling apart because of it.
BuckRogers 10y ago
GVR, the PSF and the core dev team overestimated their influence. They still truly believe the majority will come around to Python3. I agree with your sentiments and balkanization is the right word. Guido won't even read these comments. He thinks it's all some unjust slander and nonsense that will be forgotten in 3 years. :) It is a good time to jump off the Python train in general, and I say that as someone invested in Python who loves it. If possible I'd recommend people reach for Go or Elixir depending on their needs or requirements. I will admit I'm a little shocked how much of a failure Python3 adoption has been. I think if it had been Grumpy from the start it would've been a huge success. This is exactly what people want and Google should be commended for sharing this. Here's to hoping Grumpy takes on a life of its own and is the new de facto Python.

acd 10y ago

May I suggest GoPy as name

vorg 10y ago
Maybe you don't like Grumpy as a name. I think it was named after NumPy (pr. ˈnʌmpaɪ) so I guess it's pronounced ˈgrʌmpaɪ rather than ˈgrʌmpi: Around 2011 the beta releases of the static typing additions to Apache Groovy were called "grumpy" but the name was dropped after objections from the Grails crowd. I think the quality of the product is more important than the name, so Grumpy should do OK regardless if it's built and maintained properly.

philip1209 10y ago

With click.py, this could make amazing command line utilities

drej 10y ago

It's all about performance and concurrency here, but I'm quite happy about this for another reason: static compilation. If I needed really performant code, I'd write it in Go/Cython/[insert your fav language] in the first place. But if I have some existing code that I just want to run without worrying about the Python runtime being installed/correct, this is a good solution. (Sure, none of my code will work, because it's all Python3 and usually uses C/asm, but it's still early and I'm hopefull.)

tiffanyh 10y ago

>> "The front-end server that drives youtube.com and YouTube’s APIs is primarily written in Python, and it serves millions of requests per second! YouTube’s front-end runs on CPython 2.7, so we’ve put a ton of work into improving the runtime and adapting our application to work optimally within it." So has YouTube already migrated over to using Grumpy (and no longer running python in production)?

bootload 10y ago

"The downside is less development and deployment flexibility, but it offers several advantages. For one, it creates optimisation opportunities at compile time via static program analysis." Optimisation. This is a smart move, hard though. A compiler, written well allows the back end to improve the code. So the whole code base can improve with improved analysis. "The biggest advantage is that interoperability with Go code becomes very powerful and straightforward: Grumpy programs can import Go packages just like Python modules!" Extending Python (youtube codebase) with Go modules. That's interesting.

Sir_Cmpwn 10y ago

God, will Python 2 just die already?

jsmith_dev 10y ago

Win for crypto library inclusion https://twitter.com/jsmith_dev/status/816894517844418561 https://twitter.com/jsmith_dev/status/816894517844418561

MrBra 10y ago

Sometimes I really wonder what Google has against Ruby.

walrus1066 10y ago

Question: better threading performance seems to be the main motivation for this project. Could they not just use multiprocessing instead of threading for the cpu-heavy parts of the YouTube codebase, and threading or asyncio for the io-bound parts of it? Edit: for example, the fib benchmark they cite is cpu-bound. If the python code used multiprocessing, the performance would scale almost linearly with the number of processes.

webmaven 10y ago
> Could they not just use multiprocessing instead of threading for the cpu-heavy parts of the YouTube codebase, and threading or asyncio for the io-bound parts of it? Probably, but consider how much bare iron Google has lying around, much of it likely with few cores-per-CPU. Using it efficiently and not accelerating it's obsolescence is probably a priority for them. BTW, does anyone have data that would suggest how long it does take an org at the scale of an Amazon, Google, or Facebook to entirely replace their HW? I assume that it isn't only through attrition, and that Google for example currently has no servers running that date back to Y2k, but I have no idea what the "half-life" of a server is at their scale.

codebungl 10y ago

+1 to cooperative multitasking

statsmatscats 10y ago

Will you continue maintenance on this indefinitely (barring orders from higher up) or is this meant for a one shot translate and dump?

kristianp 10y ago

Insightful discussion in the comments here about performance and JITs https://lwn.net/Articles/710634/ https://lwn.net/Articles/710634/

webmaven 10y ago

The code initially landed on Github 16 days ago[0]. How long was it developed internally before that? [0] https://github.com/google/grumpy/commit/f60ee257db9d7996e3f8da148260e113516e3a90 https://github.com/google/grumpy/commit/f60ee257db9d7996e3f8...

deleted 10y ago

[deleted]