7 ms·

The main reason I moved from coding in Python to Go as my main language many years back is because concurrency was such a pain in standard Python (the other wa

by sfifs 10y ago

The main reason I moved from coding in Python to Go as my main language many years back is because concurrency was such a pain in standard Python (the other was compile time error checking). It's interesting to see the same pain has now made caused the runtime itself to be implemented in Go. It's a pity C extensions (often used in scientific computing) are not supported but Go does have support via CGO, so maybe some approach can be worked out to access C routines in the future.

vram22 10y ago
Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? Is it to do with Go's being a GC'd language? I would have thought that should not be a reason per se, since Python also has GC, but has plenty of extensions written in C. But I'm not a language internals expert. Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libraries somewhat easily (not sure about all cases or how far the ease goes).
- moosingin3space 10y ago
  It's because Go uses a different stack structure, called "segmented stacks", in order to enable cheap goroutines. Basically, Go stacks start tiny (8 KiB, as opposed to much larger C stacks), then it grows them in small segments. Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches, and ordinary C function calls conflict with this event loop.
  - vram22 10y ago
    >Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches Interesting, didn't know this (that Go code runs in an event loop). Is the reason something to do with goroutines and channels? something like, a routine gets info that data is available for it to read (on a channel, sent by another goroutine), via an event it receives? Also, can you explain this point: "which enables excellent I/O performance without kernel context-switches" ?
    - moosingin3space 10y ago
      One of the reasons an asynchronous I/O event loop can be faster than a threaded model is that the CPU spends more of its time in a single userspace thread per core, switching between clients that are ready. A threaded server will incur a kernel-space context switch each time, while an asynchronous loop will keep the processing time in userspace.
    - takeda 10y ago
      Since we also talking about Python. If you ever used AsyncIO you will see that programming in it is a bit different than you usually write code without it. Before you can call any coroutine you first need to start an event loop and schedule something in it. This essentially enables the language to schedule another async function each time you use await. Since Go by default always is async, before your main function is called, it sets up the even loop and then calls your main, which technically is also a coroutine. Your code appears to be sequential, but it is not executed that way.
      - vram22 10y ago
        Interesting ...
  - dsymonds 10y ago
    Segmented stacks in Go went away in Go 1.3 (https://golang.org/doc/go1.3#stacks; https://golang.org/doc/go1.3#stacks; June 2014). The alternate stack structure is indeed one issue. The bigger one is the GC, though; the Go runtime needs to know which pointers it is responsible for freeing, and which are the responsibility of the C code.
    - masklinn 10y ago
      > The bigger one is the GC; the Go runtime needs to know which pointers it is responsible for freeing That is not the bigger issue, and AFAIK already handled for C types. The stack/calling conventions is the reason why cgo is "not go", cgo calls have significantly more overhead than just about every other FFI (the overhead of a cgo call is ~2 orders of magnitude more than a "native" go call, or was around the same time last year, that is you could perform ~100 no-op non-inlined native calls to a do-nothing function by the time you need for a single cgo call to the same).
      - dsymonds 10y ago
        The original question was about why C extensions are not supported by Go. It's not a matter of performance; it's a matter of correctness.
- theatrus2 10y ago
  Go has cgo, which works fine for most purposes of fine; native code interop is not an issue for Go. Grumpy likely doesn't support the C extensions due to time, and complexity of having to actually emulate the GIL since Python does not have fine grained locking for structures. C extensions that work with Python data structures need to first hold the GIL.
- elementry 10y ago
  > Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? It's because Python's C API is inherently non-thread safe. The API lacks passing an interpreter pointer as a parameter (as Lua's API does for example). So Python is forced to use a terrible thread local storage hack involving the Global Interpreter Lock to swap interpreter instances which is insanely inefficient and limits compute-bound programs to a single thread. Python 3.x had a chance to fix the API and do away with the GIL once and for all, but inexplicably they did not. There was a misguided notion that C extensions between 2.x and 3.x could be interoperable.
trotterdylan 10y ago
I'm not philosophically opposed to supporting C extensions. The additional complexity just was not deemed to be warranted since the YouTube frontend doesn't use a lot of C extensions. In principle it's possible to implement something like JyNI (http://jyni.org/ http://jyni.org/) or CPyExt (https://morepypy.blogspot.de/2010/04/using-cpython-extension-modules-with.html https://morepypy.blogspot.de/2010/04/using-cpython-extension...) to bridge the CPython and Grumpy APIs. In practice, marshalling data across the interface can be very expensive.
- scrollaway 10y ago
  Out of curiosity, what C extensions does YouTube use? If this is good enough to run YouTube's python code already it's honestly super impressive. Well done.
  - trotterdylan 10y ago
    To be clear, Grumpy cannot yet run YouTube's Python codebase. There's still a lot of work to do on the standard library. There are a handful of C extensions for JSON, protobufs, etc that YouTube uses, but mostly they're small utility functions written by us to optimize particularly hot code paths.
    - driusan 10y ago
      Is the plan to use grumpc to transpile the code to Go and then work in Go in the future, or is the plan to keep coding in Python and add a grumpy step before deploying? (If the former, you could just update the code to use the standard Go JSON/etc packages..)
      - statsmatscats 10y ago
        I"m also curious about this
      - statsmatscats 10y ago
        I"m also curious about this
lelandbatey 10y ago
I "grew up" on Python, then wrote a whole bunch of Go for my job. Then this past Autumn re-visited Python to implement a networked terminal based game[1]. With what I learned about Go and concurrency, I would say that currently in Python, writing concurrent code is not very hard, and is as close to Go as you can get without actually just writing Go. Now, you may be saying "but Python has the GIL, how can concurrency be easy in Python?" I'd say, you're definitely not wrong that the GIL is a problem, but it's not much of a problem for concurrency. This goes back to the heart of Rob Pike's classic talk, "Concurrency Is Not Parallelism"[2]. To quote Wikipedia: In computer science, concurrency is the decomposability property of a program, algorithm, or problem into order-independent or partially-ordered components or units. In Python, you can pretty easily emulate the conceptual properties of Goroutines and Go channels with Python threads and queues. The problem is that doing this in Python won't net you the performance increases you get with Go. And I believe that is an important distinction. There are plenty of cases where you don't care so much about the performance benefits of parallelism, but you want the conceptual and implementation benefits of concurrency. In closing, concurrency in Python is pretty easy to work with, it just performs very poorly. [1] - https://github.com/lelandbatey/defuse_division https://github.com/lelandbatey/defuse_division [2] - https://blog.golang.org/concurrency-is-not-parallelism https://blog.golang.org/concurrency-is-not-parallelism
- Spiritus 10y ago
  Using Gevent[1] is quite similar to Go when it comes to concurrency in Python. [1] http://sdiehl.github.io/gevent-tutorial/ http://sdiehl.github.io/gevent-tutorial/
- rcarmo 10y ago
  I wrote a little task module that does precisely that and provides you with a go() function and "channels": https://github.com/rcarmo/python-utils/blob/master/taskkit.py https://github.com/rcarmo/python-utils/blob/master/taskkit.p... Obviously, it wasn't amazingly performant. But it did help a lot for doing concurrent stuff, and I've been pondering re-doing it for asyncio.
- nhumrich 10y ago
pjmlp 10y ago
I never saw Python as anything more than a scripting language to portably automate tasks across UNIX and Windows environment, even back in the Zope days. My experience with Tcl teached me to stay away from languages that don't have either a JIT or AOT compiler on their reference implementation.
- jszymborski 10y ago
  I think that that's rather dismissive of a language that runs huge web, scientific, and general purpose applications daily.
  - pjmlp 10y ago
    It might be, but that it is usually a consequence of not knowing any better or making use of existing libraries. Just like people learned 8-bit BASIC and went on to do business applications and games on it. I went Z80 ASM instead. Personally I would only use Python for shell scripting and advise for using Julia instead. Of course, others see it differently.
    - goatlover 10y ago
      Julia's nice, but it's playing catch up to R & Python. Name any statistical algorithm, and R probably has it. Name any scientific field, and there's probably a Python library for it.
    - wolf550e 10y ago
      Some people are very productive in Django and Rails (and the like), in non-performant languages. Many sites never grow to the size where running on the JVM (or AOT-compiled Go) would save you money from fewer servers vs. more coder-hours.
    - JPKab 10y ago
      Julia? Look, I enjoy playing with Julia, but have you actually used it? Because most of the people I meet who turn up their nose at Python and say nice things about Julia haven't ever used Julia. It's got great potential but has some major gaps and warts, and is just plain dog shit slow for certain things compared to SciPy, which, you may not be aware, is basically C and Fortran code wrapped in Python API.

by sfifs 10y ago

vram22 10y ago

Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? Is it to do with Go's being a GC'd language? I would have thought that should not be a reason per se, since Python also has GC, but has plenty of extensions written in C. But I'm not a language internals expert. Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libraries somewhat easily (not sure about all cases or how far the ease goes).

moosingin3space 10y ago
It's because Go uses a different stack structure, called "segmented stacks", in order to enable cheap goroutines. Basically, Go stacks start tiny (8 KiB, as opposed to much larger C stacks), then it grows them in small segments. Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches, and ordinary C function calls conflict with this event loop.
- vram22 10y ago
  >Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches Interesting, didn't know this (that Go code runs in an event loop). Is the reason something to do with goroutines and channels? something like, a routine gets info that data is available for it to read (on a channel, sent by another goroutine), via an event it receives? Also, can you explain this point: "which enables excellent I/O performance without kernel context-switches" ?
  - moosingin3space 10y ago
    One of the reasons an asynchronous I/O event loop can be faster than a threaded model is that the CPU spends more of its time in a single userspace thread per core, switching between clients that are ready. A threaded server will incur a kernel-space context switch each time, while an asynchronous loop will keep the processing time in userspace.
  - takeda 10y ago
    Since we also talking about Python. If you ever used AsyncIO you will see that programming in it is a bit different than you usually write code without it. Before you can call any coroutine you first need to start an event loop and schedule something in it. This essentially enables the language to schedule another async function each time you use await. Since Go by default always is async, before your main function is called, it sets up the even loop and then calls your main, which technically is also a coroutine. Your code appears to be sequential, but it is not executed that way.
    - vram22 10y ago
      Interesting ...
- dsymonds 10y ago
  Segmented stacks in Go went away in Go 1.3 (https://golang.org/doc/go1.3#stacks; https://golang.org/doc/go1.3#stacks; June 2014). The alternate stack structure is indeed one issue. The bigger one is the GC, though; the Go runtime needs to know which pointers it is responsible for freeing, and which are the responsibility of the C code.
  - masklinn 10y ago
    > The bigger one is the GC; the Go runtime needs to know which pointers it is responsible for freeing That is not the bigger issue, and AFAIK already handled for C types. The stack/calling conventions is the reason why cgo is "not go", cgo calls have significantly more overhead than just about every other FFI (the overhead of a cgo call is ~2 orders of magnitude more than a "native" go call, or was around the same time last year, that is you could perform ~100 no-op non-inlined native calls to a do-nothing function by the time you need for a single cgo call to the same).
    - dsymonds 10y ago
      The original question was about why C extensions are not supported by Go. It's not a matter of performance; it's a matter of correctness.
theatrus2 10y ago
Go has cgo, which works fine for most purposes of fine; native code interop is not an issue for Go. Grumpy likely doesn't support the C extensions due to time, and complexity of having to actually emulate the GIL since Python does not have fine grained locking for structures. C extensions that work with Python data structures need to first hold the GIL.
elementry 10y ago
> Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? It's because Python's C API is inherently non-thread safe. The API lacks passing an interpreter pointer as a parameter (as Lua's API does for example). So Python is forced to use a terrible thread local storage hack involving the Global Interpreter Lock to swap interpreter instances which is insanely inefficient and limits compute-bound programs to a single thread. Python 3.x had a chance to fix the API and do away with the GIL once and for all, but inexplicably they did not. There was a misguided notion that C extensions between 2.x and 3.x could be interoperable.

trotterdylan 10y ago

I'm not philosophically opposed to supporting C extensions. The additional complexity just was not deemed to be warranted since the YouTube frontend doesn't use a lot of C extensions. In principle it's possible to implement something like JyNI (http://jyni.org/ http://jyni.org/) or CPyExt (https://morepypy.blogspot.de/2010/04/using-cpython-extension-modules-with.html https://morepypy.blogspot.de/2010/04/using-cpython-extension...) to bridge the CPython and Grumpy APIs. In practice, marshalling data across the interface can be very expensive.

scrollaway 10y ago
Out of curiosity, what C extensions does YouTube use? If this is good enough to run YouTube's python code already it's honestly super impressive. Well done.
- trotterdylan 10y ago
  To be clear, Grumpy cannot yet run YouTube's Python codebase. There's still a lot of work to do on the standard library. There are a handful of C extensions for JSON, protobufs, etc that YouTube uses, but mostly they're small utility functions written by us to optimize particularly hot code paths.
  - driusan 10y ago
    Is the plan to use grumpc to transpile the code to Go and then work in Go in the future, or is the plan to keep coding in Python and add a grumpy step before deploying? (If the former, you could just update the code to use the standard Go JSON/etc packages..)
    - statsmatscats 10y ago
      I"m also curious about this
    - statsmatscats 10y ago
      I"m also curious about this

lelandbatey 10y ago

I "grew up" on Python, then wrote a whole bunch of Go for my job. Then this past Autumn re-visited Python to implement a networked terminal based game[1]. With what I learned about Go and concurrency, I would say that currently in Python, writing concurrent code is not very hard, and is as close to Go as you can get without actually just writing Go. Now, you may be saying "but Python has the GIL, how can concurrency be easy in Python?" I'd say, you're definitely not wrong that the GIL is a problem, but it's not much of a problem for concurrency. This goes back to the heart of Rob Pike's classic talk, "Concurrency Is Not Parallelism"[2]. To quote Wikipedia: In computer science, concurrency is the decomposability property of a program, algorithm, or problem into order-independent or partially-ordered components or units. In Python, you can pretty easily emulate the conceptual properties of Goroutines and Go channels with Python threads and queues. The problem is that doing this in Python won't net you the performance increases you get with Go. And I believe that is an important distinction. There are plenty of cases where you don't care so much about the performance benefits of parallelism, but you want the conceptual and implementation benefits of concurrency. In closing, concurrency in Python is pretty easy to work with, it just performs very poorly. [1] - https://github.com/lelandbatey/defuse_division https://github.com/lelandbatey/defuse_division [2] - https://blog.golang.org/concurrency-is-not-parallelism https://blog.golang.org/concurrency-is-not-parallelism

Spiritus 10y ago
Using Gevent[1] is quite similar to Go when it comes to concurrency in Python. [1] http://sdiehl.github.io/gevent-tutorial/ http://sdiehl.github.io/gevent-tutorial/
rcarmo 10y ago
I wrote a little task module that does precisely that and provides you with a go() function and "channels": https://github.com/rcarmo/python-utils/blob/master/taskkit.py https://github.com/rcarmo/python-utils/blob/master/taskkit.p... Obviously, it wasn't amazingly performant. But it did help a lot for doing concurrent stuff, and I've been pondering re-doing it for asyncio.
nhumrich 10y ago

pjmlp 10y ago

I never saw Python as anything more than a scripting language to portably automate tasks across UNIX and Windows environment, even back in the Zope days. My experience with Tcl teached me to stay away from languages that don't have either a JIT or AOT compiler on their reference implementation.

jszymborski 10y ago
I think that that's rather dismissive of a language that runs huge web, scientific, and general purpose applications daily.
- pjmlp 10y ago
  It might be, but that it is usually a consequence of not knowing any better or making use of existing libraries. Just like people learned 8-bit BASIC and went on to do business applications and games on it. I went Z80 ASM instead. Personally I would only use Python for shell scripting and advise for using Julia instead. Of course, others see it differently.
  - goatlover 10y ago
    Julia's nice, but it's playing catch up to R & Python. Name any statistical algorithm, and R probably has it. Name any scientific field, and there's probably a Python library for it.
  - wolf550e 10y ago
    Some people are very productive in Django and Rails (and the like), in non-performant languages. Many sites never grow to the size where running on the JVM (or AOT-compiled Go) would save you money from fewer servers vs. more coder-hours.
  - JPKab 10y ago
    Julia? Look, I enjoy playing with Julia, but have you actually used it? Because most of the people I meet who turn up their nose at Python and say nice things about Julia haven't ever used Julia. It's got great potential but has some major gaps and warts, and is just plain dog shit slow for certain things compared to SciPy, which, you may not be aware, is basically C and Fortran code wrapped in Python API.