6 ms·

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting componen

by Twirrim 10y ago

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

matt_wulfeck 10y ago
That's a question I have as well. I'm guessing they have a very massive cpython codebase, and the trade-off was worth it.
- MBCook 10y ago
  I'd assume that's true, but they also didn't know how well this would work. A 'little' project to make Grumpy is probably much easier to sell to management/throw away if to slow than "Let's rewrite all our code in language X to see if it goes faster".
greenhatman 10y ago
Maybe there is too much to rewrite.
ericfrederich 10y ago
It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.
- martincmartin 10y ago
  I think he's talking about moving their Python codebase to Go, so they don't have to run Python anymore.
  - ericfrederich 10y ago
    You're probably right. He seemed to be questioning the translator so I thought he was suggesting replacing CPython piece by piece with a Go implementation.
jerf 10y ago
Ask yourself how much Python code you think Google has. Then multiply it by some large single-digit number, minimum. Then compare the effort of spending 30 seconds per line on that code vs. writing a new Python interpreter/compiler/runtime, especially when you can trivially get people who are capable of doing that. There's a reason why the large companies often end up working on new runtimes/interpreters/compilers like HipHop for PHP, Hack, and so on, rather than working on the code bases written in those languages. It is very easy for it to not just be easier to leverage in at that level, but an order of magnitude or two easier. Or three.
- falcolas 10y ago
  I wouldn't think that a tool for transpiling Python to Go would be harder than creating a whole new runtime. Most of the work is already done; you can get the compiled ASTs directly out of Python itself. Of course, if it started out as a "this seems like a cool project", that skews the "a is more efficient than b" ratio significantly.
  - jerf 10y ago
    Sorry, I think my post may have been excessively specific; I've edited it to say "writing a new Python interpreter/compiler/runtime" instead of just "writing a new Python interpreter". I was trying to make a generic point, not one specifically about a Python interpreter. (At least, if I'm interpreting your reply correctly.)
  - throwaway91111 10y ago
    I mean, you can get something out of transpilers--not typically maintainable code.
- throwaway9912 10y ago
  I don't know PHP, but in the case of Python you will spend the next ten years ironing out subtle incompatibilities between CPython and this new thing. Added up, it is easily more than 30s per line.
  - 10y ago
eva1984 10y ago
It could be part of the auto migration effort.
eva1984 10y ago
It could be part of the auto migration effort.
spankalee 10y ago
Because Python running in Grumpy can import Go packages, I'd assume (I work for Google, but not on YouTube), that Python libraries could then be migrated pretty seamlessly and incrementally to Go.
dragonwriter 10y ago
Creating a new Python runtime in Go is obviously non-trivial, but so is rewriting the heart of YouTube in Go. Heck, YouTube may be a bigger codebase than Python for all I know. And there's a lot more side benefit in a better multithreading Python runtime for Google for other Python code Google has (or hosts), whereas the benefits of a YouTube rewrite are more narrowly limited to YouTube.
- nilved 10y ago
  Just because the site is popular doesn't mean the codebase is big. It's a simple site. I could probably write YouTube in Go in an afternoon and I don't even know Go.
  - zardeh 10y ago
    You could perhaps write a feature-bare clone. Here's some basic features: - search (the various search pages) - trending - channels - subscription - video uploads - history - comments - likes - upload - video editing (most of this is in browser, but still) - livestream - video analytics - payment and ad management - video comment review and moderation - translation - captioning - video replication, multiscaling, caching, and resolution management based on network speed - video recommendation - cards, overlays, annotations, etc. - music and sound effect search - antispam - dmca and copyright tooling And that was 5 minutes of poking around.
    - X-Istence 10y ago
      > resolution management based on network speed Not so much a feature of YouTube as it is a feature of HLS/Dash... but yes, it means you've gotta transcode the source video into multiple different bitrates.
      - zardeh 10y ago
        Even outside of that, I can override the bitrate in some instances. But yeah, I was mainly talking about the necessary replication, cache, and transcoding (and I actually forgot about the multiple codecs, which they also provide, in addition to multiple resolutions/bitrates)
trotterdylan 10y ago
It's been said a couple times in this thread already but basically, the YouTube Python codebase is very large and the cost of a rewrite is prohibitive.
- i336_ 10y ago
  Statistics would be very interesting to the "useless internal trivia" types.
hueving 10y ago
I take it you've never worked on a large code base with a large team? Even ignoring the actual task of rewriting the logic in a different language with different idioms, the change is going to be a massive hit to productivity for many months as people learn the new architecture of the rewrite, the tooling, and idiomatic Go. For a project the size of YouTube, that will be millions of dollars of engineering hours and weeks/months of lost productivity for an unknown gain and almost guaranteed bugs. It's a terrible value proposition so it's better to squeeze every last drop of performance out of the code base you have, which at the scale if this project includes paying engineer(s) to work on a completely new runtime.
petters 10y ago
The thing is, you can not use Python in production at Google. YouTube was acquired a long time ago and a rewrite of the inherited Pyton code base has not happened for very good reasons.

by Twirrim 10y ago

matt_wulfeck 10y ago

That's a question I have as well. I'm guessing they have a very massive cpython codebase, and the trade-off was worth it.

MBCook 10y ago
I'd assume that's true, but they also didn't know how well this would work. A 'little' project to make Grumpy is probably much easier to sell to management/throw away if to slow than "Let's rewrite all our code in language X to see if it goes faster".

greenhatman 10y ago

Maybe there is too much to rewrite.

ericfrederich 10y ago

It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.

martincmartin 10y ago
I think he's talking about moving their Python codebase to Go, so they don't have to run Python anymore.
- ericfrederich 10y ago
  You're probably right. He seemed to be questioning the translator so I thought he was suggesting replacing CPython piece by piece with a Go implementation.

jerf 10y ago

Ask yourself how much Python code you think Google has. Then multiply it by some large single-digit number, minimum. Then compare the effort of spending 30 seconds per line on that code vs. writing a new Python interpreter/compiler/runtime, especially when you can trivially get people who are capable of doing that. There's a reason why the large companies often end up working on new runtimes/interpreters/compilers like HipHop for PHP, Hack, and so on, rather than working on the code bases written in those languages. It is very easy for it to not just be easier to leverage in at that level, but an order of magnitude or two easier. Or three.

falcolas 10y ago
I wouldn't think that a tool for transpiling Python to Go would be harder than creating a whole new runtime. Most of the work is already done; you can get the compiled ASTs directly out of Python itself. Of course, if it started out as a "this seems like a cool project", that skews the "a is more efficient than b" ratio significantly.
- jerf 10y ago
  Sorry, I think my post may have been excessively specific; I've edited it to say "writing a new Python interpreter/compiler/runtime" instead of just "writing a new Python interpreter". I was trying to make a generic point, not one specifically about a Python interpreter. (At least, if I'm interpreting your reply correctly.)
- throwaway91111 10y ago
  I mean, you can get something out of transpilers--not typically maintainable code.
throwaway9912 10y ago
I don't know PHP, but in the case of Python you will spend the next ten years ironing out subtle incompatibilities between CPython and this new thing. Added up, it is easily more than 30s per line.
- 10y ago

eva1984 10y ago

It could be part of the auto migration effort.

eva1984 10y ago

It could be part of the auto migration effort.

spankalee 10y ago

Because Python running in Grumpy can import Go packages, I'd assume (I work for Google, but not on YouTube), that Python libraries could then be migrated pretty seamlessly and incrementally to Go.

dragonwriter 10y ago

Creating a new Python runtime in Go is obviously non-trivial, but so is rewriting the heart of YouTube in Go. Heck, YouTube may be a bigger codebase than Python for all I know. And there's a lot more side benefit in a better multithreading Python runtime for Google for other Python code Google has (or hosts), whereas the benefits of a YouTube rewrite are more narrowly limited to YouTube.

nilved 10y ago
Just because the site is popular doesn't mean the codebase is big. It's a simple site. I could probably write YouTube in Go in an afternoon and I don't even know Go.
- zardeh 10y ago
  You could perhaps write a feature-bare clone. Here's some basic features: - search (the various search pages) - trending - channels - subscription - video uploads - history - comments - likes - upload - video editing (most of this is in browser, but still) - livestream - video analytics - payment and ad management - video comment review and moderation - translation - captioning - video replication, multiscaling, caching, and resolution management based on network speed - video recommendation - cards, overlays, annotations, etc. - music and sound effect search - antispam - dmca and copyright tooling And that was 5 minutes of poking around.
  - X-Istence 10y ago
    > resolution management based on network speed Not so much a feature of YouTube as it is a feature of HLS/Dash... but yes, it means you've gotta transcode the source video into multiple different bitrates.
    - zardeh 10y ago
      Even outside of that, I can override the bitrate in some instances. But yeah, I was mainly talking about the necessary replication, cache, and transcoding (and I actually forgot about the multiple codecs, which they also provide, in addition to multiple resolutions/bitrates)

trotterdylan 10y ago

It's been said a couple times in this thread already but basically, the YouTube Python codebase is very large and the cost of a rewrite is prohibitive.

i336_ 10y ago
Statistics would be very interesting to the "useless internal trivia" types.

hueving 10y ago

I take it you've never worked on a large code base with a large team? Even ignoring the actual task of rewriting the logic in a different language with different idioms, the change is going to be a massive hit to productivity for many months as people learn the new architecture of the rewrite, the tooling, and idiomatic Go. For a project the size of YouTube, that will be millions of dollars of engineering hours and weeks/months of lost productivity for an unknown gain and almost guaranteed bugs. It's a terrible value proposition so it's better to squeeze every last drop of performance out of the code base you have, which at the scale if this project includes paying engineer(s) to work on a completely new runtime.

petters 10y ago

The thing is, you can not use Python in production at Google. YouTube was acquired a long time ago and a rewrite of the inherited Pyton code base has not happened for very good reasons.