7 ms·

In case anybody missed this: https://instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://instagram-engineering.com/

In case anybody missed this: https://instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://instagram-engineering.com/dismissing-python-garbage-...

karlding 6y ago
FWIW, there's actually a CPython patch from Instagram [0] that proposes "Immortal Instances" (and I understand that YouTube had their own set of patches for "Eternal Refcounts") which aims to alleviate some of those issues and reduce Copy-on-Writes. [0] https://bugs.python.org/issue40255 https://bugs.python.org/issue40255
duskwuff 6y ago
Actually, I think this was meant as a reference to Twitch's experiences with the Go garbage collector: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
- me22 6y ago
  or may be: https://blog.discord.com/why-discord-is-switching-from-go-to-rust-a190bbca2b1f https://blog.discord.com/why-discord-is-switching-from-go-to...
  - tim333 6y ago
    The Discord Go to Rust seems quite sensible engineering.
    - The_rationalist 6y ago
      Pure nonsense, they can switch to rust if they desire so but the argumentation was a complete strawman comparing a ~3 years old go to nightly rust.. The Go garbage collector has improved significantly since then but doing an update instead of a rewrite would have made a less "exciting" blog post I guess..
    - pron 6y ago
      Sensible? I guess when you dig yourself into a hole, wanting to move to a shallower hole seems sensible. It showed that they understood neither their requirements nor their chosen tools, not surprising given that their stack is Elixir, Go and Rust, languages that there aren't too many experts for, and such consistent choices are a clear display of inexperience. There's just no way that technical sensibility is a chief concern of a company that goes from Elixir to Go to Rust within a few years. If they had started with a runtime with better GCs or chosen, say, C/C++ from the beginning, none of this would have been necessary. They clearly choose hype first and everything else second. If that's their technical evaluation process, they are likely to choose wrong the first time, the second time, the third time, and however many times they need to rethink their choices, and every time their rationale would seem sensible when it becomes clear that their previous, ill-advised, choice is wrong, and before their new, untested choice shows its own problems. Making a bad choice and then wishing to fix it by making another using the same process that's led to the first doesn't inspire confidence.
      - jason0597 6y ago
        Yes, if they used C/C++ from the beginning, they wouldn't have run into GC issues, but they would run into another whole set of issues exclusive to C/C++ (free-after-use, buffer overflows, annoying threading bugs, etc.)
        pron 6y ago
        Just like they've run into issues with Elixir and are going to run into plenty of serious issues with Rust. The belief that your chosen language/runtime does not have some very serious issues is the mark of inexperience. The only thing you can do is know what they are and what you can live with. Choosing a hyped language makes the first part very hard; their history shows that they consistently fail on the second part, too. It looks like it's a very inexperienced team, which tech has plenty of, but turning their blunders into success stories is precisely the kind of amateurism dressed in bluster that this post is rightly ridiculing.
        tim333 6y ago
- nappa-leon 6y ago
  I work for twitch and I've seen that article before and it doesn't seem to fit the pattern of we messed with GC and it didn't fix out problem. The solution in the article uses weird features and behaviors of the golang GC and how VMs allocate memory but it did work very well for them.
  - EastSmith 6y ago
    Can you please remove the channel points button for me? Thanks
not2b 6y ago
Yes, but the Python garbage collector is only needed to handle cyclic references. Objects whose reference count drops to zero are still freed even with GC turned off. So AWS didn't make a mint from the experiment.
- saagarjha 6y ago
  I would also expect Instagram to be on Facebook's infrastructure these days ;)
- detaro 6y ago
  Right. The article describes them saving resources in the end from it, but I guess that was too reasonable for satire.
- pseudosavant 6y ago
  I'm sure the cost of getting all of the devs to write Python that safely runs without GC is far les than 10% of your AWS bill for Python services... Maybe at Instagram it is, but it won't be at 99% of companies. It isn't like most Python devs you interview are going to have any kind of a reasonable answer for "How would you write XYZ - with the GC disabled?" I'd bet the ones with a good answer cost >10% more on average.
- The_rationalist 6y ago
  So swift using reference counting instead of a GC was BS marketing where the python approach (using both where it make sense) is a precedent and a superior solution?
hinkley 6y ago
See also Hotmail when they started migrating it to Microsoft tech. The servers were stable for a day or two, so they cooked up a reaper process that would sequentially reboot every machine in the cluster, wait for it to start routing traffic, then reboot the next one. Every machine got at least one reboot a day from this. The moral authority with which I used to share that anecdote has eroded quite a bit in the years of Continuous Deployment. Most of my endpoints reboot at least once a week, except for holidays where we occasionally discover longevity bugs. With autoscaling some instances will be up for hours at a time (and if we are not very careful we'll completely pollute our server statistics).
- JJMcJ 6y ago
  One advantage of CGI is that the worker process terminates, and all the resources get freed. Not that I recommend CGI for anything but very low usage sites, typically internal ones with a fixed user base.
  - hinkley 6y ago
    Not to worry. We'll cloud-enable fastcgi and call it Serverless.
  - deleted 6y ago
    [deleted]
- redis_mlc 6y ago
  eBay did the scheduled reboot thing around 2000, but on a massive scale. Their front-end servers were Windows ISAPI, and leaked memory like a sieve. So there was a documented process for rebooting groups of servers with a diagram dividing the front-end servers into rebootable clusters and the time between reboot intervals. And yes, docker/k8s sure reminds me of those days when reviewing logs today and seeing some pods with unexplained non-zero restart counts. Source: worked there, read the documentation.

In case anybody missed this: https://instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://instagram-engineering.com/dismissing-python-garbage-...

karlding 6y ago

FWIW, there's actually a CPython patch from Instagram [0] that proposes "Immortal Instances" (and I understand that YouTube had their own set of patches for "Eternal Refcounts") which aims to alleviate some of those issues and reduce Copy-on-Writes. [0] https://bugs.python.org/issue40255 https://bugs.python.org/issue40255

duskwuff 6y ago

Actually, I think this was meant as a reference to Twitch's experiences with the Go garbage collector: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...

me22 6y ago
or may be: https://blog.discord.com/why-discord-is-switching-from-go-to-rust-a190bbca2b1f https://blog.discord.com/why-discord-is-switching-from-go-to...
- tim333 6y ago
  The Discord Go to Rust seems quite sensible engineering.
  - The_rationalist 6y ago
    Pure nonsense, they can switch to rust if they desire so but the argumentation was a complete strawman comparing a ~3 years old go to nightly rust.. The Go garbage collector has improved significantly since then but doing an update instead of a rewrite would have made a less "exciting" blog post I guess..
  - pron 6y ago
    Sensible? I guess when you dig yourself into a hole, wanting to move to a shallower hole seems sensible. It showed that they understood neither their requirements nor their chosen tools, not surprising given that their stack is Elixir, Go and Rust, languages that there aren't too many experts for, and such consistent choices are a clear display of inexperience. There's just no way that technical sensibility is a chief concern of a company that goes from Elixir to Go to Rust within a few years. If they had started with a runtime with better GCs or chosen, say, C/C++ from the beginning, none of this would have been necessary. They clearly choose hype first and everything else second. If that's their technical evaluation process, they are likely to choose wrong the first time, the second time, the third time, and however many times they need to rethink their choices, and every time their rationale would seem sensible when it becomes clear that their previous, ill-advised, choice is wrong, and before their new, untested choice shows its own problems. Making a bad choice and then wishing to fix it by making another using the same process that's led to the first doesn't inspire confidence.
    - jason0597 6y ago
      Yes, if they used C/C++ from the beginning, they wouldn't have run into GC issues, but they would run into another whole set of issues exclusive to C/C++ (free-after-use, buffer overflows, annoying threading bugs, etc.)
      - pron 6y ago
        Just like they've run into issues with Elixir and are going to run into plenty of serious issues with Rust. The belief that your chosen language/runtime does not have some very serious issues is the mark of inexperience. The only thing you can do is know what they are and what you can live with. Choosing a hyped language makes the first part very hard; their history shows that they consistently fail on the second part, too. It looks like it's a very inexperienced team, which tech has plenty of, but turning their blunders into success stories is precisely the kind of amateurism dressed in bluster that this post is rightly ridiculing.
        tim333 6y ago
nappa-leon 6y ago
I work for twitch and I've seen that article before and it doesn't seem to fit the pattern of we messed with GC and it didn't fix out problem. The solution in the article uses weird features and behaviors of the golang GC and how VMs allocate memory but it did work very well for them.
- EastSmith 6y ago
  Can you please remove the channel points button for me? Thanks

not2b 6y ago

Yes, but the Python garbage collector is only needed to handle cyclic references. Objects whose reference count drops to zero are still freed even with GC turned off. So AWS didn't make a mint from the experiment.

saagarjha 6y ago
I would also expect Instagram to be on Facebook's infrastructure these days ;)
detaro 6y ago
Right. The article describes them saving resources in the end from it, but I guess that was too reasonable for satire.
pseudosavant 6y ago
I'm sure the cost of getting all of the devs to write Python that safely runs without GC is far les than 10% of your AWS bill for Python services... Maybe at Instagram it is, but it won't be at 99% of companies. It isn't like most Python devs you interview are going to have any kind of a reasonable answer for "How would you write XYZ - with the GC disabled?" I'd bet the ones with a good answer cost >10% more on average.
The_rationalist 6y ago
So swift using reference counting instead of a GC was BS marketing where the python approach (using both where it make sense) is a precedent and a superior solution?

hinkley 6y ago

See also Hotmail when they started migrating it to Microsoft tech. The servers were stable for a day or two, so they cooked up a reaper process that would sequentially reboot every machine in the cluster, wait for it to start routing traffic, then reboot the next one. Every machine got at least one reboot a day from this. The moral authority with which I used to share that anecdote has eroded quite a bit in the years of Continuous Deployment. Most of my endpoints reboot at least once a week, except for holidays where we occasionally discover longevity bugs. With autoscaling some instances will be up for hours at a time (and if we are not very careful we'll completely pollute our server statistics).

JJMcJ 6y ago
One advantage of CGI is that the worker process terminates, and all the resources get freed. Not that I recommend CGI for anything but very low usage sites, typically internal ones with a fixed user base.
- hinkley 6y ago
  Not to worry. We'll cloud-enable fastcgi and call it Serverless.
- deleted 6y ago
  [deleted]
redis_mlc 6y ago
eBay did the scheduled reboot thing around 2000, but on a massive scale. Their front-end servers were Windows ISAPI, and leaked memory like a sieve. So there was a documented process for rebooting groups of servers with a diagram dividing the front-end servers into rebootable clusters and the time between reboot intervals. And yes, docker/k8s sure reminds me of those days when reviewing logs today and seeing some pods with unexplained non-zero restart counts. Source: worked there, read the documentation.