6 ms·
Nice, I laughed out loud when I read these two paragraphs: > Initially, we tried messing with some garbage collector parameters we didn’t really understand, bu
by est31 7y ago
Nice, I laughed out loud when I read these two paragraphs:
> Initially, we tried messing with some garbage collector parameters we didn’t really understand, but to our surprise that didn’t magically solve our problems so instead we disabled garbage collection altogether. This increased our memory usage, but our automatic on-demand scaler handled this for us, as the graph below shows
> Today we are making some of the code that we can afford to open source available on our GitHub page. It is useless by itself and is heavily tied to our infrastructure, but you can star it to make us seem more relevant.
- segfaultbuserr 7y ago> Fun fact: As our spreadsheet met strong statistical guarantees of randomness, we were able to reuse it to replace our application’s CSPRNG. Made my day.
- frinxor 7y agoMy favorite
- rndgermandude 7y agoIn case anybody missed this: https://instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://instagram-engineering.com/dismissing-python-garbage-...
- karlding 7y agoFWIW, there's actually a CPython patch from Instagram [0] that proposes "Immortal Instances" (and I understand that YouTube had their own set of patches for "Eternal Refcounts") which aims to alleviate some of those issues and reduce Copy-on-Writes. [0] https://bugs.python.org/issue40255 https://bugs.python.org/issue40255
- duskwuff 7y agoActually, I think this was meant as a reference to Twitch's experiences with the Go garbage collector: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
- me22 7y agoor may be: https://blog.discord.com/why-discord-is-switching-from-go-to-rust-a190bbca2b1f https://blog.discord.com/why-discord-is-switching-from-go-to...
- tim333 7y agoThe Discord Go to Rust seems quite sensible engineering.
- The_rationalist 7y agoPure nonsense, they can switch to rust if they desire so but the argumentation was a complete strawman comparing a ~3 years old go to nightly rust.. The Go garbage collector has improved significantly since then but doing an update instead of a rewrite would have made a less "exciting" blog post I guess..
- pron 7y agoSensible? I guess when you dig yourself into a hole, wanting to move to a shallower hole seems sensible. It showed that they understood neither their requirements nor their chosen tools, not surprising given that their stack is Elixir, Go and Rust, languages that there aren't too many experts for, and such consistent choices are a clear display of inexperience. There's just no way that technical sensibility is a chief concern of a company that goes from Elixir to Go to Rust within a few years. If they had started with a runtime with better GCs or chosen, say, C/C++ from the beginning, none of this would have been necessary. They clearly choose hype first and everything else second. If that's their technical evaluation process, they are likely to choose wrong the first time, the second time, the third time, and however many times they need to rethink their choices, and every time their rationale would seem sensible when it becomes clear that their previous, ill-advised, choice is wrong, and before their new, untested choice shows its own problems. Making a bad choice and then wishing to fix it by making another using the same process that's led to the first doesn't inspire confidence.
- not2b 7y agoYes, but the Python garbage collector is only needed to handle cyclic references. Objects whose reference count drops to zero are still freed even with GC turned off. So AWS didn't make a mint from the experiment.
- saagarjha 7y agoI would also expect Instagram to be on Facebook's infrastructure these days ;)
- detaro 7y agoRight. The article describes them saving resources in the end from it, but I guess that was too reasonable for satire.
- pseudosavant 7y agoI'm sure the cost of getting all of the devs to write Python that safely runs without GC is far les than 10% of your AWS bill for Python services... Maybe at Instagram it is, but it won't be at 99% of companies. It isn't like most Python devs you interview are going to have any kind of a reasonable answer for "How would you write XYZ - with the GC disabled?" I'd bet the ones with a good answer cost >10% more on average.
- The_rationalist 7y agoSo swift using reference counting instead of a GC was BS marketing where the python approach (using both where it make sense) is a precedent and a superior solution?
- hinkley 7y agoSee also Hotmail when they started migrating it to Microsoft tech. The servers were stable for a day or two, so they cooked up a reaper process that would sequentially reboot every machine in the cluster, wait for it to start routing traffic, then reboot the next one. Every machine got at least one reboot a day from this. The moral authority with which I used to share that anecdote has eroded quite a bit in the years of Continuous Deployment. Most of my endpoints reboot at least once a week, except for holidays where we occasionally discover longevity bugs. With autoscaling some instances will be up for hours at a time (and if we are not very careful we'll completely pollute our server statistics).
- JJMcJ 7y agoOne advantage of CGI is that the worker process terminates, and all the resources get freed. Not that I recommend CGI for anything but very low usage sites, typically internal ones with a fixed user base.
- hinkley 7y agoNot to worry. We'll cloud-enable fastcgi and call it Serverless.
- deleted 6y ago[deleted]
- redis_mlc 7y agoeBay did the scheduled reboot thing around 2000, but on a massive scale. Their front-end servers were Windows ISAPI, and leaked memory like a sieve. So there was a documented process for rebooting groups of servers with a diagram dividing the front-end servers into rebootable clusters and the time between reboot intervals. And yes, docker/k8s sure reminds me of those days when reviewing logs today and seeing some pods with unexplained non-zero restart counts. Source: worked there, read the documentation.
- JJMcJ 7y ago> Tomorrow we will try to understand our $43,398,207.34 bill for April's AWS usage.
- redis_mlc 7y agoAnd after understanding it, nobody will be dumb enough to volunteer to delete a third of the billed resources.
- thirteen37 7y agoThey will just have a program do the deletion randomly and call it chaos engineering.
- blaser-waffle 7y agoThere has to be at least 6-7 good blog posts to come out of that, too.
- redis_mlc 7y agoThat's an interesting idea! Would work for EC2, not so well for resources you don't alert on like S3 buckets, etc.