29 ms·
By the title alone, this means nothing. How much would it cost otherwise? What is the percentage savings? In TFA, it gets better though: "Steve: That’s pretty
by coldtea 3y ago
By the title alone, this means nothing. How much would it cost otherwise? What is the percentage savings?
In TFA, it gets better though: "Steve: That’s pretty easy. When I started on the spam team, we had close to 1,400 servers running. When we converted several parts to Elixir, we reduced that by around 95%. One of the systems that ran on 200 Python servers now runs on four Elixir servers (it can actually run on two servers, but we felt that four provided more fault tolerance). The combined effect of better architecture and Elixir saved Pinterest over $2 million per year in server costs. In addition, the performance and reliability of the systems went up despite running on drastically less hardware. When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Despite running on less hardware, the response times dropped significantly, as did errors."
- naillo 3y agoI think it's a generally practiced strategy at this point to spew out blog dev posts on company blogs to build SEO or act as an ad, regardless of quality.
- threeseed 3y ago> When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Would be curious to know how they tried to optimise the Java stack. Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating or leaking.
- TacticalCoder 3y agoThey rewrote, which is known to help too. Going from 30 to 15 instances is not bad but it's very likely that a Java-to-Java rewrite would have helped go down too. The big one however is going from 200 Python servers to 4 Erlang ones: a 50x reduction is quite something and a Python-to-Python rewrite would not have allowed to achieved a 50x gain: > All this is possible because Elixir, and the Erlang platform underneath, are fundamentally designed for always-online software with many users. When you use the right tool for the job, the benefits are clear.
- chii 3y ago> it's very likely that a Java-to-Java rewrite would have helped go down too. it depends. The elixir programming paradigm might be one where you are able to more easily write efficient, but still highly concurrent code, where as it would take more work to do the same in java.
- elcritch 3y agoElixir / Erlang does open up a lot of paradigms. However it'll be interesting to see how Java fairs with its new virtual threads. There was already Akka so it should happen fairly quickly. Still I'd prefer Elixir. The BEAM VM just runs lighter.
- OkayPhysicist 3y agoIf it's cludged on like Java's attempt at supporting basic functional programming, it's guaranteed to painful regardless.
- ddorian43 3y agoProject loom should make it better in dev experience and easier to reason + better library evosystem.
- ddorian43 3y agoIs this a cpu-heavy workload? If it's mostly io-heavy, I could see a gevent + pypy approach getting 50x better.
- retrocryptid 3y ago[flagged]
- infecto 3y agoWhy is that?
- 3y ago
- klohto 3y agoThat’s exactly the thing? Why would you bother optimizing the code, looking for overallocations, leakage, tweak parameters, when you can just take a friendlier language for the same benefits.
- acdha 3y agoI think the question is whether they significantly changed the architecture at the same time. For example, reading the description of the Python migration sounds like they applied a lot of experience which would have benefited any language, and micro-optimizations like what you described would have been a rounding error on those larger changes: https://medium.com/pinterest-engineering/fighting-spam-with-guardian-a-real-time-analytics-and-rules-engine-938e7e61fa27 https://medium.com/pinterest-engineering/fighting-spam-with-...
- asabil 3y agoFaster doesn’t mean more responsive, in fact very often an increase in throughput leads to an increase in latency.
- deleted 3y ago[deleted]
- abrookewood 3y agoErlang and Elixir are famous for maintaining very low latency under load - it's due to the way the Erlang scheduler works.
- boxed 3y ago> except memory That's an enormous "except". Absolutely gigantic.
- DoesntMatter22 3y agoElixir is slower than plain PHP according to the techempower benchmarks. I'm not even sure how that's possible but it is. By like a factor of 2 iirc. I'm not sure how elixir is that slow since it's compiled.
- te_chris 3y agoPHP is fast these days raw-speed wise. Concurrency and parallelism matter more than speed as they translate directly to utilisation.
- abrookewood 3y agoYes, calling something Fast can mean many different things. Erlang/Elixir are not 'fast' at number crunching, but they are very good at concurrency.
- Cthulhu_ 3y agoIt depends on what is actually being benchmarked though; if it's the simple JSON payload, more time will be spent on HTTP parsing (done in nginx for the PHP benchmark so really really fast) and some JSON parsing (done in a C library for the PHP benchmark so really really fast). Basically, how much PHP is actually being benchmarked? Are you looking at a benchmark that compares real-world usage, or a microbenchmark like hello world or a small JSON payload?
- DoesntMatter22 3y agoThat's a reasonable point. This is Techempower as I mentioned originally. In pretty much all the benchmarks PHP smokes Elixir
- coldtea 3y ago>Elixir is slower than plain PHP according to the techempower benchmarks. I'm not even sure how that's possible but it is. You're "Not even sure how that's possible" as if implying PHP is slow? Even old pre-7 PHP was much faster than Ruby, Python, and others. And most PHP libs are just wrappers over C code.
- sph 3y agoThe advantages of Elixir are not performance-related. There is a lot of focus on raw performance on web-related services, when in reality most of their running time is spent waiting for IO. If there are two things the BEAM excels at, is IO and turning almost any problem into half a dozen processes that are scheduled and run in parallel, if not geographically-distributed, with 1/50th the effort of any other language. We live in a world with 32+ core CPUs. If your load is not spread uniformly all over those cores, you're losing a ton of performance. Handling requests over separate threads, like 99% of languages do, still isn't enough if all the business logic runs on the same thread. I'm currently writing a web crawler in Elixir, and it is easier to design it so every request is done and processed in parallel, than to write a naive sequential one you'd do in any other language in half a day.
- deleted 3y ago[deleted]
- _m8fo 3y ago[flagged]
- mcintyre1994 3y agoI think Node is still single core by default? Elixir (or rather the Beam) will handle core utilisation so if you start a load of Elixir processes they’ll be spread across multiple cores.
- _m8fo 3y ago[flagged]
- troupo 3y agoIt doesn't effectively do that. It does about 10% of that, ineffectively. Instead of running a single VM with full knowledge of how to run lightweight processes, designed to fully take advantage of modern multi-core CPUs with multiple guarantees enforced by the runtime you have multiple single-threaded VMs awkwardly communicating with each other over a bolted-on API
- Cthulhu_ 3y agoI think it's more important to look at the re-architecting than the different language. Second, I think certain architectures - like the actor model described in the article - work better and more intuitive if you use a different language. That said, I'm sure a 2x performance improvement could've been done in Java as well if they did a re-architecture. They could also have made a lateral movement and go to a different JVM language, like Scala that also has an actor concurrency model + accompanying syntax.
- zaphar 3y agoThis is an important point. Sometimes when you need to do a major rearchitecture of a system it can help to choose a language that is more appropriate to that architecture. The Elixir/Erlang ecosystem has a better story for Actor Model development so it makes sense to choose them for the new architecture. It depends on the team and the specifics of the new architecture because the devil is always in the details. It may be that the new language isn't enough of a win to justify the switch and sometimes the win from the new architecture is big enough that sticking with the current language makes sense. But a knee jerk response of: This is mostly just good because they rewrote/rearchitected it, ignores the benefits of using a language or technology that fits the new architecture better.
- OkayPhysicist 3y ago> certain architectures... work better and more intuitive if you use a different language. This is the key point that people miss when pretending that languages are interchangeable. The entire point of making a programming language is to make certain types of ways to solve problems easier to express. This constitutes a language's "pretty path". By providing such pretty paths, languages necessarily make less desirable paths, which will be painful to slog through. If you try writing a functional pipeline in Java, you're going to have a much worse time than doing the same in Elixir. If you try to do Object-Oriented class towers in Scheme, it's going to be painful. Etc, Etc. You can write a Rust program and a C program that compile to the exact same binary, but I can put a whole stack of cash on which one's going to be easier.
- coldtea 3y ago>Would be curious to know how they tried to optimise the Java stack They say "The combined effect of better architecture and Elixir". So the post is mostly useless.
- pclmulqdq 3y agoNot if Elixir enables you to use a better architecture. See all of the people who lifted linked-list structures out of C into C++/Rust vectors.
- neeleshs 3y agoYep. We got a 10x improvement on throughput for our backend runtime, which was in Java, by moving to a better architecture for performance hotspots.. using Java again. In a rewrite with a different design/architecture, that new design typically accounts for most gains, rather than language. A language may make some parts of that rewrite simpler.
- javanonymous 3y agoI've gotten 100x improvement with no code change by just adding an index in the database table. An inexperienced developer might have blamed the database and insisted on moving to NoSQL because of "web scale". If they got the chance to rewrite it, they could have pointed to the performance increase as a proof that they were right.
- no_wizard 3y agoThey really should teach benchmark training more widely in the industry. Even though I'm readily here to sing the praises of Elixir when warranted, nothing beats actually profiling end to end the workload(s) in question that need to be improved. Sometimes, it really is optimizing the database that matters most, like adding an index (or using window functions or stored procedures, as in many cases I've had in the past)
- neeleshs 3y agoYes, that too. Indexes and caching
- panick21_ 3y agoAnd of course you can just go and buy a much faster JVM from Azul rather then rewrite everything.
- no_wizard 3y agothen I have to pay Azul for the faster JVM and I may still not sufficiently cut my server costs, or the cost of the JVM paces the reduction I have in server costs. 2 million a year is several developers compensation saved every year. It also opens the door to more savings down the road, potentially, as existing workloads may discover they can use the same approaches to reduce cost / overhead.
- panick21_ 3y agoI was just adding that in addition to the truth that normal Java could just do it. But if that is really not enough, then paying for a few Azul licenses is an addition can save you time and money.
- jjav 3y ago>Would be curious to know how they tried to optimise the Java stack. Fairly safe to day not at all. The JVM is extremely fast, very efficient and very scalable (you can write java code that scales linearly with available cores). If performance or scalability is a metric to care about, it is nearly impossible to outscore java. You can, with very skilled C/C++ developers, but it's going to be difficult to find those people and it'll be a lot of work. If you need extreme performance on a reasonable budget, you can't do better than java. I know java isn't hip anymore, so this is not a popular truth, but it is.
- andersa 3y agoIt's really strange to me how people continue to build services using python, knowing it is 100 times slower than appropriate languages, and then get surprised by it being 100 times slower, so they eventually rewrite it.
- acdha 3y agoIt’s extremely rare for a system to be 10x slower, much less 100x, and developer productivity is huge. When you see huge numbers being tossed around for an entire system, they almost always mean “our first architecture wasn’t right for the problem” and the question to ask is how much time it would have taken the same team to discover the correct shape of the problem with the other candidates.
- zelphirkalt 3y agoI think it is a lack of know-how. Most businesses/managements are not capable of making the decision to go for an ecosystem like Elixir, because they either don't even know it exists (that is also true for many devs) or they do not dare to do anything non-conventional or non-mainstream, or they have the wrong impression, that the "programming language does not matter". (Well, it does! Since it connects you to an ecosystem that comes with it and its language design choices influence how easily you can do things ...) So then Python comes along and you find loads of devs for that. Once Python is entrenched, businesses have a hard time telling their devs to actually learn something new. And few devs will already explore things like Elixir on their own in their free time. And so they continue to hire Python devs. (One could also replace "Python" with "Java" or "NodeJS" or similar, the principles remain the same.)
- IshKebab 3y agoSo really "Python cost Pinterest $2M/year"...
- miroljub 3y agoSecond system syndrome. They did savings by re-implementing their services and attribute those savings to the new tool / programming language. I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python.
- throwaway894345 3y agoI think second syndrome is probably a significant factor, but I can also believe that ditching Python was also a significant factor. EDIT: I’m being rate limited because I guess my comments are too spicy for the HN mods, but anyway I agree that there’s no reason other non-Python languages would fare much worse than Elixir.
- miroljub 3y agoEven if we take this as a factor, there's still no reason why would a Go or Java or SBCL implementation be significantly worse than Elixir.
- 83457 3y agoIsn't the major difference here compared to others the BEAM?
- throwaway894345 3y agoYou need to advance a theory about how the BEAM could make a 95% improvement in a way the alternatives cannot. For example, a Go has a very good asynchronous story as well and while it may not be exactly as good as BEAM languages, it makes up for it in straight line execution performance. I’ve personally rewritten a few carefully optimized Python applications in pretty naive Go (without significant rearchitecting) and witnessed 50X-1000X performance improvements. And moreover Go allows for more optimization beyond what Python allows (e.g., consolidating allocations and lifting them out of the hot path).
- 3y ago
- princevegeta89 3y agoThe problem with Elixir is that it is such a foreign language to most of the junior developers and a radical shift to dig into coming from object-oriented and other higher-level languages. This is worsened by the fact that there are not many jobs for Elixir in addition to Development Tooling, IDE Support. To someone who starts their job on an Elixir codebase, it is just not a smooth onboarding at all. While the performance aspect is unparalleled compared to most of the popular scripting languages in the last decade, the price to pay to settle into Elixir seems huge to me.
- no_wizard 3y agosounds like a feature not a bug. Poor onboarding is a cultural problem in my experience, not really a technology one. When you get junior (or even non junior) developers onboarded in a new language, you have a unique opportunity to break them of bad habits and expand horizons. Yes, there is a cost to it as it extends in the short term the time it takes to get developers ramped up, however the long tail payoff is huge
- princevegeta89 3y ago>>> the long tail payoff is huge In what terms, exactly?
- no_wizard 3y agoassuming that the long tail is well executed Anything the business can control for: architectural designs, server costs, approaches to building out features / services for the business etc. When you can mold someone's experience via a new language to model a domain, they become very efficient to it, when they have no prior notions to fall back on. How many times have developers gone down the wrong path because of X did it this way? type thinking. When you can sufficiently remove that so all that is left to think about is the problem space, you do make more gains around that problem space. My thesis from (albeit anecdotal) experience, is that when you have developers working in a new paradigm (often, this corresponds with a new language) you have better chances at establishing these things than having to consistently try and override a developers prior notions about how something should work / look. The trade off is higher ramp times and slower on-boarding, of course. In the short term, it can be more costly.
- hinkley 3y agoHow many man hours did it take to achieve that? We just did a Prometheus migration that I suspect will take us 5 years to break even on the development effort investment. And I'm not even counting opportunity costs, which were immeasurable. I like Elixir and I want it to do well, but bad articles make that harder, not easier.