25 ms·

Elixir saves Pinterest $2M a year in server costs

• kunley 3y ago
  Buzzwords like "always-online software", but I think it boils down to handling orders of magnitude more requests while using not so much of memory.
• signa11 3y ago
  money quote : ''' The combined effect of better architecture and Elixir saved Pinterest over $2 million per year in server costs. '''
  • deleted 3y ago
    [deleted]
• coldtea 3y ago
  By the title alone, this means nothing. How much would it cost otherwise? What is the percentage savings? In TFA, it gets better though: "Steve: That’s pretty easy. When I started on the spam team, we had close to 1,400 servers running. When we converted several parts to Elixir, we reduced that by around 95%. One of the systems that ran on 200 Python servers now runs on four Elixir servers (it can actually run on two servers, but we felt that four provided more fault tolerance). The combined effect of better architecture and Elixir saved Pinterest over $2 million per year in server costs. In addition, the performance and reliability of the systems went up despite running on drastically less hardware. When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Despite running on less hardware, the response times dropped significantly, as did errors."
  • naillo 3y ago
    I think it's a generally practiced strategy at this point to spew out blog dev posts on company blogs to build SEO or act as an ad, regardless of quality.
  • threeseed 3y ago
    > When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Would be curious to know how they tried to optimise the Java stack. Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating or leaking.
    • TacticalCoder 3y ago
      They rewrote, which is known to help too. Going from 30 to 15 instances is not bad but it's very likely that a Java-to-Java rewrite would have helped go down too. The big one however is going from 200 Python servers to 4 Erlang ones: a 50x reduction is quite something and a Python-to-Python rewrite would not have allowed to achieved a 50x gain: > All this is possible because Elixir, and the Erlang platform underneath, are fundamentally designed for always-online software with many users. When you use the right tool for the job, the benefits are clear.
      • chii 3y ago
        > it's very likely that a Java-to-Java rewrite would have helped go down too. it depends. The elixir programming paradigm might be one where you are able to more easily write efficient, but still highly concurrent code, where as it would take more work to do the same in java.
        • elcritch 3y ago
          Elixir / Erlang does open up a lot of paradigms. However it'll be interesting to see how Java fairs with its new virtual threads. There was already Akka so it should happen fairly quickly. Still I'd prefer Elixir. The BEAM VM just runs lighter.
          • OkayPhysicist 3y ago
            If it's cludged on like Java's attempt at supporting basic functional programming, it's guaranteed to painful regardless.
  • andersa 3y ago
    It's really strange to me how people continue to build services using python, knowing it is 100 times slower than appropriate languages, and then get surprised by it being 100 times slower, so they eventually rewrite it.
    • acdha 3y ago
      It’s extremely rare for a system to be 10x slower, much less 100x, and developer productivity is huge. When you see huge numbers being tossed around for an entire system, they almost always mean “our first architecture wasn’t right for the problem” and the question to ask is how much time it would have taken the same team to discover the correct shape of the problem with the other candidates.
    • zelphirkalt 3y ago
      I think it is a lack of know-how. Most businesses/managements are not capable of making the decision to go for an ecosystem like Elixir, because they either don't even know it exists (that is also true for many devs) or they do not dare to do anything non-conventional or non-mainstream, or they have the wrong impression, that the "programming language does not matter". (Well, it does! Since it connects you to an ecosystem that comes with it and its language design choices influence how easily you can do things ...) So then Python comes along and you find loads of devs for that. Once Python is entrenched, businesses have a hard time telling their devs to actually learn something new. And few devs will already explore things like Elixir on their own in their free time. And so they continue to hire Python devs. (One could also replace "Python" with "Java" or "NodeJS" or similar, the principles remain the same.)
  • IshKebab 3y ago
    So really "Python cost Pinterest $2M/year"...
    • Mawr 3y ago
      More like "Python got Pinterest to the point of being able to afford to spend extra $2M/year".
      • IshKebab 3y ago
        I don't know why people think Python is the only productive language.
  • miroljub 3y ago
    Second system syndrome. They did savings by re-implementing their services and attribute those savings to the new tool / programming language. I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python.
    • throwaway894345 3y ago
      I think second syndrome is probably a significant factor, but I can also believe that ditching Python was also a significant factor. EDIT: I’m being rate limited because I guess my comments are too spicy for the HN mods, but anyway I agree that there’s no reason other non-Python languages would fare much worse than Elixir.
      • miroljub 3y ago
        Even if we take this as a factor, there's still no reason why would a Go or Java or SBCL implementation be significantly worse than Elixir.
        • 83457 3y ago
          Isn't the major difference here compared to others the BEAM?
          • throwaway894345 3y ago
            You need to advance a theory about how the BEAM could make a 95% improvement in a way the alternatives cannot. For example, a Go has a very good asynchronous story as well and while it may not be exactly as good as BEAM languages, it makes up for it in straight line execution performance. I’ve personally rewritten a few carefully optimized Python applications in pretty naive Go (without significant rearchitecting) and witnessed 50X-1000X performance improvements. And moreover Go allows for more optimization beyond what Python allows (e.g., consolidating allocations and lifting them out of the hot path).
        • 3y ago
  • princevegeta89 3y ago
    The problem with Elixir is that it is such a foreign language to most of the junior developers and a radical shift to dig into coming from object-oriented and other higher-level languages. This is worsened by the fact that there are not many jobs for Elixir in addition to Development Tooling, IDE Support. To someone who starts their job on an Elixir codebase, it is just not a smooth onboarding at all. While the performance aspect is unparalleled compared to most of the popular scripting languages in the last decade, the price to pay to settle into Elixir seems huge to me.
    • no_wizard 3y ago
      sounds like a feature not a bug. Poor onboarding is a cultural problem in my experience, not really a technology one. When you get junior (or even non junior) developers onboarded in a new language, you have a unique opportunity to break them of bad habits and expand horizons. Yes, there is a cost to it as it extends in the short term the time it takes to get developers ramped up, however the long tail payoff is huge
      • princevegeta89 3y ago
        >>> the long tail payoff is huge In what terms, exactly?
        • no_wizard 3y ago
          assuming that the long tail is well executed Anything the business can control for: architectural designs, server costs, approaches to building out features / services for the business etc. When you can mold someone's experience via a new language to model a domain, they become very efficient to it, when they have no prior notions to fall back on. How many times have developers gone down the wrong path because of X did it this way? type thinking. When you can sufficiently remove that so all that is left to think about is the problem space, you do make more gains around that problem space. My thesis from (albeit anecdotal) experience, is that when you have developers working in a new paradigm (often, this corresponds with a new language) you have better chances at establishing these things than having to consistently try and override a developers prior notions about how something should work / look. The trade off is higher ramp times and slower on-boarding, of course. In the short term, it can be more costly.
  • hinkley 3y ago
    How many man hours did it take to achieve that? We just did a Prometheus migration that I suspect will take us 5 years to break even on the development effort investment. And I'm not even counting opportunity costs, which were immeasurable. I like Elixir and I want it to do well, but bad articles make that harder, not easier.
• deleted 3y ago
  [deleted]
• cynicalsecurity 3y ago
  Compared to Python. No wonder.
  • mkl 3y ago
    And to Java. That's more of a surprise. Most of the gains may have come from architecture changes though: "The combined effect of better architecture and Elixir saved Pinterest over $2 million per year in server costs".
  • veavo 3y ago
    [flagged]
    • jeremycarter 3y ago
      Python is actually bad for the environment.
  • Cthulhu_ 3y ago
    Python is fine to build a first version; you should never pick a language or architecture because you THINK you may need its performance, that's cargo cult. Pinterest could not have predicted they needed 200 servers for this workload - and they probably didn't for a long time.
• redocneknurd 3y ago
  Pinterest - ah that app that has hijacked all images in google search result. Created an account once but has not really used it.
  • deleted 3y ago
    [deleted]
  • interactivecode 3y ago
    While the google images thing is kinda shitty. Pinterest is really really good at image search and image recommendations.
    • cmrdporcupine 3y ago
      Yeah, one thing it really shines at is recommending eating disorder content to teenage girls.
      • TacticalCoder 3y ago
        To be honest the world at large doesn't have an anorexia problem but an obesity epidemy. 50 millions children under the age of 5 are already obese (and I think 93% of obese children shall stay obese their entire life). So... Anorexia may be a hill to die on and it's not fun for parents of anorexic kids but priorities, priorities and priorities. The real eating disorder worldwide leads to obesity, not anorexia and I wish all the hate and energy spent thin-shaming was instead redirected towards fighting the more important eating disorder. The numbers from the WHO here are scary and only ever growing: https://www.who.int/news-room/fact-sheets/detail/obesity-and-overweight https://www.who.int/news-room/fact-sheets/detail/obesity-and...
        • retrocryptid 3y ago
          Though I think the original comment was in reference to teen-aged girls and not the world at large.
• earnesti 3y ago
  If you are starting from AWS, it is likely there are many ways to save tons of money by just moving away from AWS.
  • quickthrower2 3y ago
    I think they stayed on AWS to be fair. But also to be fair they were probably doing something dumb, and not doing that dumb thing is saving them $2M, and at the same time they move to Elixir. OTOH to be fair to Elixir maybe it made it easier to do that refactor compared to say Actor models in Java or whatever.
• lambda_dn 3y ago
  They could have just used different Java/Python libraries and changed the architecture for the same result. The choice of a Programming Language alone has minimal impact on performance at that scale.
  • octacat 3y ago
    Rewrite would give you a more optimized code almost always, because you know what you are writing. Though the article says that they rewrote the notification system, and erlang/elixir is pretty amazing for that stuff. From the point "memory footprint per long-running connection".
  • 0xpgm 3y ago
    Can you give a concrete example where such a result was achieved by only switching libraries and modifying architecture?
    • ris58h 3y ago
      https://news.ycombinator.com/item?id=37305247 https://news.ycombinator.com/item?id=37305247
    • Incolotopo 3y ago
      I mean there are plenty of examples in the wild left and right. Have you ever seen basic db optimization? Alone in my companies people were just using stuff wrong. Performance and Architecture are a after thought in our industry. A normal developer doesn't think about it. There was one query in my company which was running in one region slower than in another one and there was also an explain statment available. No one looked at the explain statement and thought "huh why does this simple select use so much memory". People weretrying to see why the regions themselfs were different not what the problem with the query was.
  • nickjj 3y ago
    > They could have just used different Java/Python libraries and changed the architecture for the same result. Yep, their blog post is carefully worded to say "The combined effect of better architecture and Elixir" but they didn't mention how much of it is related to architecture or what specifically they did with Elixir to make things faster. It feels like a marketing piece for their consulting services. I mean they put: > Rewrote an #AWS APIGateway & #lambda service that was costing us about $16000 / month in #elixir. Its running in 3 nodes that cost us about $150 / month They saved 100x here by moving from an expensive architecture (Serverless lambdas) to potentially reserved instances which are reasonably affordable, at least for cloud standards. I remember once needing to parse XML in Python. I started with the easy approach of using the first XML parsing library I found which was xmltodict. Eventually I stumbled upon lxml which improved overall performance by 20x and I didn't have to rewrite much code at all. Sometimes it's easy to get big wins in your existing language if you know what the problem is.
• metaphor 3y ago
  For context, $2MM ~= 0.3% of FY22 cost of revenue[1]. [1] https://www.sec.gov/Archives/edgar/data/1506293/000150629323000023/pins-20221231.htm#i044669580c194f6c993501787404b002_79 https://www.sec.gov/Archives/edgar/data/1506293/000150629323...
  • cebert 3y ago
    I also wonder what the carbon emission reduction would be from this change.
    • metaphor 3y ago
      An even less significant rounding error; on the order of ~0.002% relative to AWS carbon footprint at best using FY23 forward net sales as a rough proxy.
      • tgv 3y ago
        Since there are really many companies and people using AWS, the effect would be noticeable if everyone managed such a decrease. Calling it "an even less significant rounding error" borders on bad faith.
        • metaphor 3y ago
          > Since there are really many companies and people using AWS, the effect would be noticeable if everyone managed such a decrease. Your condition is predicated on "everyone" at Pinterest revenue scale justifying the business case of throwing away their existing tech stack and adopting Elixir with an equivalent architecture---never mind that the article is coming from a company that sells Elixir consulting services---while major CSPs voluntarily bend over to notionally substantial revenue decline...and I'm bordering on bad faith?? At least I bothered to put a supporting numerical estimate on the hypothetical; you're just handwaving greenwashed bullshit with improbable, unsupported outcomes.
• m1keil 3y ago
  Does anyone has any followup information about the companies and people mentioned? Sadly last blog from Cory O'Daniel is from 2019 and Bleacher Report engineering blog post shows 2019 as the last post. Pinterest has 81 public repos on GitHub but only 3 with Elixir code base.
  • di4na 3y ago
    Pinterest is hard to hear from, but BR definitely still use quite some elixir. We see them regularly around conferences.
  • Coryodaniel 3y ago
    waves
    • m1keil 3y ago
      Haha, Hi Cory, I would really like if 5 years after that article and blog posts if you are still in the Elixir ecosystem?
      • Coryodaniel 3y ago
        Yep. Massdriver [0] is built on elixir and golang. All user facing APIs and cloud provisioning state is written in elixir and runs on pretty much just fumes. Cloud systems interaction is written in golang because cloud stuff. Email is in profile if you’d like to chat. Always happy to talk elixir and golang 0: https://massdriver.cloud https://massdriver.cloud
  • bluesnowmonkey 3y ago
    Elixir still gets limited use at Pinterest. I think there's only one significant system still actively developed in Elixir, though it's pretty non-trivial both in terms of cost and complexity.
• sanitycheck 3y ago
  I wonder how much of their traffic is angry people who ended up there by mistake. They could save a lot more than $2M if they just set robots.txt to disallow everything.
  • pictur 3y ago
    it hurt
  • qwerty456127 3y ago
    Or allow convenient usage without registration. It seems whatever I look for mostly is there, I just go away because I don't want to sign-up&in nor be tracked. I wouldn't realy mind if there were some reasonable non-intrusive content-relevant ads though. I probably might even sign-up some day if I weren't repelled by this being required every time I come. I even stopped reading Quora, and, most recently, Twitter because of this - they started requiring signing-in while I don't want to stay signed in and be tracked even though I actually have respective accounts.
  • kramerger 3y ago
    That perfectly describes my usage of Pinterest. People have written browser extensions to remove Pinterest from search results, as it is almost always a dead-end
    • giancarlostoro 3y ago
      Maybe to you, but my wife uses Pinterest a ton. It is where a lot of the women I know go to for ideas for just about everything, from house decor to even the ideas for our wedding. Pinterest is useful just not for us nerdy guys. I am not sure why Google keeps it though or how they benefit, unless Pinterest uses AdSense exclusively, then one can determine that its some sort of partnership. You would think Google would be smarter about who to send over to Pinterest if thats the case.
      • bradleykingz 3y ago
        It's one thing to use Pinterest intentionally, a whole other to have it shoved down your throat.
• andersrs 3y ago
  And then they take that $2M and bribe someone at Google to ensure 59% of the image search results are from Pinterest.
  • scq 3y ago
    I'm really surprised Google Images doesn't block Pinterest.
    • dazc 3y ago
      Google are returning results for image search without the overhead cost. This is the only logical reason I can think of why Pinterest still exists.
      • xk_id 3y ago
        I was never sure where all the hate is coming from. Pinterest is actually one of my favourite apps, that I use regularly – and I use very few apps. I actually really love it and after years I still find new uses for it. As far as I can tell, they have a pretty loyal user base.
        • itronitron 3y ago
          The hate stems from Pinterest requiring an account, which they didn't start out requiring, just like Instagram which I also don't visit.
          • robbintt 3y ago
            An account is not required anymore.
        • cmrdporcupine 3y ago
          The hate comes from them hiding where the original content is from. They hijack search results and then pull you into an endless circle of harvested content, making it difficult to find the link to the original source of the material.
• vb-8448 3y ago
  I wonder how much of the savings comes from the refactoring and how much from the difference between the speed of python and elixir.
  • laserbeam 3y ago
    It's not unusual to gain 50-100x performance gains on code when moving from python to a compiled language without much refactoring. Dropping from 200 to 4 servers is well within the lines of performance gains one can expect. Yes, those numbers are for code, network speeds are not directly impacted the same way. But again, does not sound unreasonable.
    • elcritch 3y ago
      While Elixir is technically a compiled language, it's only compiled in the same sense that Python compiles to byte codes.
• deleted 3y ago
  [deleted]
• dcastm 3y ago
  Now they can hire 4 more engineers :P
• lifeisstillgood 3y ago
  I argue every project has to go through three phases - no idea what's wrong - No idea what's right - We know what was right, but the world has changed now.
  • 0xpgm 3y ago
    Could you elaborate? I'm finding it hard to connect this to the topic.
    • lifeisstillgood 3y ago
      Yeah. There is a phase of discovery - we simply don't know what (some combination of) the project goals, the user needs, the market, the business model and the architectures possible. By doing something we find out. often our guesses are good. Often the company politics mean we build some atrocious monster. The point here is we are elimating what's wrong (the choices are /right/ not (provably) wrong/ (provably) wrong/ and not even wrong (as a famous physicist would mark his students) Once this is done (maybe proof of concept, maybe research maybe years of fighting the politics) we can build out something that's not wrong. If we time it well and get it right it's a unicorn. It's Facebook. if we time it right but get it merely not wrong, it's friendster or myspace. And of course by the time we have learnt all that, mobile hits and we have to relearn
  • Cthulhu_ 3y ago
    I mean you're not wrong; they built working software in Python, solved a problem, scaled it, and learned about the problem it solved. Once they understood it, they could monitor it. Once they had numbers, they could look into improving it, taking their learnings on the problem and picking a different language / architecture to solve it.
• quickthrower2 3y ago
  > The combined effect of better architecture and Elixir saved Pinterest over $2 million per year in server costs So it may not even be Elixir that is saving most of that money. Who knows.
  • Ensorceled 3y ago
    My bet is that the bulk of the savings came from the re-architecture and, at best, Elixir sped up the path to get there.
    • Coryodaniel 3y ago
      I wrote the lambda piece, you could say re-arch saved us the money (although we could only re-arch cheaply because we already had other massive scale serverful systems in place), but I don’t believe I could have made it as efficient and dependency free in any other language. i had been developing in ruby for over a decade and about 5 years of Node exp at the time. im not a language enthusiast and mostly write in go today.
      • Ensorceled 3y ago
        Go is my current favourite as well. Heck, just replacing python with Go would have done similar numbers. The key is “replacing python” and “rearch”, Elixir was just the mechanism.
• hcks 3y ago
  It’s interesting because while this is an extreme case of performance improvement, the ROI doesn’t seem amazing. "rewriting in another language reduced the number of servers by 95%" is hard to beat, but at the same time, this saves "only" 2m a year, or about 0.3% of FY22 cost of revenue (per another post) Pinterest per employee revenue seems to be around 1m, which basically suggest that this could even be a worse than average allocation of resources. My takeaway would be "don’t bother with this kind of optimisation before you reach a scale where you can afford to do marginal improvements"
  • manmal 3y ago
    Wouldn't you also need a smaller ops team that way, further reducing your costs?
    • diarrhea 3y ago
      Managing 200 Python servers/nodes certainly sounds operationally challenging.
      • magicCat 3y ago
        i disagree, not much more then 2 servers.. you should automate it fully for documentation purposes alone at this scale & test failover regulary if you do this with 4 or 200 machines, does not make a big difference.
        • di4na 3y ago
          Actually it does because the chance of hitting strange edge cases grow significantly, as do the runtime of your deployment, plus networking problems risks.
        • manmal 3y ago
          Running 200 servers will force you to automate everything and also handle (relatively) rare edge cases, meaning added complexity, huge up front dev costs, and a continuous dev effort just to stay afloat. Work has to be balanced in a clever way, making tradeoffs that you otherwise wouldn't need to make. The mental model shifts from individual servers to a whole ecosystem that shows emergent behavior (= things break in a totally new way). Maybe some of that also applies to 4 servers, but to a way lesser extent.
• sergioisidoro 3y ago
  I am very skeptical about this. The title makes it look like it's all about Elixir, but there seems to have been a fair amount of re-architecturing. "One of the systems that ran on 200 Python servers now runs on four Elixir servers" This alone is a major telltale.
  • Cthulhu_ 3y ago
    For sure; Elixir comes with a whole new architecture as well, and they COULD have gained a significant performance improvement if they rewrote / re-architected it in Python. However, could they have done a 50x performance improvement in Python? And what about the other numbers, like speed and concurrency? That said, I'm confident they crunched the numbers and did the tradeoffs; after all, adding another language and/or architecture will make your company more complex, makes hiring more complex.
    • __alexs 3y ago
      > I'm confident they crunched the numbers and did the tradeoffs; Here is the math they did. Pros: Looks good in my promotion packet. Cons: Need a bigger garage to fit all of these new sports cars.
    • sergioisidoro 3y ago
      Depends. Were they already using gunicorn? What about cPython? Was the code written by a junior developer without much consideration for memorization and dictionary access? And sure, you can get some performance improvements by rewriting things in those languages, at the expense of losing the entire python open source environment. So I would need way more information about the previous system to take this even remotely seriously. Python scales quite reasonably for most small to medium companies.
    • hinkley 3y ago
      They would have had to create a bad implementation of half of Erlang to accomplish it. Or you could just fuckin' use Elixir.
  • throwaway894345 3y ago
• retrocryptid 3y ago
  I am confident enterprise users will find ways to make elixir $4 less productive.
• fergie 3y ago
  Anecdotally, I am beginning to hear more and more about organisations moving away from high level cloud infrastructure (such as lambda and cloud gateway) and going back to plain old virtual servers (like EC2), or even on-prem. Often the cost of supposedly "cheap" cloud environments is WAY more than you might expect and all booked as operational rather than capital expenditure (the latter being often preferable to shareholders).
  • devjab 3y ago
    I'm not sure why that would be too surprising to you. In the enterprise organization I've worked with over the past few decades IT strategy is always long term and always cost based. 10-15 years ago organizations moved from the basement to placing their owned hardware in rented racks in data centers that were run by 3'rd party organization because it was cheaper. Then Azure came along and made it sort of a "no-brainer" to move into Azure because you already had a lot of Microsoft products and Azure was cheap. Now with so many Azure price hikes and those 3'rd party data centers improving their business models, the pendulum is swinging away from Azure. That doesn't mean that the move into Azure wasn't the right one at the time, or that it was more expensive than not going into Azure. It's simply that the market evolves.
  • whstl 3y ago
    My previous company went from very expensive cloud CI/CD servers to on-prem off-the-shelf servers. The cost and the incredible performance gains we got by moving to a bunch of local computers was enough to make the whole thing pay for itself in about two months. Yep, physical computers costed less than two months of cloud. Plus the gains in productivity from having to wait minutes instead of hours. Maintenance was never a problem, and we didn't need to hire new people to take extra care of the servers. My current company is thinking of doing the same for AI servers. It's just too expensive in AWS.
  • _rwo 3y ago
    > (...) and going back to plain old virtual servers Some of us never left ;)
  • PestoDiRucola 3y ago
• TekMol 3y ago
  My problem with Elixir is that the syntax is too noisy: defmodule Math do def sum(a, b) do a+b end end Compare that with Python: def sum(a, b): return a+b Having the name of the file be the name of the module and whitespace defining the blocks is the perfect approach for me.
  • h0l0cube 3y ago
    This is a fairer example for Elixir (if comparing succinctness)... def sum(a, b), do: a+b
  • sph 3y ago
    On the other hand, dealing with dependencies, deployment, even containerisation is a breeze in Elixir, so it balances out.
  • Cthulhu_ 3y ago
    Unfair comparison, your Elixir example includes a module definition. That said, code volume is never the issue, code clarity is. Right now you're comparing trivial examples, but what does in this case Pinterest's high volume spam detection code look like? I wouldn't judge a language on microscopic code snippets, or disregard them as "too noisy" when it usually isn't a deciding factor in programming languages.
    • thaumasiotes 3y ago
      > Unfair comparison, your Elixir example includes a module definition. Huh? TekMol explicitly points out that the python code also includes a module definition. It's just that the python module definition takes up zero characters in the source file. (Note that this is not always true, but in this case it is.) It takes more than that, in the name of the file, but the Elixir file will also have to have a name, and the comparison ignores both of those equally.
  • kungfooguru 3y ago
    Or Erlang for even less noise ;) -module(math). sum(A, B) -> A + B.
• qwerty456127 3y ago
  By the way, why Elixir and not just Erlang? (I mean a genuine question, not a dislike to Elixir, I don't code either of these yet and am curious)
  • Cthulhu_ 3y ago
    Elixir is easier out of the box; Erlang, or rewriting small parts into 'pure' Erlang would be an option if needs be, but it's generally better to go with the easier of the two.
• samber 3y ago
  They just stopped coding lambda... This is not a matter of a language. Lambda is costly.
  • Coryodaniel 3y ago
    Author of the “Lambda” piece [0], all of the cost savings I discussed came from API Gateway alone. The lambdas were negligible. That being said, I don’t think we could have processed the traffic more efficiently in another language. We were processing 12M requests per hour. You could run the entire thing on 2 vCPU we “over provisioned the crap out of it” by running it on 4 pods w 2 vCPU as the upper request limit. > Our service didn’t quite grow exponentially in use, but it did hockey stick. It went from free, to a few hundred bucks, to around $12,000 just for API Gateway. No Kinesis. No Lambda. Just API Gateway. > A good part of this entire system still runs in Lambda, although it will be moving into Elixir over time to make it easier to reason about and develop on locally. > What everyone should do is think about where your service is going, and can you afford those costs when you get there. If you don’t have a team of ops people and you aren’t familiar with serverful stuff, spending $30k/mo on HTTP requests might be cheaper than an ops team. [0] https://medium.com/coryodaniel/from-erverless-to-elixir-48752db4d7bc https://medium.com/coryodaniel/from-erverless-to-elixir-4875...
    • Incolotopo 3y ago
      Its still not very easy to understand if this is now good or not. 3-5k requests per second is not that much and highly depends on what you actually do. A lot of things can be done just with nginx. nginx is very fast.
      • Coryodaniel 3y ago
        I don’t recall the payload size, but these were all POSTs w/ decent sized bodies, it took about 60 Kinesis shards to avoid writethroughputexceeded.
• dghughes 3y ago
  What is the point of Pinterest other than annoying people in search results?
  • tiborsaas 3y ago
    You collect images. I know, revolutionary.
  • tommica 3y ago
    Using it?
  • cambaceres 3y ago
    I am autistic and doesn't detect irony very well, but if you ask sincerely I can tell you that I at least use it for inspiration on interior design.
• dgan 3y ago
  If am rewriting python, I d like yo switch for a typed language though... It makes further refactoring easier, everything else aside
• danjc 3y ago
  These extreme claims strain credibility. "We built this thing really shabbily and look how much we saved using a different stack. Oh and we also architected it properly second time around."
  • jongjong 3y ago
    Agreed. I get triggered whenever I see a headline like "We moved from tech A to tech B and reduced/increased C by X%" No, that's not what happened! What happened is you put more thought into it the second time. I hate this management BS; if the project fails, blame the engineers, if the project succeeds, praise the tech... They just do everything to turn engineers into commodities and most of us just play along.
• kramerger 3y ago
  > José: Why did you choose Elixir? > Steve: We chose Elixir because we were looking for a system that was easy for programmers to understand and could take better advantage of our servers. I was intrigued at Elixir’s combination of friendly syntax, powerful metaprogramming features, and incorporation of the Actor model. Am I wrong or is this guy in some sort of bubble where only functional languages are taught?
  • capableweb 3y ago
    You seem to be wrong, at least based on the quote you chose to use here, as Steve doesn't even mention anything related to functional programming?
    • kramerger 3y ago
      Elixir is a functional language for the Erlang VM. Most C++/Java/Rust/... programmers wpuld have hard time understanding it. But if you are in a bubble where everyone uses Haskell and talks about Monads, then OPs statement may be valid
      • yesco 3y ago
        I think Elixir is closer to an easier to read Lisp than to Haskell. In fact if someone claimed to know Rust, but couldn't figure out Elixir then I'd simply assume they were lying about their Rust experience.
      • capableweb 3y ago
        Monads and Haskell is not a requirement to understand a functional programming language. Also, I find it hard to believe that anyone who knows N+1 programming languages would have a hard time understanding Elixir quickly, it looks like most mainstream programming languages used today, with slightly different syntax for some things. Take a look at https://elixir-lang.org/crash-course.html https://elixir-lang.org/crash-course.html and you'll see what I mean, it's basically Ruby with some slight modifications.
• jonobird1 3y ago
  Honestly sounds like correlation vs causation. I doubt doubt Elixir is amazing - it is. But maybe the old code was just written poorly from a performance perspective, and then they rewrote it, and now it's performant.
• zarroboogs 3y ago
  Does anyone know if Onlyfans was rewritten in Elixir when it was acquired?
• hermannj314 3y ago
  If anyone is struggling with analysis paralysis, remember it is ok to do things wrong the first time because then you can farm that sweet internet karma bragging about how you fixed your crappy first iteration. If you always do things right the first time, you don't get to brag about putting out the fire your started.
  • mabbo 3y ago
    Exactly. Goal number one is a working product. All you know is python? Write it in Python.
• varispeed 3y ago
  Surely this means they could donate at least $1M a year to the Elixir team?
  • Incolotopo 3y ago
    yes but besides a somehow snarky comment, it doesn't make sense from a company perspective. This wasn't done to have mone to donate to Elixir but for revenue. And besides that, you don't even know if they do
    • varispeed 3y ago
      Big corporations have a habit of promoting open source, then gullible developers produce exceptional work from which said big corporations make millions or billions without paying a penny back. Then developers learn they can't pay bills with exposure tokens and employment doesn't offer the lifestyle they had hoped for. But there is no worry, they get replaced by next cohort of gullibles believing in open source. What I am trying to say, it should be illegal for big corporations to use open source without paying royalties to all contributors.
• sharts 3y ago
  Couldn't they have just moved their python to Cython / Pyrex? Or changed their arch and kept the tech stack the same? I wonder how much time and team hours were spent learning the new framework and it's caveats vs if they'd just spent a little time optimizing their existing stack. All too often common with non-perf folks making sweeping changes without rigorously measuring along the way. I guess it makes for good headlines.
  • pjmlp 3y ago
    Yeah, but then it wouldn't be career driven development, with a nice blog post.
• deleted 3y ago
  [deleted]
• andrewstuart 3y ago
  200 servers running Python. I bet it was an architecture problem more than a language problem. I’d be expecting to leave most of that in place and fix the hotspots that are dragging performance down. Maybe some compiled code in certain places. Almost certainly a detailed analysis would have yielded 20 things that could be tuned. Posts like this are almost something to be ashamed of “we rewrote an entire subsystem because we wanted to use a language we like”. That’s really failing in your responsibility to the company.
  • pclmulqdq 3y ago
    And they got a 50x reduction in server cost - 200 servers down to 4. I have never heard of a performance optimization project with such substantial gains without a complete rewrite.
    • pessimizer 3y ago
      It's important to remember the BEAM is a monster. I'll believe almost anything after I've seen big projects run on such small resources. I've been busy believing anything about BEAM since Whatsapp got so big on nothing, never mind Discord.
  • EMM_386 3y ago
    > Posts like this are almost something to be ashamed of “we rewrote an entire subsystem because we wanted to use a language we like”. That’s really failing in your responsibility to the company. I would tend to agree. I don't understand these sorts of decisions to rewrite huge sites in niche languages and frameworks. You tend to get one engineer who really likes some language, has some clout, and is able to pitch it as a good idea. And then the blog posts about how it saves the company untold millions, while hiding all of the associated costs. Not only is it niche, so your costs going forward will be higher, it even required investing in new "tools and strategies" just to bechmark it. "Elixir has proven so efficient that testing the limits of our services became a challenge unto itself, requiring investment in new benchmarking tools and strategies". I wouldn't personally gamble on a robust job market for Elixir developers in a decade. And those there are will be able to command higher wages. Sure, they will be out there. But so were COBOL developers for Y2K.
• mgaunard 3y ago
  Imagine how much it could save if written in C++.
• truth_seeker 3y ago
  >> 12 million requests / hour with sub-second latency, ~300GB of throughput / day. around 3.3 k requests per second ....is that a lot ???
• rawoke083600 3y ago
  Could they use the money to get a ux team ?? I m still not sure what to click to make the picture go big :s
  • swader999 3y ago
    I don't think 2m is enough to fix Pintrest UX.
    • rawoke083600 3y ago
      I was afraid I might be the only one !
• paxys 3y ago
  So the cost of 3-4 FTEs at a company that employs >4000. Cool. And how much does the change cost them in terms of: - Not being able to hire talent when they need because they don't know or aren't willing to work with Elixir. - Spending extra time training engineers. - Having to hire more senior engineers/those versed in functional programming vs generalists. - The rewrite itself (I'm sure it didn't happen magically overnight). Unless the number had a few more zeros at the end of it, there's no scenario where this project made real business sense vs being some principal engineer's vanity project. $2M is what a company like Pinterest spends on an off-site or a random exec's spot bonus. There are probably analytics dashboards that no one looks at that cost >$2M/yr to produce.
  • Dowwie 3y ago
    It wasn't too long ago that engineers had to walk over hot coals to get their company to use Python. All of the things you're saying were applicable to Python when the company "only" had programmers who knew Java.
• anonymousDan 3y ago
  Can anyone comment on whether there are plans to exploit the new Linux iouring APIs in the BEAM runtime? For example can it help avoid having dedicated OS threads for file IO?
• haolez 3y ago
  On a related note, what's superior about Elixir compared to Erlang if we disregard Phoenix?
  • AlchemistCamp 3y ago
    Newer tooling, macros, protocols and access to all of the libraries available for either.
• mberning 3y ago
  How many FTEs were required with elixir knowledge? This day and age it doesn’t take many engineers to burn up $2M in comp and ben. Not saying it’s a bad idea as the cost savings will accumulate over time, but it’s probably not pure savings.
• smcleod 3y ago
  How is Pinterest even still a thing? I’ve never heard of anyone using it and for everyone I know with an Adblocker it’s the first thing to go.
• say_it_as_it_is 3y ago
  A lot of enraged engineers are here arguing about the claims, ignoring the real problem at hand: Pinterest still exists. WTF is going on with people still using Pinterest?
• emmanueloga_ 3y ago
  Steve: Elixir saved Pinterest over $2 million per year in server costs. Jose: Can we have some of that money? Steve: No.
• cutler 3y ago
  That Java comparison may soon become dated when JDK 21 hits stable in a few weeks.
• deleted 3y ago
  [deleted]
• cutler 3y ago
  Whatever merits of Elixir there are only 10 Elixir jobs in the whole of the UK according to Indeed.com. Adopiton matters.
  • weatherlight 3y ago
    there's a joke that there reason there's so few jobs is 1) they are very high paying, and that the engineers are so productive they only need a few engineers to get work done.
• sgt 3y ago
  Clojure comes to mind. I think personally I'd rather rewrite into Clojure than Elixir. If only for the sake of REPL.
  • weatherlight 3y ago
    Elixir has a repl too! and I mean a repl in the LISP sense of a REPL. I can open a repl on my machine and peer into a production machine and its internals.
    • sgt 3y ago
      I think people underestimate Elixir. Is it easy to pick up?
      • weatherlight 3y ago
        Yes. I think so. It has an algol type/ style syntax. If you can wrap your mind around immutability of data structures and understand how to use enumerators instead of loops for iterators, you're golden.
• brigadier132 3y ago
  Guys, Pinterest hasn't used Elixir at all for more than half a decade. I think there was a single rate limiting service that used it and I have no idea if that was swapped out or not.
  • bluesnowmonkey 3y ago
    This is incorrect. We continue to use it in a large rule engine for spam filtering.
    • brigadier132 3y ago
      I was there and it was not being used in new projects. Maybe that spam filtering service has existed for that long. I would definitely not say that pinterest has fully bought into using elixir because of cost savings which is what the article seems to be insinuating.
      • bluesnowmonkey 3y ago
        There was a point like 8 (ish?) years ago where Pinterest wanted to consolidate on a single backend language internally, after it became apparent Python was not suitable, and multiple contenders were being tried in different places. I think at that time Steve was advocating for Elixir to be it. However it ended up being Java. Nowadays, kind of ironically after the consolidation, we have backend services in at least six languages I can think of. Which is just the reality of how things play out for a tech company doing acquisitions. But yeah we’re not making new ones in Elixir.
        • weatherlight 3y ago
          that's unfortunate and makes me sad.
• dimgl 3y ago
  Why is everyone so angry and skeptical in these comments? I agree the article is operating on old information, but what's going on here?
  • slotrans 3y ago
    It's a vendor post, written to sell you something. Accordingly, it lacks any nuance or perspective and makes claims it doesn't even attempt to back up.
  • corytheboyd 3y ago
    It’s hacker news, hooray! Personally I don’t mind the skepticism. I glean insight from it that I would not have on my own. Rarely it’s downright malevolent, but the tone is usually about the same. I agree with the skeptics about this article, it’s a flashy headline that can’t be accurate, per the points being raised.
• moomoo11 3y ago
  I use Go and it gives me a good sweet spot between cost and perf. I’m nowhere close to performance requirements of Pinterest. But my auth service runs on a single 1vcpu, 512mb ram, and 10gb ssd. I use leveldb and swap on the 10gb. I’ve benchmarked it to handle 8-9k rps while delivering 150ms max response time. Not bad for a few bucks a month.
  • melx 3y ago
    And? Comparing apples and oranges? I'm running seven Elixir/Phoenix apps (one of these is umbrella with 12 apps) on shared CPU cheap VPS with 1gb memory. I actually ran out of disk space recently than anything else so far.
  • hu3 3y ago
    Good thing about Go is that it is easy to learn and hire for.
    • moomoo11 3y ago
      Go is pretty easy. The hard thing about go is maintaining proper structure. Sounds like not a big deal but trust me I’ve learned the hard way lol.
• adamgamble 3y ago
  "The combined effect of better architecture and Elixir saved Pinterest..." I appreciate that they also call out better architecture here. People often read these kinds of headlines and think "Oh Elixir is better than python" or whatever. I remember when twitter had fail whales constantly and they rewrote it from ruby to java I think. At the time everyone assumed it was all ruby's fault, and that might be partly true. But it's also true that the engineers who rebuilt it understood the problem much better now, and knew the major pain points. They also completely changed the architecture to be better suited to the problem. I submit that a lot of rewrites could happen in the same language and still have major gains. All that being said Elixir is great, and particularly well suited to these kinds of problems.
• ksec 3y ago
  >Bleacher Report, a top real time sports and media website, was hitting scaling problems due to their business success. With Elixir, they were able to 8x the average daily traffic load, going from 150 servers to only 5: The site was originally written on the Ruby on Rails framework but Bleacher Report reached the point where they could no longer scale it, according to Dave Marks, BR’s senior engineering director. And they said they could do it with 2. That is anywhere from 30x to 75x. Although I think the current work on Fibre and Async could have reduced it by 10x to 20x as well. Rails just need to adopt it. It is unfortunate they didn't try to optimise it but instead went with Elixir.
• twelve40 3y ago
  I'm glad to read this and I think these advantages have their place, but... Startups rarely die because of server bill specifically. More often it's lack of PMF, slow iteration, expensive labor. $2M is pocket change for Pinterest, they probably spend more on office snacks, so even if you grow to that scale, the savings in this example are not exactly life-or-death. What is life-or-death, however, is how easy it is to hire for, how many quality libraries (implementation speed) and how many library maintainers work on any given stack. If it's easier to hire for Python or even Java, I'll use one of those. For stability, we used Elixir's Oban library to schedule jobs for financial transactions, and a bug in Oban regularly crippled our card authorizer, leading to our customers unable to use their cards and million-dollar losses. Oban, with all due respect, has probably 10x fewer maintainers than comparable solutions/libraries in other stacks. Maybe if we had a "true Elixir guru" on our team, we could have fixed it or even rewritten it from scratch, but we live in the real world, so we ripped it out, replaced it with a more boring solution and are much happier for it, much fewer after-hours pager duty panic attacks. Is it hitting the servers harder? Maybe, I don't care, it just works and the cost difference is likely negligible.
  • AlchemistCamp 3y ago
    The bigger cost is that it takes a lot more devs to do the same thing in Python or especially in Java. Productivity per developer is hands-down Elixir's strongest selling point, not reducing server costs. Python and Java have a multitude of web framework authors and yet none have made anything with the capabilities and ergonomics of Phoenix. I don't think they will either. It would take features those languages lack.
    • twelve40 3y ago
      > more devs to do the same thing in Python or especially in Java i guess it totally depends on "the thing" in question, but do you have any references for that assertion? that common web framework-ish, orm-ish stuff takes a lot more devs in python and especially in java?
      • AlchemistCamp 3y ago
• lchen_hn 3y ago
  Isn’t Pinterest the top most blocked domain in kagi search? Actually they occupy most of the top 10: https://news.ycombinator.com/item?id=37006082 https://news.ycombinator.com/item?id=37006082
• consoomer 3y ago
  Do people still use Pinterest?
• querulous 3y ago
  fwiw this is from 2018 and i would say the conclusion from those who were around pinterest at this time and shortly after would not be nearly as positive
• Loxicon 3y ago
  I don't get it. What did Elixir have to do with the decrease in servers and costs... specifically? Why do people write hollow drivel that doesn't actually say anything?