10 ms·

There is a big difference between developers who know their hardware stack, and those who don't. Performance efficiency is often an order of magnitude out of wh

by _xnmw 5y ago

There is a big difference between developers who know their hardware stack, and those who don't. Performance efficiency is often an order of magnitude out of whack for companies that prefer a horizontal-first scaling approach (i.e. lots of containers in the cloud, k8s, etc.). As an extreme example, consider Plaid, which at one point had 4000 Node containers [1] running in parallel to handle just as many requests per second. My guess is they could've easily done with a few beefy baremetal machines, if they knew their hardware well, and architected it sensibly from the beginning. Money wasn't a problem for them because they were able to just throw $350k a year at AWS to make it work (and millions more at their dev team). But money isn't the only thing. This sort of approach sacrifices real-time latency (which is felt by the user), reliability (Plaid is notorious for having a ~20% fail rate for requests), flexibility (Plaid is notoriously slow at fixing bugs), developer costs (a dedicated DevOps team), agility (very slow deployment times), not to mention the priceless value of simplicity, a system that can fit in the mind of a single experienced developer, so they can write and test performant code on their own machine. But hey, Plaid is now a unicorn, so who cares about efficiency. [1] https://plaid.com/blog/how-we-parallelized-our-node-service-by-30x/ https://plaid.com/blog/how-we-parallelized-our-node-service-...

p_l 5y ago
Funnily enough, k8s thrives the best when hw is your constraint, in my opinion...
- marcos100 5y ago
  When you program without performance in mind, hw is always your constraint. k8s is a blessing because you can scale out easily and is a curse because you can scale out easily.
  - p_l 5y ago
    Thing is, k8s is not about scaling out. A core part of it is about efficient usage of hw you have (which also maps well to avoid expensive cloud bill). The more I optimized, the more I could leverage k8s to optimize hw usage.
    - adwn 5y ago
      > A core part of it is about efficient usage of hw you have In what way? If the content of your containers is a wasteful, inefficient mess, then there's nothing Kubernetes can do to make it less inefficient.
      - p_l 5y ago
        It can't fix what's inside your containers, but it can fix how efficient you're in packing your workloads onto available hw. I.e. Given a set of Workloads, set of available hw, k8s will help you find maximal amount of Workloads that you can fit. vs. for example manually trying to fit single-purpose VM instances in EC2, with inevitable overheads, etc.
    - marcos100 5y ago
      I agree with you. It will efficiently allocate resources for an inefficient application. But when your system can't handle the load anymore, you'll have to allocate more resources and k8s will handle it for you like magic. It's so easy that people can use thousands of machines to handle the load. This with the "hardware is cheap, dev time is expensive", makes it common to see those kind of very large systems.
      - p_l 5y ago
        Well, yes. Except I get to live with "hardware is more expensive than dev time", so I tend to cheat like a proper dirty cheating wizard with k8s' binpacking ;) (my motto is "what do you call a company using Heroku? Bankrupt" :P)
josephg 5y ago
Yes I absolutely agree. I posted a blog post near the start of the year looking at how much performance automerge (written in javascript using immutablejs) leaves on the table. At the time, automerge took 5 minutes to run a benchmark, and my own code could do the same thing in 0.06 seconds. Since then I’ve sped up my code by another order of magnitude - it can now run the same benchmark in 0.005s. (4.9ms). I don’t want to badmouth automerge - they’re working on it and have made big improvements this year. But that ratio - 5min:5ms - gives you a sense of the mountain of inefficiency we live with on modern computers with a lot of modern software. Modern CPUs and ram are obscenely, incomprehensibly fast. With the exception of a few very specific tasks, you should never be waiting for a modern computer to do anything. We sent men to the moon in fewer cycles than it takes for modern windows to boot, by several orders of magnitude. Complaining about the global chip shortage is a bit like a hoarder complaining about a room shortage in their overflowing house. (With carve outs in my scorn for video editing, AAA game engines, AI, etc)
- bob1029 5y ago
  > Modern CPUs and ram are obscenely, incomprehensibly fast. With the exception of a few very specific tasks, you should never be waiting for a modern computer to do anything. I really wish we could drive this point home for everyone. The amount of work you can get out of 1 x86 core is insane if you know how to feed it properly. The upper bound is somewhere around 500 million transactions per second per core. I have personally built systems that can achieve somewhere around 14 million per second. If you can reliably handle the entire serialized workload on a single thread you start to retool for the idea that you can fit the entire business in 1 box. Being able to run everything out of a single computer is an incredible virtue, not some curse that must be cast to the clouds.
  - DaiPlusPlus 5y ago
    > 500 million transactions per second per core. An instruction is not a transaction. Commodity x86/x64 systems do not implement transactional memory or any model of computation that could be reasonably described as transactional as far as I'm aware.
perryizgr8 5y ago
> 4000 Node containers [1] running in parallel to handle just as many requests per second. This looks crazy to me at first glance. 4k requests per second, for any reasonable sized request, should be easily handled by a single C thread. With plenty of room to grow.
- _xnmw 5y ago
  You're not the only one who found it insane. Check the HN discussion at the time: https://news.ycombinator.com/item?id=21782196 https://news.ycombinator.com/item?id=21782196
gjulianm 5y ago
When I see these stories about horizontal scaling on thousands of nodes, I wonder how many people don't know how much performance can be squeezed out of beefy machines. At my job we build probes for network monitoring, and we're able to ingest quite a lot of traffic (up to 14Mpps per link, for a fully saturated 10G link in our hardest tests), ingest a good volume logs or metrics from remote systems, do some extra processing and serve a web application in just a single instance. We use C/C++ for the traffic capture and processing, Postgres/Elastic for storage (depending on the balance performance/flexibility), some AWK for quick file processing (you'd be surprised how fast can it be) and then Python for plumbing code. On the other hand, I've seen technical blogposts and whitepapers on how to optimize a "big data ingestion system" that requires clustering and tweaking and whatnot for a fraction of what we're able to do with a single machine. People should try harder to scale up before starting a cluster and scaling out.
- christophilus 5y ago
  I was in a discussion on HN just yesterday where the assertion was made that startups should default to the cloud because the upfront cost of allocating real hardware is too high. I’ve got a $20 Netcup server that can probably handle the traffic of 90% of any projects I’ve ever worked on (including my stint at big tech cos). It’s amazing how cheap and fast modern computing is when compared to the standard cloud offerings.
  - CogitoCogito 5y ago
    If they don't want to pay upfront for hardware, they can also "default to cloud" while just using cloud-based compute instances.
  - fullstackchris 5y ago
    ... but netcup is just another cloud offering? What are you getting at here? I mean, it always makes sense to shop around and look at pricings before choosing one, but my unresearched guess is that there are so many providers nowadays that the pricings across all of them are within cents / dollars for the same service (based on RAM, storage, etc)
thecupisblue 5y ago
It's a multidimensional problem. 1.) Rush to build. Backpressure and deadlines lead to "lets just spin it up thje easier way" 2.) We said "premature optimisation is the root of all evil". This ended up in devs not thinking about performance at all because "hurr durr" premature optimisation. I've witnessed this mentality causing huge problems in performance due to simplest decisions that would be normal in other circumstances, but snowball into a huge perf issue. Exactly as the title says, best practices can slow your application down, especially when used out of context. Some include "I'll serialise this into json and deserialise it again 5 lines later", "I'll use a regex to parse this simple string", "I'll nest 3 map calls here on a list", "I'll find it later in the list again". 3.) People forgot/never learned what the stack is. Honestly, when people say software was harder to write back in the day, I'm never sure if I agree with it or not. One one hand, yes, we had less tools and materials at our disposal. But at least the thinking through it was easier, since with the small set of tools you have, you could go up and down the stack in your head and know that if you do X, Y and Z, that's what would happen. Nowadays, we have so many tools and layers that it takes a huge amount of knowledge, time and mental energy to reason about your whole stack. We have people specialised in tiny subsets of the stack, so getting performance squeezed out at overall stack level requires people who understand more of the stack, which is getting rarer and rarer by day. 4.) We have lost touch with the power of hardware. Consider that most of senior devs are from an age where hardware was way less powerful and in the last 20 years, the power and availability of hardware has increased so much that it is hard for us humans to grasp - especially since the machines we use on a day-to-day basis still work nearly the same. Yet, we hear all these stories about hundreds of lambdas serving thousands of requests, so most of devs don't even have an idea of how much a single machine can serve but picked up "oh it's cloud scale" mentality. Combine this with "fear of low resources" and the ease of spinning up a new machine when resources are low and the answer is obvious - spin it up. 5.) We still think linearly. People don't think exponentially most of the time. We don't assume that line of code will net us 30x speed improvements. We don't assume this loop will go quadratic. We tend to ignore tiny accumulations until it's too late. And when met with the performance wall, we discard optimisation because its hard to believe that we can easily 10x or 20x the speed since in our heads "it's just unnatural". You can't change tires on a car and have it go 2000mph suddenly. But with code you can, that's the problem.
sokoloff 5y ago
> they were able to just throw $350k a year at AWS to make it work (and millions more at their dev team) Throwing $350K/yr at AWS sounds like peanuts to me, to be honest. How many extra engineers would you hire if you could completely avoid $350K/yr in expenses? One? There are other, totally valid, reasons to understand your computing environment, but a mid-six figure bill isn’t one of them for anyone with millions in engineer payroll.
- tsimionescu 5y ago
  Your estimation is that one engineer costs ~30k per month?
  - sokoloff 5y ago
    My primary estimation is that one engineer for a startup can create more enterprise value than $350K/yr by working on the product rather than optimizing hosting cost.
  - bostik 5y ago
    In the expensive tech hubs you can expect to pay at least $150k/year in salary for a competent engineer. Add all the admin overhead and other per-employee expenses (1.5x multiplier?) and 350k/year doesn't buy you anywhere near two full-time engineers. But sure - given a tech-heavy environment, $30k/month for a competent, senior engineer with in-demand skillset sounds like it's in the ballpark.
  - argiopetech 5y ago
    I was taught to estimate burdened (i.e., with company's cut of taxes, healthcare, cost of tooling and cubicle space, administration, etc.) engineering time at $100/hr on average. More recently I've learned the average burdening correction is 130% of salary. By these estimations, you get one engineer and $150k leftover (not quite enough for the second,) or two $130k/yr engineers (respectively.) Edit: This is not to imply that 130% is the "correct" number for your industry or area, only that it's what we use in my neck of the woods.
dangus 5y ago
You bring up a good point: solving the business problem is more important than the technical implementation. Software and SaaS is a very high margin business compared to storing widgets in a warehouse and selling them in a retail store. So, you're right: who cares about efficiency? (Well, it is important, but it isn't always the top priority.)
hinkley 5y ago
You have to be careful with Node because being single threaded but highly async, you could be running anywhere from ten requests per process down to three processes per core and one request per process, depending on workload. But I’m in a similar boat, and when I think about our ratio of cores to requests per second, it’s scandalous. When a problem is big and the patterns are set, there’s not much you can do, and the people I flag as potential recruits have a habit of quitting instead of diving onto this grenade. I have learned all new ways that some known (to me) anti patterns can screw you over. And it grates that Tony Hoare said this 40 years ago: I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. If something stays slow it’s because someone worked their butt off making sure that none of the deficiencies were obvious.
hinkley 5y ago
> often an order of magnitude out of whack for companies that prefer a horizontal-first scaling approach This is practically worth its own thread, so I will. I think it’s the principle of the excluded middle biting us on the ass again. Vertical first risks having people assume shared state across events. Everything collects in memory, because look at all of the “unused” memory that nobody will complain about me using for my dumb feature because nobody is looking yet (when the moments are still teachable). To your hardware point, many people don’t understand how that “free” memory is used by the OS to reduce mean response time. So they overload the machine and don’t bother trying to do anything until they hit 80%, when the penalties have started racking up at 60%. So we shall force the issue by giving them tons of tiny machines, as if going to the other extreme ever fixed a goddamned thing. You don’t want tall. You don’t want wide. You want square. I saved us almost 10% on hardware by swapping in larger VMs and fewer of them. We don’t have a homogenous workload, but a good fraction is, so larger machines can share a bit of OS cache and statistical clusters of slow events can be load balanced better, improving p95. But at some point the performance improvements we’ve made will take us in the other direction, because then we’ll have too high a fraction of traffic on a single VM. That’s not a problem by itself, but could be half of a production incident.
novok 5y ago
Plaid is webscraping banks as a service, so I think it was good of them to use node and to priortize dev speed & hiring than to make a well tuned native monolith made by more expensive developers in a stack that had a slower iteration speed. All for a relatively smaller hosting bill in exchange for slower dev speed for something that isn't really compute, io or storage bound at all. You make the well tuned pieces of software when your dropbox writing core i/o bound infra in rust, like magic carpet. What is the cause of the failure rate for plaid? Is it their tech stack, or the fact they are fighting with bank's anti-fraud systems constantly to make their fundamentally fragile web scraping architecture work?
- deleted 5y ago
  [deleted]
KronisLV 5y ago
> Performance efficiency is often an order of magnitude out of whack for companies that prefer a horizontal-first scaling approach (i.e. lots of containers in the cloud, k8s, etc.). This is a curious point, considering that removing the constraint of having to make a single instance work well could lead to relaxed (read: lazy) development practices and therefore suboptimal performance that needs horizontal scaling to remain viable. That said, i also enjoy sleeping during the nights and not getting paged, so running a single instance monolith makes little sense if that's a goal of mine. Horizontal scaling gives people the ability to deal with short term problems that would otherwise be escalated by just bumping up the resources that are available to the deployment and the failure of individual instances is no longer a showstopper. In my eyes, the best approach would be to do both - expect everything to work with just a single instance (with proper integration and load testing, possibly chaos engineering just to be sure), but run multiple instances in practice (with the exact same tests as before, plus additional ones for byzantine failures and unreliable networks being written in addition). Why can't we take best of both worlds - a focus on the quality of engineering, but also as much scalability as needed. Sure, you might disagree, saying that personnel expenses dwarf those needed for just running sub-optimal code on multiple nodes, but my salary is such that you could hire a team of people like me for the cost of 1 developer in the US. And yet, ignorance about the performance and how optimized code is in many "business centric" domains (e.g. web development vs OS development) seems universal. No one wants to pay for nearly perfect code in 5 years, but rather good enough code in 2 years.
cco 5y ago
> This sort of approach sacrifices real-time latency (which is felt by the user)... By far the biggest contributors to latency in user present operations was either latency in the bank's responses or "stuff" required to integrate with the long tail of banks. In fact we found that latency below ~5 or so seconds was _too_ fast for users to trust, they didn't feel confident that it was working if it returned too quickly. As time wore on, latency grew but it wasn't due to technical issues, it was bank caused. > ...not to mention the priceless value of simplicity, a system that can fit in the mind of a single experienced developer... I've got to say that the bank integrations were _amazing_ to work on. The tooling was incredible and it was super easy to work with. If there was a problem with "a bank", you could spin up the entire bank integration stack very simply, took maybe 10 seconds, less if you kept it up to date, and plunk away. Your point stands for much of the rest of the stack, pretty complicated. > ...agility (very slow deployment times) I think it was on the order of single minutes to reach each environment, I don't recall deployment times being onerous. > ...reliability (Plaid is notorious for having a ~20% fail rate for requests) Business decision and the difficulty of building on ever shifting sands (banks change things), the problems are not technical.
- _xnmw 5y ago
  > I've got to say that the bank integrations were _amazing_ to work on. The tooling was incredible and it was super easy to work with. If there was a problem with "a bank", you could spin up the entire bank integration stack very simply, took maybe 10 seconds, less if you kept it up to date, and plunk away. Care to elaborate? From my experience as a Plaid customer, bank integrations seem extremely finicky, unpredictable and buggy. I have support cases going back months which should've been simple fixes (the bug was acknowledged, yet not fixed). > I think it was on the order of single minutes to reach each environment, I don't recall deployment times being onerous. I'm referring to the Plaid blog post linked above, which claimed that deployment times were slow.
  - cco 5y ago
    > Care to elaborate? From my experience as a Plaid customer, bank integrations seem extremely finicky, unpredictable and buggy. I have support cases going back months which should've been simple fixes (the bug was acknowledged, yet not fixed). For some of my time I worked on the support team, so trust me, I feel your pain! I may have even been the one to say, "Sorry, no ETA" :( I share in your frustration with slow bug fixes as well, I was always banging the drum to get more engineers on the problem but it is definitely worth appreciating the scale and complexity. 10,000 banks, each bank has tens to hundred of account types, banks get bought, merged, change their backend. Brokerages also carry a gigantic chunk of complexity, you'd think stock tickers are standardized across brokerages, but absolutely not! And pulling consistent security prices, even just end of day, very complicated. You start to do the scaling math and you quickly realize that you can't fund _every_ bug fix, it just isn't feasible. With that said, do they fund it enough? My opinion is no, more resources should be assigned to fixing bank integrations. But I don't cut the checks ;) re:deploys The deploy times were slow for awhile, I forget for how long, they grew to maybe 20-30 minutes? But for most of my time they were _fast_. And you got a nice Slack message when they hit different envs so you could be doing something else and come back to watch the outcome/test when you got the ping.

by _xnmw 5y ago

p_l 5y ago

Funnily enough, k8s thrives the best when hw is your constraint, in my opinion...

marcos100 5y ago
When you program without performance in mind, hw is always your constraint. k8s is a blessing because you can scale out easily and is a curse because you can scale out easily.
- p_l 5y ago
  Thing is, k8s is not about scaling out. A core part of it is about efficient usage of hw you have (which also maps well to avoid expensive cloud bill). The more I optimized, the more I could leverage k8s to optimize hw usage.
  - adwn 5y ago
    > A core part of it is about efficient usage of hw you have In what way? If the content of your containers is a wasteful, inefficient mess, then there's nothing Kubernetes can do to make it less inefficient.
    - p_l 5y ago
      It can't fix what's inside your containers, but it can fix how efficient you're in packing your workloads onto available hw. I.e. Given a set of Workloads, set of available hw, k8s will help you find maximal amount of Workloads that you can fit. vs. for example manually trying to fit single-purpose VM instances in EC2, with inevitable overheads, etc.
  - marcos100 5y ago
    I agree with you. It will efficiently allocate resources for an inefficient application. But when your system can't handle the load anymore, you'll have to allocate more resources and k8s will handle it for you like magic. It's so easy that people can use thousands of machines to handle the load. This with the "hardware is cheap, dev time is expensive", makes it common to see those kind of very large systems.
    - p_l 5y ago
      Well, yes. Except I get to live with "hardware is more expensive than dev time", so I tend to cheat like a proper dirty cheating wizard with k8s' binpacking ;) (my motto is "what do you call a company using Heroku? Bankrupt" :P)

josephg 5y ago

Yes I absolutely agree. I posted a blog post near the start of the year looking at how much performance automerge (written in javascript using immutablejs) leaves on the table. At the time, automerge took 5 minutes to run a benchmark, and my own code could do the same thing in 0.06 seconds. Since then I’ve sped up my code by another order of magnitude - it can now run the same benchmark in 0.005s. (4.9ms). I don’t want to badmouth automerge - they’re working on it and have made big improvements this year. But that ratio - 5min:5ms - gives you a sense of the mountain of inefficiency we live with on modern computers with a lot of modern software. Modern CPUs and ram are obscenely, incomprehensibly fast. With the exception of a few very specific tasks, you should never be waiting for a modern computer to do anything. We sent men to the moon in fewer cycles than it takes for modern windows to boot, by several orders of magnitude. Complaining about the global chip shortage is a bit like a hoarder complaining about a room shortage in their overflowing house. (With carve outs in my scorn for video editing, AAA game engines, AI, etc)

bob1029 5y ago
> Modern CPUs and ram are obscenely, incomprehensibly fast. With the exception of a few very specific tasks, you should never be waiting for a modern computer to do anything. I really wish we could drive this point home for everyone. The amount of work you can get out of 1 x86 core is insane if you know how to feed it properly. The upper bound is somewhere around 500 million transactions per second per core. I have personally built systems that can achieve somewhere around 14 million per second. If you can reliably handle the entire serialized workload on a single thread you start to retool for the idea that you can fit the entire business in 1 box. Being able to run everything out of a single computer is an incredible virtue, not some curse that must be cast to the clouds.
- DaiPlusPlus 5y ago
  > 500 million transactions per second per core. An instruction is not a transaction. Commodity x86/x64 systems do not implement transactional memory or any model of computation that could be reasonably described as transactional as far as I'm aware.

perryizgr8 5y ago

> 4000 Node containers [1] running in parallel to handle just as many requests per second. This looks crazy to me at first glance. 4k requests per second, for any reasonable sized request, should be easily handled by a single C thread. With plenty of room to grow.

_xnmw 5y ago
You're not the only one who found it insane. Check the HN discussion at the time: https://news.ycombinator.com/item?id=21782196 https://news.ycombinator.com/item?id=21782196

gjulianm 5y ago

When I see these stories about horizontal scaling on thousands of nodes, I wonder how many people don't know how much performance can be squeezed out of beefy machines. At my job we build probes for network monitoring, and we're able to ingest quite a lot of traffic (up to 14Mpps per link, for a fully saturated 10G link in our hardest tests), ingest a good volume logs or metrics from remote systems, do some extra processing and serve a web application in just a single instance. We use C/C++ for the traffic capture and processing, Postgres/Elastic for storage (depending on the balance performance/flexibility), some AWK for quick file processing (you'd be surprised how fast can it be) and then Python for plumbing code. On the other hand, I've seen technical blogposts and whitepapers on how to optimize a "big data ingestion system" that requires clustering and tweaking and whatnot for a fraction of what we're able to do with a single machine. People should try harder to scale up before starting a cluster and scaling out.

christophilus 5y ago
I was in a discussion on HN just yesterday where the assertion was made that startups should default to the cloud because the upfront cost of allocating real hardware is too high. I’ve got a $20 Netcup server that can probably handle the traffic of 90% of any projects I’ve ever worked on (including my stint at big tech cos). It’s amazing how cheap and fast modern computing is when compared to the standard cloud offerings.
- CogitoCogito 5y ago
  If they don't want to pay upfront for hardware, they can also "default to cloud" while just using cloud-based compute instances.
- fullstackchris 5y ago
  ... but netcup is just another cloud offering? What are you getting at here? I mean, it always makes sense to shop around and look at pricings before choosing one, but my unresearched guess is that there are so many providers nowadays that the pricings across all of them are within cents / dollars for the same service (based on RAM, storage, etc)

thecupisblue 5y ago

It's a multidimensional problem. 1.) Rush to build. Backpressure and deadlines lead to "lets just spin it up thje easier way" 2.) We said "premature optimisation is the root of all evil". This ended up in devs not thinking about performance at all because "hurr durr" premature optimisation. I've witnessed this mentality causing huge problems in performance due to simplest decisions that would be normal in other circumstances, but snowball into a huge perf issue. Exactly as the title says, best practices can slow your application down, especially when used out of context. Some include "I'll serialise this into json and deserialise it again 5 lines later", "I'll use a regex to parse this simple string", "I'll nest 3 map calls here on a list", "I'll find it later in the list again". 3.) People forgot/never learned what the stack is. Honestly, when people say software was harder to write back in the day, I'm never sure if I agree with it or not. One one hand, yes, we had less tools and materials at our disposal. But at least the thinking through it was easier, since with the small set of tools you have, you could go up and down the stack in your head and know that if you do X, Y and Z, that's what would happen. Nowadays, we have so many tools and layers that it takes a huge amount of knowledge, time and mental energy to reason about your whole stack. We have people specialised in tiny subsets of the stack, so getting performance squeezed out at overall stack level requires people who understand more of the stack, which is getting rarer and rarer by day. 4.) We have lost touch with the power of hardware. Consider that most of senior devs are from an age where hardware was way less powerful and in the last 20 years, the power and availability of hardware has increased so much that it is hard for us humans to grasp - especially since the machines we use on a day-to-day basis still work nearly the same. Yet, we hear all these stories about hundreds of lambdas serving thousands of requests, so most of devs don't even have an idea of how much a single machine can serve but picked up "oh it's cloud scale" mentality. Combine this with "fear of low resources" and the ease of spinning up a new machine when resources are low and the answer is obvious - spin it up. 5.) We still think linearly. People don't think exponentially most of the time. We don't assume that line of code will net us 30x speed improvements. We don't assume this loop will go quadratic. We tend to ignore tiny accumulations until it's too late. And when met with the performance wall, we discard optimisation because its hard to believe that we can easily 10x or 20x the speed since in our heads "it's just unnatural". You can't change tires on a car and have it go 2000mph suddenly. But with code you can, that's the problem.

sokoloff 5y ago

> they were able to just throw $350k a year at AWS to make it work (and millions more at their dev team) Throwing $350K/yr at AWS sounds like peanuts to me, to be honest. How many extra engineers would you hire if you could completely avoid $350K/yr in expenses? One? There are other, totally valid, reasons to understand your computing environment, but a mid-six figure bill isn’t one of them for anyone with millions in engineer payroll.

tsimionescu 5y ago
Your estimation is that one engineer costs ~30k per month?
- sokoloff 5y ago
  My primary estimation is that one engineer for a startup can create more enterprise value than $350K/yr by working on the product rather than optimizing hosting cost.
- bostik 5y ago
  In the expensive tech hubs you can expect to pay at least $150k/year in salary for a competent engineer. Add all the admin overhead and other per-employee expenses (1.5x multiplier?) and 350k/year doesn't buy you anywhere near two full-time engineers. But sure - given a tech-heavy environment, $30k/month for a competent, senior engineer with in-demand skillset sounds like it's in the ballpark.
- argiopetech 5y ago
  I was taught to estimate burdened (i.e., with company's cut of taxes, healthcare, cost of tooling and cubicle space, administration, etc.) engineering time at $100/hr on average. More recently I've learned the average burdening correction is 130% of salary. By these estimations, you get one engineer and $150k leftover (not quite enough for the second,) or two $130k/yr engineers (respectively.) Edit: This is not to imply that 130% is the "correct" number for your industry or area, only that it's what we use in my neck of the woods.

dangus 5y ago

You bring up a good point: solving the business problem is more important than the technical implementation. Software and SaaS is a very high margin business compared to storing widgets in a warehouse and selling them in a retail store. So, you're right: who cares about efficiency? (Well, it is important, but it isn't always the top priority.)

hinkley 5y ago

You have to be careful with Node because being single threaded but highly async, you could be running anywhere from ten requests per process down to three processes per core and one request per process, depending on workload. But I’m in a similar boat, and when I think about our ratio of cores to requests per second, it’s scandalous. When a problem is big and the patterns are set, there’s not much you can do, and the people I flag as potential recruits have a habit of quitting instead of diving onto this grenade. I have learned all new ways that some known (to me) anti patterns can screw you over. And it grates that Tony Hoare said this 40 years ago: I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. If something stays slow it’s because someone worked their butt off making sure that none of the deficiencies were obvious.

hinkley 5y ago

> often an order of magnitude out of whack for companies that prefer a horizontal-first scaling approach This is practically worth its own thread, so I will. I think it’s the principle of the excluded middle biting us on the ass again. Vertical first risks having people assume shared state across events. Everything collects in memory, because look at all of the “unused” memory that nobody will complain about me using for my dumb feature because nobody is looking yet (when the moments are still teachable). To your hardware point, many people don’t understand how that “free” memory is used by the OS to reduce mean response time. So they overload the machine and don’t bother trying to do anything until they hit 80%, when the penalties have started racking up at 60%. So we shall force the issue by giving them tons of tiny machines, as if going to the other extreme ever fixed a goddamned thing. You don’t want tall. You don’t want wide. You want square. I saved us almost 10% on hardware by swapping in larger VMs and fewer of them. We don’t have a homogenous workload, but a good fraction is, so larger machines can share a bit of OS cache and statistical clusters of slow events can be load balanced better, improving p95. But at some point the performance improvements we’ve made will take us in the other direction, because then we’ll have too high a fraction of traffic on a single VM. That’s not a problem by itself, but could be half of a production incident.

novok 5y ago

Plaid is webscraping banks as a service, so I think it was good of them to use node and to priortize dev speed & hiring than to make a well tuned native monolith made by more expensive developers in a stack that had a slower iteration speed. All for a relatively smaller hosting bill in exchange for slower dev speed for something that isn't really compute, io or storage bound at all. You make the well tuned pieces of software when your dropbox writing core i/o bound infra in rust, like magic carpet. What is the cause of the failure rate for plaid? Is it their tech stack, or the fact they are fighting with bank's anti-fraud systems constantly to make their fundamentally fragile web scraping architecture work?

deleted 5y ago
[deleted]

KronisLV 5y ago

> Performance efficiency is often an order of magnitude out of whack for companies that prefer a horizontal-first scaling approach (i.e. lots of containers in the cloud, k8s, etc.). This is a curious point, considering that removing the constraint of having to make a single instance work well could lead to relaxed (read: lazy) development practices and therefore suboptimal performance that needs horizontal scaling to remain viable. That said, i also enjoy sleeping during the nights and not getting paged, so running a single instance monolith makes little sense if that's a goal of mine. Horizontal scaling gives people the ability to deal with short term problems that would otherwise be escalated by just bumping up the resources that are available to the deployment and the failure of individual instances is no longer a showstopper. In my eyes, the best approach would be to do both - expect everything to work with just a single instance (with proper integration and load testing, possibly chaos engineering just to be sure), but run multiple instances in practice (with the exact same tests as before, plus additional ones for byzantine failures and unreliable networks being written in addition). Why can't we take best of both worlds - a focus on the quality of engineering, but also as much scalability as needed. Sure, you might disagree, saying that personnel expenses dwarf those needed for just running sub-optimal code on multiple nodes, but my salary is such that you could hire a team of people like me for the cost of 1 developer in the US. And yet, ignorance about the performance and how optimized code is in many "business centric" domains (e.g. web development vs OS development) seems universal. No one wants to pay for nearly perfect code in 5 years, but rather good enough code in 2 years.

cco 5y ago

> This sort of approach sacrifices real-time latency (which is felt by the user)... By far the biggest contributors to latency in user present operations was either latency in the bank's responses or "stuff" required to integrate with the long tail of banks. In fact we found that latency below ~5 or so seconds was _too_ fast for users to trust, they didn't feel confident that it was working if it returned too quickly. As time wore on, latency grew but it wasn't due to technical issues, it was bank caused. > ...not to mention the priceless value of simplicity, a system that can fit in the mind of a single experienced developer... I've got to say that the bank integrations were _amazing_ to work on. The tooling was incredible and it was super easy to work with. If there was a problem with "a bank", you could spin up the entire bank integration stack very simply, took maybe 10 seconds, less if you kept it up to date, and plunk away. Your point stands for much of the rest of the stack, pretty complicated. > ...agility (very slow deployment times) I think it was on the order of single minutes to reach each environment, I don't recall deployment times being onerous. > ...reliability (Plaid is notorious for having a ~20% fail rate for requests) Business decision and the difficulty of building on ever shifting sands (banks change things), the problems are not technical.

_xnmw 5y ago
> I've got to say that the bank integrations were _amazing_ to work on. The tooling was incredible and it was super easy to work with. If there was a problem with "a bank", you could spin up the entire bank integration stack very simply, took maybe 10 seconds, less if you kept it up to date, and plunk away. Care to elaborate? From my experience as a Plaid customer, bank integrations seem extremely finicky, unpredictable and buggy. I have support cases going back months which should've been simple fixes (the bug was acknowledged, yet not fixed). > I think it was on the order of single minutes to reach each environment, I don't recall deployment times being onerous. I'm referring to the Plaid blog post linked above, which claimed that deployment times were slow.
- cco 5y ago
  > Care to elaborate? From my experience as a Plaid customer, bank integrations seem extremely finicky, unpredictable and buggy. I have support cases going back months which should've been simple fixes (the bug was acknowledged, yet not fixed). For some of my time I worked on the support team, so trust me, I feel your pain! I may have even been the one to say, "Sorry, no ETA" :( I share in your frustration with slow bug fixes as well, I was always banging the drum to get more engineers on the problem but it is definitely worth appreciating the scale and complexity. 10,000 banks, each bank has tens to hundred of account types, banks get bought, merged, change their backend. Brokerages also carry a gigantic chunk of complexity, you'd think stock tickers are standardized across brokerages, but absolutely not! And pulling consistent security prices, even just end of day, very complicated. You start to do the scaling math and you quickly realize that you can't fund _every_ bug fix, it just isn't feasible. With that said, do they fund it enough? My opinion is no, more resources should be assigned to fixing bank integrations. But I don't cut the checks ;) re:deploys The deploy times were slow for awhile, I forget for how long, they grew to maybe 20-30 minutes? But for most of my time they were _fast_. And you got a nice Slack message when they hit different envs so you could be doing something else and come back to watch the outcome/test when you got the ping.