3 ms·

Nobody involved in this project should be allowed to ever be in the same room as a computer again.

jrockway 7y ago
Why? They had a 12 factor -ish app that scaled the normal way; run more copies. Eventually that got expensive. They had the observability to figure out what was making it expensive and whether or not their fixes had an effect. They then saved $300,000. Seems like everything went right to me. I would be worried if the blog post was "we randomly tweaked some stuff and we can't measure it but it's a little better" or "we rewrote it in go and in the rewrite introduced 87 new bugs while fixing 42 old bugs". They engineered a solution, built from good investment in infrastructure, rather than ninja-ing a hack. That, to me, is a very good thing. A lot of people seem deeply upset that Node was involved, but I think that's a red herring. The problem they had -- allocate a large chunk of memory, keep a reference to it while it is slowly sent to another server, free memory -- is going to happen in any language. (I don't super agree with their solution of "make the server faster" because one day it's going to be slow for some other reason and this problem will crop up again. Instead they probably just need a fixed amount of memory to dedicate to this process and to drop the debug payload when the buffer is full. Or just put it in the request path if it's crucial that it be produced every time no matter what. At least that will apply backpressure to calling services, pop the circuit breaker, and redirect requests to a region where S3 isn't broken. But I don't think the debug information is THAT important ;)
- PixyMisa 7y ago
  To save $300,000 they first needed to waste $300,000 by reinventing a problem that was solved in 1967.
  - dwild 7y ago
    I don't know how many software engineer they got on that team, but considering how much they raised and how much their product is used, $300k seems actually quite cheap for something that people consider here as being an awfully big mistake.
    - ubu7737 7y ago
      Unfortunately it never works out the way you hope it will, once the VC money shows up. ROI will become even more of a pattern for everything, which means you burn the existing platform to dust. Or you hire a team of actual engineers to turn the PoC into an actual product. My judgment tells me the management at Plaid doesn't know how to tell the difference, so it's sort of a determined outcome.
- ubu7737 7y ago
  The fundamental issue is that they are using a broken pattern for granularity of work. Any idiot can put together a lot of containers and farm work out, that's how Apache/CGI worked for years. But it's not scalable. They didn't understand the language runtime well enough, so they punted the problem to parallelism. At scale, they are now repairing the model to "save money" on what was essentially a no-architecture solution.
- GordonS 7y ago
  > Why? They had a 12 factor -ish app that scaled the normal way So, yes, horizontal scaling is good, especially for stateless workloads - but that doesn't mean you run the most hopelessly under-performing code imaginable on each node, so you basically have to scale out like this! I mean, seriously, 4000 containers to serve 4000 concurrent requests? I mean, I can't even... I honestly can't believe the attempts in this thread to justify such an utterly, horrendously bad architecture - there are 1001 better, simpler even, ways to approach this. Yes, premature optimisation is bad, but optimisation here was nowhere near premature.
  - jrockway 7y ago
    I'm going to disagree. When you start a business, you have no idea what it's going to grow into, or if it's going to grow. So you start simple. The design was good enough for there to one day be too many customers. That is huge. When this happened, they started a second copy of their app, and could now handle twice as many customers. Repeat 3998 more times. Now the toy app is making some real money, so you can afford to deep-dive into the system and fix the technical problems. They avoided the real issue that kills startups, having a customer call you because they want to buy your service and you saying "sorry, we aren't accepting any new customers right now because Hacker News comments don't like our software architectures."
bdcravens 7y ago
I would say the same thing about hiring people who make snide dismissals.
dang 7y ago
This comment breaks the site guidelines and is not cool. Would you please read https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

Why? They had a 12 factor -ish app that scaled the normal way; run more copies. Eventually that got expensive. They had the observability to figure out what was making it expensive and whether or not their fixes had an effect. They then saved $300,000. Seems like everything went right to me. I would be worried if the blog post was "we randomly tweaked some stuff and we can't measure it but it's a little better" or "we rewrote it in go and in the rewrite introduced 87 new bugs while fixing 42 old bugs". They engineered a solution, built from good investment in infrastructure, rather than ninja-ing a hack. That, to me, is a very good thing. A lot of people seem deeply upset that Node was involved, but I think that's a red herring. The problem they had -- allocate a large chunk of memory, keep a reference to it while it is slowly sent to another server, free memory -- is going to happen in any language. (I don't super agree with their solution of "make the server faster" because one day it's going to be slow for some other reason and this problem will crop up again. Instead they probably just need a fixed amount of memory to dedicate to this process and to drop the debug payload when the buffer is full. Or just put it in the request path if it's crucial that it be produced every time no matter what. At least that will apply backpressure to calling services, pop the circuit breaker, and redirect requests to a region where S3 isn't broken. But I don't think the debug information is THAT important ;)

PixyMisa 7y ago
To save $300,000 they first needed to waste $300,000 by reinventing a problem that was solved in 1967.
- dwild 7y ago
  I don't know how many software engineer they got on that team, but considering how much they raised and how much their product is used, $300k seems actually quite cheap for something that people consider here as being an awfully big mistake.
  - ubu7737 7y ago
    Unfortunately it never works out the way you hope it will, once the VC money shows up. ROI will become even more of a pattern for everything, which means you burn the existing platform to dust. Or you hire a team of actual engineers to turn the PoC into an actual product. My judgment tells me the management at Plaid doesn't know how to tell the difference, so it's sort of a determined outcome.
ubu7737 7y ago
The fundamental issue is that they are using a broken pattern for granularity of work. Any idiot can put together a lot of containers and farm work out, that's how Apache/CGI worked for years. But it's not scalable. They didn't understand the language runtime well enough, so they punted the problem to parallelism. At scale, they are now repairing the model to "save money" on what was essentially a no-architecture solution.
GordonS 7y ago
> Why? They had a 12 factor -ish app that scaled the normal way So, yes, horizontal scaling is good, especially for stateless workloads - but that doesn't mean you run the most hopelessly under-performing code imaginable on each node, so you basically have to scale out like this! I mean, seriously, 4000 containers to serve 4000 concurrent requests? I mean, I can't even... I honestly can't believe the attempts in this thread to justify such an utterly, horrendously bad architecture - there are 1001 better, simpler even, ways to approach this. Yes, premature optimisation is bad, but optimisation here was nowhere near premature.
- jrockway 7y ago
  I'm going to disagree. When you start a business, you have no idea what it's going to grow into, or if it's going to grow. So you start simple. The design was good enough for there to one day be too many customers. That is huge. When this happened, they started a second copy of their app, and could now handle twice as many customers. Repeat 3998 more times. Now the toy app is making some real money, so you can afford to deep-dive into the system and fix the technical problems. They avoided the real issue that kills startups, having a customer call you because they want to buy your service and you saying "sorry, we aren't accepting any new customers right now because Hacker News comments don't like our software architectures."

bdcravens 7y ago

I would say the same thing about hiring people who make snide dismissals.

dang 7y ago

This comment breaks the site guidelines and is not cool. Would you please read https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html