5 ms·

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering pract

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for an API workload is absolute insanity. These engineers are building stuff for banking. Banking!! There is literally no way I'm going near Plaid with a very long bargepole after reading this. It I was someone senior at Plaid, I'd be pulling this blog post before it harms reputation any further.

sicromoft 7y ago
This comment says more about you than it does about Plaid. Their "insane" design met business requirements successfully enough to grow them into a multi-billion dollar company. Did you consider the likely (and more charitable) explanation that they were aware their design was "bad", but had higher priorities until now? If I were you, I'd be pulling your comment before it harms your reputation any further. :)
- GordonS 7y ago
  > Did you consider the likely (and more charitable) explanation that they were aware their design was "bad" I think marketing and VC valuations grew them into a multi-billion dollar company; whether they remain so, to a large part relies on how fast they burn through VC cash - so, not looking too good on that front... No even half-way competent engineer would come up with such a complex, unperformant solution to a simple problem - I think a higher priority should be hiring engineers who actually have a clue what they're doing. As for meeting business requirements... while this might have worked for a while, it was plainly not a good way to meet them, and given Plaid are in the banking sector, really doesn't bode well for the future (I'm having flashforwards already to security breaches, plaintext passwords etc...).
  - ubu7737 7y ago
    Downvoted for angering the VC class...
- ubu7737 7y ago
  > If I were you, I'd be pulling your comment before it harms your reputation any further. :) This is the douchiest thing I've seen all day. This trust-fund kid thinks he's Donald John Trump. It is flat-out hilarious to hear that a supposed tech company had more pressing concerns than technology.
- rockostrich 7y ago
  >multi-billion dollar company WeWork is a "multi-billion dollar company" in the same way that Plaid is. Private funding valuations don't really mean anything anymore.
RSZC 7y ago
It's important to keep in mind that efficiency isn't usually particularly important for a startup. I'm sure they knew when they initially set up this system that it wasn't performant...but it was nice and quick and easy and gets the feature out the door. Why should they worry about $100k or whatever when they're funded for > $350M? Their bottleneck is engineer hours, not dollars. Instead the rational thing to do is build something quick and dirty and optimize later, and that's exactly what they've done.
- GordonS 7y ago
  I understand that, and plenty times myself I've "done the simplest" thing - sometimes you need to ship an MVP, fast. The difference here is that what they did wasn't even the simplest thing - it was a crazy, insanely wasteful thing that just happened to work for a while. Being honest, for me, it's an indefensible approach. > Why should they worry about $100k or whatever when they're funded for > $350M? Their bottleneck is engineer hours, not dollars Arg, but this rubs me up the wrong way! Any half-way competent engineer could have built something simpler and much more performant, and likely in many less hours too. Sometimes stopping, thinking and discussing for a few minutes or hours will save numerous hours. I mean, how many hours did they spend on this "diagnosis" alone?
- rockostrich 7y ago
  >Their bottleneck is engineer hours, not dollars. Their bottleneck was software being able to scale past a hard stop. I guess having a known breaking point of scalability is a good thing? But building things in a way where you either have to overhaul your development runtime or not be able to scale past a certain point is pretty terrible. It seems like the only reason they did this was because they really felt the pain of it from the business and dev side and they were lucky enough that they had traffic spikes to raise these issues. If they had more consistent day-to-day traffic then this would have just hit a breaking point one day and they would've been fucked until it was fixed.
yjftsjthsd-h 7y ago
I mean, that's always the thing, isn't it? If a company publishes about the problems it has, the question is whether other companies have the same problems and just hide it, or whether this company is actually worse. This comes up a lot with gitlab, for instance; remember the time they discovered they had no backups? At most companies, customers would never find out about that, so I'm not sure that them telling us about it usefully informs my view of their competence. Similarly, here, the only way we'd know about this if they didn't say anything would be poor performance, which would be... less than surprising, on a financial website, in my experience. So maybe they suck, or maybe they're equal with others but a little more open; I don't know how to tell.
- GordonS 7y ago
  Thanks for a reasoned response to what I realise was a very negative comment. I do agree with what you've said, and I do feel a little bad for slamming them when they're being transparent. OTOH, I do still feel this is so bad they need to be called out on it, and it really does scare me off using them. Given they're being transparent, it boggles the mind that they're tried to justify this, rather than just owning it, admitting it was the result of letting a junior do some resumed-driven-developlemt (or however it came about).
  - rockostrich 7y ago
    I posted it on my engineering org's random channel. The 4,000 instances of the same service thing immediately got a laugh out of everyone. How a tech company operated like this is beyond me...
    - GordonS 7y ago
      Hah, I actually did the same, and it was basically a stream of WTFs?! To put a positive spin on this (perhaps a first for me in this thread!), I plan on doing something of an internal post-mortem with my team, where we'll look at the deficiencies of this design, try to reason about how on earth it came to fruition, and critique our in-place review processes to make sure something like this never happens to us.
kevstev 7y ago
I kind of alluded to it in my reply, but I tend to agree- they spent a lot of time and hard work- looking in all the wrong places! Its hard to imagine how they missed the forest for the trees so badly here. Worse is- they never really explain where that 30x improvement came from- or if they even understand it themselves? They talk a lot about getting their memory issues under control, but hardly at all about actual parallelism- and it seems that even then they confuse it with merely speeding up operations that are blocking. I kind of expected this post to be "We did a whoops and had a blocking call to a DB/fs/compression call/whatever. This was all happening in the event loop and not being farmed out to the threadpool by libuv. We fixed it and now look like heroes to our CTO!"
- tracker1 7y ago
  What they talk about are issues that blocked them from parallelism per node and how they resolved the issues. I'm not sure what additional information you're expecting? Though I'm somewhat surprised they didn't use Worker patterns per node with self monitoring for health above and beyond what they already did.
mharroun 7y ago
In their defense. It looks like they have over 400 employees and raised over 350 million in funding. On all things that truly matter currently they seem like a very sucessfull company. I can guarantee you a VPE or CTO who can say they helped do that... but ran into a scaling issue from their success will have no issue with employment and no reason to be ashamed. All the more impressive if it was just a bunch of junior engineers.
- ubu7737 7y ago
  "In their defense: money they've raised"? Is this supposed to be a contribution to the thread?
throwGuardian 7y ago
> no way I'm going near Plaid with a very long bargepole after reading this But you'd go to a competitor who hasn't published a blog post, whose internal code you haven't audited and simply presume is just fine? In plaid's defense, lack of performance tuning isn't necessarily a lack of security focus.
- GordonS 7y ago
  > In plaid's defense, lack of performance tuning isn't necessarily a lack of security focus Come on, this is not about "performance tuning", where you're trying to eek out every last drop of performance - it's about a completely indefensible, complex, wasteful solution to a simple problem. I'd say engineering insanity at this level is very worrisome for what they've done at the security side of things.
- ubu7737 7y ago
  ROFL "performance tuning" this is not tuning this is architecture. Some people think you can just write software, sell it to customers, and it's "tuning" to make it work properly. You should be fired from whatever job you have. My guess is that you have no job, you are fronting USD. In which case you have absolutely no place in this conversation and you should be ashamed of yourself for speaking up. A fool and his money are easily parted.
  - dang 7y ago
    We've banned this account for breaking the site guidelines and ignoring our request to stop. If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - deleted 7y ago
    [deleted]
tracker1 7y ago
For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. The simplest solution is to scale to one worker per node initially if you're doing anything compute intensive... once you've done that, and/or you need better performance for any number of reasons including cost, then you can do more. Now, I'm not sure I would have gotten to 4k nodes before I started to re-evaluate parallelism or better scaling options, but the initial implementation is absolutely fine.
- GordonS 7y ago
  > For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. I get it, but come on - this was not a "performance optimisation" issue, but one of bad architecture; an architecture that certainly doesn't inspire confidence in the priorities you mention: accuracy, simplicity, safety.
  - tracker1 7y ago
    Having a worker that does one request, processes that one request and returns a result isn't accurate, simple or safe? Scaling that simplistic interface in a 1:1 manner across many systems via ECS is pretty straight forward. Now, in addition to probably optimizing what they've done, converting to, for example another container system, like K8s where they can scale vertically a bit better may have been another approach. The biggest issue that I see is gRPC doesn't work that great with Node. You can't use it with cluster which means you have to self-manage threads/processes and it adds complexity there. Yeah, there's definitely issues that come into scaling in terms of performance optimization.... but where they started from isn't unreasonable imho.
- ubu7737 7y ago
  Where I work compliance is job #1. That doesn't prevent us from thinking about performance. GTFO with this nonsense.
mirekrusin 7y ago
Frankly the reality is so surreal there, it's actually surprising it doesn't mention uploading csv files over ftp, generating excel files, having cameras pointing on monitors of legacy systems that read data (no, this is not a joke) or spawning a promise for Martha to cross check something and click ok somewhere behind two bastion hosts, three firewalls and one and a half soap integrations. I'm not defending "4k containers because even loop can be blocked" which is silly, just reminding of the context - in other words you can do shittiest automated thing there and you're a hero. Next year-or-two hero is going to be somebody shrinking it by another X orders of magnitude.
bjacokes 7y ago
Hi, Plaid engineer here (not the author, but I helped with the post). I don't think we've tried to assert that the old system is perfect. We went into some detail in the post about why it took us this far. Certainly, the single request per container approach wouldn't scale if our unit economics were different. We didn't get into this too much in the post, but the Node service sits behind a couple of layers of Go services, so the we had more control over scaling API traffic than it might appear. Likewise, I hope we didn't give the impression that the new system is perfect. We've explored other languages for integrations in the past (even Haskell, at one point), and are continuing to do so. A migration away from our years-old Node integrations codebase would be a massive undertaking at this point. Absent that, it doesn't seem consistent to say "you're incompetent for handling 1 request per container" and also "you're incompetent for writing this post" – if you believe the former then it makes sense to be an advocate for this project, at least until a language migration can be done. I think the set of hoops we had to jump through in order to add concurrent requests without adding latency is a good demonstration of why we didn't do this sooner. It wasn't a massive undertaking by any means, but it wasn't trivial. At any rate, we're not really looking for a gold star here – just putting this out there and hoping this will be useful for others who are, as other commenters have put it, building their own "Frankensteins" :)
- danudey 7y ago
  I mean, my read of this is: 1. We used a system which uses event loops to achieve great concurrency, but we turned that off because we don't trust it. 2. Instead, we spent $300k/yr rolling out one-process-per-API as though we were using Apache 1.3. 3. We used an arbitrary JSON library without knowing anything about its performance characteristics, which it turns out were inordinately bad It's not that this wasn't a great exercise in engineering and problem-solving, or that it's not a great demonstration of how to solve scaling problems at scale, those are definitely true. It's more that "we spent $300k/yr more than we needed to so our engineers didn't need to learn how to use our technology stack properly." I'm not meaning to be harsh, I've kludged enough garbage into production in my lifetime, but more that the fact that you got into that situation in the first place gives a poor impression of either your development team or your development processes.
z3t4 7y ago
Linus Torvalds quote: "You need to grow thick skin". I also got a knee-jerk reaction when reading the first part, but the article explained it well, given that they probably don't want to give out too much information. So how would you have engineered it? I would just send the data uncompressed granted that the receiving server is probably in the same data-center with switches capable of handling Tbit's of data per second. I liked the article, but would have wanted more details. I love optimizations, it's such a drug, the rush when you make something x times faster. This article doesn't give me a bad impression. Contrary I'm thinking about sending an application.

sicromoft 7y ago

This comment says more about you than it does about Plaid. Their "insane" design met business requirements successfully enough to grow them into a multi-billion dollar company. Did you consider the likely (and more charitable) explanation that they were aware their design was "bad", but had higher priorities until now? If I were you, I'd be pulling your comment before it harms your reputation any further. :)

GordonS 7y ago
> Did you consider the likely (and more charitable) explanation that they were aware their design was "bad" I think marketing and VC valuations grew them into a multi-billion dollar company; whether they remain so, to a large part relies on how fast they burn through VC cash - so, not looking too good on that front... No even half-way competent engineer would come up with such a complex, unperformant solution to a simple problem - I think a higher priority should be hiring engineers who actually have a clue what they're doing. As for meeting business requirements... while this might have worked for a while, it was plainly not a good way to meet them, and given Plaid are in the banking sector, really doesn't bode well for the future (I'm having flashforwards already to security breaches, plaintext passwords etc...).
- ubu7737 7y ago
  Downvoted for angering the VC class...
ubu7737 7y ago
> If I were you, I'd be pulling your comment before it harms your reputation any further. :) This is the douchiest thing I've seen all day. This trust-fund kid thinks he's Donald John Trump. It is flat-out hilarious to hear that a supposed tech company had more pressing concerns than technology.
rockostrich 7y ago
>multi-billion dollar company WeWork is a "multi-billion dollar company" in the same way that Plaid is. Private funding valuations don't really mean anything anymore.

RSZC 7y ago

It's important to keep in mind that efficiency isn't usually particularly important for a startup. I'm sure they knew when they initially set up this system that it wasn't performant...but it was nice and quick and easy and gets the feature out the door. Why should they worry about $100k or whatever when they're funded for > $350M? Their bottleneck is engineer hours, not dollars. Instead the rational thing to do is build something quick and dirty and optimize later, and that's exactly what they've done.

GordonS 7y ago
I understand that, and plenty times myself I've "done the simplest" thing - sometimes you need to ship an MVP, fast. The difference here is that what they did wasn't even the simplest thing - it was a crazy, insanely wasteful thing that just happened to work for a while. Being honest, for me, it's an indefensible approach. > Why should they worry about $100k or whatever when they're funded for > $350M? Their bottleneck is engineer hours, not dollars Arg, but this rubs me up the wrong way! Any half-way competent engineer could have built something simpler and much more performant, and likely in many less hours too. Sometimes stopping, thinking and discussing for a few minutes or hours will save numerous hours. I mean, how many hours did they spend on this "diagnosis" alone?
rockostrich 7y ago
>Their bottleneck is engineer hours, not dollars. Their bottleneck was software being able to scale past a hard stop. I guess having a known breaking point of scalability is a good thing? But building things in a way where you either have to overhaul your development runtime or not be able to scale past a certain point is pretty terrible. It seems like the only reason they did this was because they really felt the pain of it from the business and dev side and they were lucky enough that they had traffic spikes to raise these issues. If they had more consistent day-to-day traffic then this would have just hit a breaking point one day and they would've been fucked until it was fixed.

yjftsjthsd-h 7y ago

I mean, that's always the thing, isn't it? If a company publishes about the problems it has, the question is whether other companies have the same problems and just hide it, or whether this company is actually worse. This comes up a lot with gitlab, for instance; remember the time they discovered they had no backups? At most companies, customers would never find out about that, so I'm not sure that them telling us about it usefully informs my view of their competence. Similarly, here, the only way we'd know about this if they didn't say anything would be poor performance, which would be... less than surprising, on a financial website, in my experience. So maybe they suck, or maybe they're equal with others but a little more open; I don't know how to tell.

GordonS 7y ago
Thanks for a reasoned response to what I realise was a very negative comment. I do agree with what you've said, and I do feel a little bad for slamming them when they're being transparent. OTOH, I do still feel this is so bad they need to be called out on it, and it really does scare me off using them. Given they're being transparent, it boggles the mind that they're tried to justify this, rather than just owning it, admitting it was the result of letting a junior do some resumed-driven-developlemt (or however it came about).
- rockostrich 7y ago
  I posted it on my engineering org's random channel. The 4,000 instances of the same service thing immediately got a laugh out of everyone. How a tech company operated like this is beyond me...
  - GordonS 7y ago
    Hah, I actually did the same, and it was basically a stream of WTFs?! To put a positive spin on this (perhaps a first for me in this thread!), I plan on doing something of an internal post-mortem with my team, where we'll look at the deficiencies of this design, try to reason about how on earth it came to fruition, and critique our in-place review processes to make sure something like this never happens to us.

kevstev 7y ago

I kind of alluded to it in my reply, but I tend to agree- they spent a lot of time and hard work- looking in all the wrong places! Its hard to imagine how they missed the forest for the trees so badly here. Worse is- they never really explain where that 30x improvement came from- or if they even understand it themselves? They talk a lot about getting their memory issues under control, but hardly at all about actual parallelism- and it seems that even then they confuse it with merely speeding up operations that are blocking. I kind of expected this post to be "We did a whoops and had a blocking call to a DB/fs/compression call/whatever. This was all happening in the event loop and not being farmed out to the threadpool by libuv. We fixed it and now look like heroes to our CTO!"

tracker1 7y ago
What they talk about are issues that blocked them from parallelism per node and how they resolved the issues. I'm not sure what additional information you're expecting? Though I'm somewhat surprised they didn't use Worker patterns per node with self monitoring for health above and beyond what they already did.

mharroun 7y ago

In their defense. It looks like they have over 400 employees and raised over 350 million in funding. On all things that truly matter currently they seem like a very sucessfull company. I can guarantee you a VPE or CTO who can say they helped do that... but ran into a scaling issue from their success will have no issue with employment and no reason to be ashamed. All the more impressive if it was just a bunch of junior engineers.

ubu7737 7y ago
"In their defense: money they've raised"? Is this supposed to be a contribution to the thread?

throwGuardian 7y ago

> no way I'm going near Plaid with a very long bargepole after reading this But you'd go to a competitor who hasn't published a blog post, whose internal code you haven't audited and simply presume is just fine? In plaid's defense, lack of performance tuning isn't necessarily a lack of security focus.

GordonS 7y ago
> In plaid's defense, lack of performance tuning isn't necessarily a lack of security focus Come on, this is not about "performance tuning", where you're trying to eek out every last drop of performance - it's about a completely indefensible, complex, wasteful solution to a simple problem. I'd say engineering insanity at this level is very worrisome for what they've done at the security side of things.
ubu7737 7y ago
ROFL "performance tuning" this is not tuning this is architecture. Some people think you can just write software, sell it to customers, and it's "tuning" to make it work properly. You should be fired from whatever job you have. My guess is that you have no job, you are fronting USD. In which case you have absolutely no place in this conversation and you should be ashamed of yourself for speaking up. A fool and his money are easily parted.
- dang 7y ago
  We've banned this account for breaking the site guidelines and ignoring our request to stop. If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- deleted 7y ago
  [deleted]

tracker1 7y ago

For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. The simplest solution is to scale to one worker per node initially if you're doing anything compute intensive... once you've done that, and/or you need better performance for any number of reasons including cost, then you can do more. Now, I'm not sure I would have gotten to 4k nodes before I started to re-evaluate parallelism or better scaling options, but the initial implementation is absolutely fine.

GordonS 7y ago
> For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. I get it, but come on - this was not a "performance optimisation" issue, but one of bad architecture; an architecture that certainly doesn't inspire confidence in the priorities you mention: accuracy, simplicity, safety.
- tracker1 7y ago
  Having a worker that does one request, processes that one request and returns a result isn't accurate, simple or safe? Scaling that simplistic interface in a 1:1 manner across many systems via ECS is pretty straight forward. Now, in addition to probably optimizing what they've done, converting to, for example another container system, like K8s where they can scale vertically a bit better may have been another approach. The biggest issue that I see is gRPC doesn't work that great with Node. You can't use it with cluster which means you have to self-manage threads/processes and it adds complexity there. Yeah, there's definitely issues that come into scaling in terms of performance optimization.... but where they started from isn't unreasonable imho.
ubu7737 7y ago
Where I work compliance is job #1. That doesn't prevent us from thinking about performance. GTFO with this nonsense.

mirekrusin 7y ago

Frankly the reality is so surreal there, it's actually surprising it doesn't mention uploading csv files over ftp, generating excel files, having cameras pointing on monitors of legacy systems that read data (no, this is not a joke) or spawning a promise for Martha to cross check something and click ok somewhere behind two bastion hosts, three firewalls and one and a half soap integrations. I'm not defending "4k containers because even loop can be blocked" which is silly, just reminding of the context - in other words you can do shittiest automated thing there and you're a hero. Next year-or-two hero is going to be somebody shrinking it by another X orders of magnitude.

bjacokes 7y ago

Hi, Plaid engineer here (not the author, but I helped with the post). I don't think we've tried to assert that the old system is perfect. We went into some detail in the post about why it took us this far. Certainly, the single request per container approach wouldn't scale if our unit economics were different. We didn't get into this too much in the post, but the Node service sits behind a couple of layers of Go services, so the we had more control over scaling API traffic than it might appear. Likewise, I hope we didn't give the impression that the new system is perfect. We've explored other languages for integrations in the past (even Haskell, at one point), and are continuing to do so. A migration away from our years-old Node integrations codebase would be a massive undertaking at this point. Absent that, it doesn't seem consistent to say "you're incompetent for handling 1 request per container" and also "you're incompetent for writing this post" – if you believe the former then it makes sense to be an advocate for this project, at least until a language migration can be done. I think the set of hoops we had to jump through in order to add concurrent requests without adding latency is a good demonstration of why we didn't do this sooner. It wasn't a massive undertaking by any means, but it wasn't trivial. At any rate, we're not really looking for a gold star here – just putting this out there and hoping this will be useful for others who are, as other commenters have put it, building their own "Frankensteins" :)

danudey 7y ago
I mean, my read of this is: 1. We used a system which uses event loops to achieve great concurrency, but we turned that off because we don't trust it. 2. Instead, we spent $300k/yr rolling out one-process-per-API as though we were using Apache 1.3. 3. We used an arbitrary JSON library without knowing anything about its performance characteristics, which it turns out were inordinately bad It's not that this wasn't a great exercise in engineering and problem-solving, or that it's not a great demonstration of how to solve scaling problems at scale, those are definitely true. It's more that "we spent $300k/yr more than we needed to so our engineers didn't need to learn how to use our technology stack properly." I'm not meaning to be harsh, I've kludged enough garbage into production in my lifetime, but more that the fact that you got into that situation in the first place gives a poor impression of either your development team or your development processes.

z3t4 7y ago

Linus Torvalds quote: "You need to grow thick skin". I also got a knee-jerk reaction when reading the first part, but the article explained it well, given that they probably don't want to give out too much information. So how would you have engineered it? I would just send the data uncompressed granted that the receiving server is probably in the same data-center with switches capable of handling Tbit's of data per second. I liked the article, but would have wanted more details. I love optimizations, it's such a drug, the rush when you make something x times faster. This article doesn't give me a bad impression. Contrary I'm thinking about sending an application.