4 ms·

If I were to guess, they've probably invested heavily in building a bunch of "high availability distributed systems" when they should have just stuck to MySQL o

If I were to guess, they've probably invested heavily in building a bunch of "high availability distributed systems" when they should have just stuck to MySQL or Postgres.

sergiotapia 7y ago
Clever code is bad code. Always aim for simple, obvious code. Do you need kubernetes? Nope. 90% chance you don't. You're right, they probably are having issues with some lasagna architecture right now.
- tombert 7y ago
  I don't disagree with the thesis of your comment but I feel like people often use this reasoning to simply avoid doing any kind of architecting. Yes, something with Postgres and PHP is probably ok for a lot of usecases, but a) Robin Hood is a pretty big target, and might exceed the simple limits of that, and b) there are plenty of cases where Kubernetes (or any other big orchestration framework like Mesos or Nomad or Docker Swarm) will simplify the codebase. I'll agree that engineers are sometimes a bit too eager to just jump on the new HN tech a bit too early, but it's not like that tech isn't useful, and it's not like it doesn't serve some kind of purpose.
  - marcus_holmes 7y ago
    I agree on the need for architecting, except that there's a tendency to do that waaaay prematurely. You don't know where the best place to fragment your monolith is until you've got to the point where there monolith is struggling to keep up, because then you know where the slow bits are and how to scale them. And there's no need to split the whole thing. You can just split off the slow bits. Starting with Kubernetes from the get-go seems insane to me.
    - tombert 7y ago
      I don't run a big successful website, but I do run a Kubernetes cluster at home in my basement. Why? Because the computers I run everything on are six Nvidia Jetson Nanos; I'm a fan of running these because I like having access to a good GPU, and I can leave them running 24/7 without having to feel too guilty about power usage. Just one Nvidia Jetson Nano wouldn't be enough to handle all my server needs (movie streaming, video transcoding, random odds and ends of projects I have), and using a framework like Docker Swarm or Kubernetes makes this relatively easy. Is having the ability to linearly scale my home server overkill? Maybe, but at the same time it certainly felt necessary for the job, and carries the nice advantage that I don't have to rearchitect everything later while trying to shoehorn in the "old" way of doing stuff for compatibility sake. > You don't know where the best place to fragment your monolith is until you've got to the point where there monolith is struggling to keep up, because then you know where the slow bits are and how to scale them. Sure, it's not a silver bullet, and I certainly wouldn't claim it as such. That said, there are reasonable places to expect bottlenecks that can benefit from an architecture; things hitting a spinning disk or making an external HTTP call tend to be slow, so it is often better to model them asynchronously and buffer through a message queue; even if you don't have concrete numbers on your side it's not an unreasonable assumption to make.
- ftlio 7y ago
  I don't want to know how complicated someone has made their life if K8s can't simplify it.
- takeda 7y ago
  They recently had a blog about moving to kubernetes https://robinhood.engineering/building-an-application-deployment-platform-on-kubernetes-f6af367d1e05 https://robinhood.engineering/building-an-application-deploy...
- siscia 7y ago
  Yeahhh I am all down for simplify the stack as much as possible, but up to your necessity. Eventually you will reach a point where you will need clever and fast code.
PacifyFish 7y ago
At the volume and frequency they need to process, it probably makes sense to do something more complicated than MySQL...
- throwsprtsdy 7y ago
  Or less complicated, like piles of CSV files.
  - anon176 7y ago
    Or piles of JSON.. and scala/spark
737maxtw 7y ago
Wat? Dont get me wrong I'm not a fan of over engineering things. But those two statements are not mutually exclusive. You can build scalable distributed systems with postgres as a backend, heck if I was building one that would be a core part of my stack...
- DelightOne 7y ago
  How do you scale Postgres so far? Or do you move as much as you can to the outside?
  - clwg 7y ago
    You can use Citus(recently bought by Microsoft) to scale postgres https://www.citusdata.com/ https://www.citusdata.com/
rjkennedy98 7y ago
That is pure speculation. And people that use MySQL have outages also especially at this level of load. At my company we are just using PHP and MSSQL and we have our DB go out even on our internal apps on nearly a daily basis due to load problems. I can't imagine our DBs holding up to millions of users or need to process millions of ticker symbols or trades.
- riteshpatel 7y ago
  Sounds like missing indexes to me if outages are that frequent.
- 3fe9a03ccd14ca5 7y ago
  > we have our DB go out even on our internal apps on nearly a daily basis. What?? How could your team live with that?
  - rjkennedy98 7y ago
    Its extremely difficult to cut over millions of lines of code and 15+ years of development. We are working on it, but the db was never properly sharded and that is very difficult to do at a late stage. When you work on big enough projects the common sense approaches break down. FB messager doesn't use MySQL, Amazon shopping cart doesn't use Postgres. There is a reasons why. This isn't a wordpress app, it needs to process 10s of thousands of updates a second and still be a responsive app with 99.9% uptime and low SLAs on requests.
astannard 7y ago
you might be right: https://robinhood.engineering/building-an-application-deployment-platform-on-kubernetes-f6af367d1e05 https://robinhood.engineering/building-an-application-deploy...
- DelightOne 7y ago
  Neat. I think everyone did similar templating for similar applications. For example Golang apps which have logging/metrics already built-in instead of needing sidecars. > One notable project is to provide an intuitive, user-friendly frontend for executing GitOps workflows on Applications and Component manifests. Is this supposed to automatically update manifests, deploy them to staging namespace/automatic tests, enable automatic canary deployments? > We also hope to go even further and create a one-touch infrastructure-provisioning interface to abstract away manifests altogether, and place application-centric abstractions even more front-and-center for application developers. How does that look like? Archetype-Specific web interfaces/manifests? I'm kinda sad that this isn't available to the general public since I'd love to avoid constant repetition/writing my _limited_ own version.
ecnahc515 7y ago
It's funny you mention this, because I've heard from people who've interviewed with Robinhood that they're still on a single monolothic RDS instance for a good portion of their main application. If I had to guess, a single RDS instance would be a more likely cause, something like a table growing too large (an audit log table perhaps).

If I were to guess, they've probably invested heavily in building a bunch of "high availability distributed systems" when they should have just stuck to MySQL or Postgres.

sergiotapia 7y ago

Clever code is bad code. Always aim for simple, obvious code. Do you need kubernetes? Nope. 90% chance you don't. You're right, they probably are having issues with some lasagna architecture right now.

tombert 7y ago
I don't disagree with the thesis of your comment but I feel like people often use this reasoning to simply avoid doing any kind of architecting. Yes, something with Postgres and PHP is probably ok for a lot of usecases, but a) Robin Hood is a pretty big target, and might exceed the simple limits of that, and b) there are plenty of cases where Kubernetes (or any other big orchestration framework like Mesos or Nomad or Docker Swarm) will simplify the codebase. I'll agree that engineers are sometimes a bit too eager to just jump on the new HN tech a bit too early, but it's not like that tech isn't useful, and it's not like it doesn't serve some kind of purpose.
- marcus_holmes 7y ago
  I agree on the need for architecting, except that there's a tendency to do that waaaay prematurely. You don't know where the best place to fragment your monolith is until you've got to the point where there monolith is struggling to keep up, because then you know where the slow bits are and how to scale them. And there's no need to split the whole thing. You can just split off the slow bits. Starting with Kubernetes from the get-go seems insane to me.
  - tombert 7y ago
    I don't run a big successful website, but I do run a Kubernetes cluster at home in my basement. Why? Because the computers I run everything on are six Nvidia Jetson Nanos; I'm a fan of running these because I like having access to a good GPU, and I can leave them running 24/7 without having to feel too guilty about power usage. Just one Nvidia Jetson Nano wouldn't be enough to handle all my server needs (movie streaming, video transcoding, random odds and ends of projects I have), and using a framework like Docker Swarm or Kubernetes makes this relatively easy. Is having the ability to linearly scale my home server overkill? Maybe, but at the same time it certainly felt necessary for the job, and carries the nice advantage that I don't have to rearchitect everything later while trying to shoehorn in the "old" way of doing stuff for compatibility sake. > You don't know where the best place to fragment your monolith is until you've got to the point where there monolith is struggling to keep up, because then you know where the slow bits are and how to scale them. Sure, it's not a silver bullet, and I certainly wouldn't claim it as such. That said, there are reasonable places to expect bottlenecks that can benefit from an architecture; things hitting a spinning disk or making an external HTTP call tend to be slow, so it is often better to model them asynchronously and buffer through a message queue; even if you don't have concrete numbers on your side it's not an unreasonable assumption to make.
ftlio 7y ago
I don't want to know how complicated someone has made their life if K8s can't simplify it.
takeda 7y ago
They recently had a blog about moving to kubernetes https://robinhood.engineering/building-an-application-deployment-platform-on-kubernetes-f6af367d1e05 https://robinhood.engineering/building-an-application-deploy...
siscia 7y ago
Yeahhh I am all down for simplify the stack as much as possible, but up to your necessity. Eventually you will reach a point where you will need clever and fast code.

PacifyFish 7y ago

At the volume and frequency they need to process, it probably makes sense to do something more complicated than MySQL...

throwsprtsdy 7y ago
Or less complicated, like piles of CSV files.
- anon176 7y ago
  Or piles of JSON.. and scala/spark

737maxtw 7y ago

Wat? Dont get me wrong I'm not a fan of over engineering things. But those two statements are not mutually exclusive. You can build scalable distributed systems with postgres as a backend, heck if I was building one that would be a core part of my stack...

DelightOne 7y ago
How do you scale Postgres so far? Or do you move as much as you can to the outside?
- clwg 7y ago
  You can use Citus(recently bought by Microsoft) to scale postgres https://www.citusdata.com/ https://www.citusdata.com/

rjkennedy98 7y ago

That is pure speculation. And people that use MySQL have outages also especially at this level of load. At my company we are just using PHP and MSSQL and we have our DB go out even on our internal apps on nearly a daily basis due to load problems. I can't imagine our DBs holding up to millions of users or need to process millions of ticker symbols or trades.

riteshpatel 7y ago
Sounds like missing indexes to me if outages are that frequent.
3fe9a03ccd14ca5 7y ago
> we have our DB go out even on our internal apps on nearly a daily basis. What?? How could your team live with that?
- rjkennedy98 7y ago
  Its extremely difficult to cut over millions of lines of code and 15+ years of development. We are working on it, but the db was never properly sharded and that is very difficult to do at a late stage. When you work on big enough projects the common sense approaches break down. FB messager doesn't use MySQL, Amazon shopping cart doesn't use Postgres. There is a reasons why. This isn't a wordpress app, it needs to process 10s of thousands of updates a second and still be a responsive app with 99.9% uptime and low SLAs on requests.

astannard 7y ago

you might be right: https://robinhood.engineering/building-an-application-deployment-platform-on-kubernetes-f6af367d1e05 https://robinhood.engineering/building-an-application-deploy...

DelightOne 7y ago
Neat. I think everyone did similar templating for similar applications. For example Golang apps which have logging/metrics already built-in instead of needing sidecars. > One notable project is to provide an intuitive, user-friendly frontend for executing GitOps workflows on Applications and Component manifests. Is this supposed to automatically update manifests, deploy them to staging namespace/automatic tests, enable automatic canary deployments? > We also hope to go even further and create a one-touch infrastructure-provisioning interface to abstract away manifests altogether, and place application-centric abstractions even more front-and-center for application developers. How does that look like? Archetype-Specific web interfaces/manifests? I'm kinda sad that this isn't available to the general public since I'd love to avoid constant repetition/writing my _limited_ own version.

ecnahc515 7y ago

It's funny you mention this, because I've heard from people who've interviewed with Robinhood that they're still on a single monolothic RDS instance for a good portion of their main application. If I had to guess, a single RDS instance would be a more likely cause, something like a table growing too large (an audit log table perhaps).