4 ms·

You'd be surprised at how fast you can create something with an SQL database, a well used backend framework (eg. Rails) and a well used frontend framework. The

You'd be surprised at how fast you can create something with an SQL database, a well used backend framework (eg. Rails) and a well used frontend framework. The things big tech companies do are usually to avoid scaling problems. In particular scaling on the engineering side. The above can scale to billions on the user side if needed but 10k engineers constantly updating and messing with a traditional SQL schema doesn't work. You'd be crazy to try to replicate something like the big tech companies database storage systems when you're just starting out. You don't have 10k engineers. You may think "but i want to scale to a billion users". Well you still don't have those problems. The scaling problems big tech companies have are scaling to 10k engineers.

arvinsim 5y ago
One cannot predict the future but I agree that for most indie outfits, YAGNI would apply.
urthor 5y ago
It's also that, in the past, the capacity of our monoliths to scale was far lower. There's many apps that need say, a 100GB database in 2022. Those apps also needed 100GB database in 2007, when horizontal scaling was the hottest thing around. Nowadays however, Moore's law has steadily overtaken an order of magnitude of use cases from 2007. Maybe 1/10th of the 2007 use cases still need that kind of enormous, big tech scaling.
- adra 5y ago
  Scale vertically and you're still a single point of failure away from a big, potentially fatal outage. The coding and architecture can often be trivial though. Definitely good for internal systems and more green programmers. Scale horizontally and you're often constraining and or complicating your design/architecture, but you can handle outages far better, if not seamlessly. It's much harder to get right, so it's usually best for more senior teams that have specific need for it.
  - Nextgrid 5y ago
    I've seen more outages (or worse - silent data corruption, inconsistency, or deviations from the intended business rules) due to home-brewed, horizontally-scalable engineering playgrounds having a flaw than a good old Postgres on bare-metal hardware suddenly going dark.
  - latch 5y ago
    It's wrong to mix horizontal scaling with availability. They are two different things that solve two different problems and solving one doesn't necessarily mean you've solved the other. You can scale horizontally in a way that decreases your total availability, e.g. by sharding.
    - urthor 5y ago
      Essentially. Limited horizontal replication is not the same as PAXOS.
  - robryan 5y ago
    This really depends. The majority of services out there that can get away with a single box for a scale perspective can probably afford just to take the downtime.
  - lelanthran 5y ago
    > Scale vertically and you're still a single point of failure away from a big, potentially fatal outage. The coding and architecture can often be trivial though. Definitely good for internal systems and more green programmers. How fatal[1]? Not all outages are fatal. Scale vertically and you'll have some downtime if any node between (and in) your user and your system goes down. For 999 out of a thousand businesses, it won't be fatal. If your business is so fragile that the first 24 hour outage kills it, then it would have been killed anyway in the near future. [1] I know, this is liking asking "how pregnant?"...
  - gunnarmorling 5y ago
    > Scale vertically and you're still a single point of failure away from a big, potentially fatal outage. When folks talk about scaling up for stateful resources (database), this typically includes having stand-by replicas to which one can fail over within seconds. Scaling up the DB and scaling out stateless compute nodes is more than enough for many, many use cases. Having a DB machine with one 1TB RAM isn't anything special these days.
- rzzzt 5y ago
  Immediately reminded of https://yourdatafitsinram.net/ https://yourdatafitsinram.net/
  - DerArzt 5y ago
    I took a look at that and holy cow! TIL there are machines out there that have 64TB of RAM!
    - urthor 5y ago
      It's more that they've cheated a bit. 8 socket NUMA is a bit of a silly one, but it has been around in enterprise tech for a LONG time.
- Joeri 5y ago
  Stackoverflow can and has run on a single MS SQL server instance. Modern RDBMS’s on recent hardware can handle pretty ridiculous loads.
Yhippa 5y ago
The best part is when a company is building a line-of-business app which they know will get maybe a hundred users and end up throwing tools and processes at it that are complete overkill.
gocartStatue 5y ago
I would not be surprised, but it easily becomes a trap once you grow. With CRUDs / "Smart UI's" I'd always sign a contract in blood / note in ADR that we choose this to prototype quickly, but we'll stop and rebuild it before someone asks if he can "save his 20 fields grid filter he has to click through every day". Or (another Greg Young reference) - optimize for the ability to delete your code.

arvinsim 5y ago

One cannot predict the future but I agree that for most indie outfits, YAGNI would apply.

urthor 5y ago

It's also that, in the past, the capacity of our monoliths to scale was far lower. There's many apps that need say, a 100GB database in 2022. Those apps also needed 100GB database in 2007, when horizontal scaling was the hottest thing around. Nowadays however, Moore's law has steadily overtaken an order of magnitude of use cases from 2007. Maybe 1/10th of the 2007 use cases still need that kind of enormous, big tech scaling.

adra 5y ago
Scale vertically and you're still a single point of failure away from a big, potentially fatal outage. The coding and architecture can often be trivial though. Definitely good for internal systems and more green programmers. Scale horizontally and you're often constraining and or complicating your design/architecture, but you can handle outages far better, if not seamlessly. It's much harder to get right, so it's usually best for more senior teams that have specific need for it.
- Nextgrid 5y ago
  I've seen more outages (or worse - silent data corruption, inconsistency, or deviations from the intended business rules) due to home-brewed, horizontally-scalable engineering playgrounds having a flaw than a good old Postgres on bare-metal hardware suddenly going dark.
- latch 5y ago
  It's wrong to mix horizontal scaling with availability. They are two different things that solve two different problems and solving one doesn't necessarily mean you've solved the other. You can scale horizontally in a way that decreases your total availability, e.g. by sharding.
  - urthor 5y ago
    Essentially. Limited horizontal replication is not the same as PAXOS.
- robryan 5y ago
  This really depends. The majority of services out there that can get away with a single box for a scale perspective can probably afford just to take the downtime.
- lelanthran 5y ago
  > Scale vertically and you're still a single point of failure away from a big, potentially fatal outage. The coding and architecture can often be trivial though. Definitely good for internal systems and more green programmers. How fatal[1]? Not all outages are fatal. Scale vertically and you'll have some downtime if any node between (and in) your user and your system goes down. For 999 out of a thousand businesses, it won't be fatal. If your business is so fragile that the first 24 hour outage kills it, then it would have been killed anyway in the near future. [1] I know, this is liking asking "how pregnant?"...
- gunnarmorling 5y ago
  > Scale vertically and you're still a single point of failure away from a big, potentially fatal outage. When folks talk about scaling up for stateful resources (database), this typically includes having stand-by replicas to which one can fail over within seconds. Scaling up the DB and scaling out stateless compute nodes is more than enough for many, many use cases. Having a DB machine with one 1TB RAM isn't anything special these days.
rzzzt 5y ago
Immediately reminded of https://yourdatafitsinram.net/ https://yourdatafitsinram.net/
- DerArzt 5y ago
  I took a look at that and holy cow! TIL there are machines out there that have 64TB of RAM!
  - urthor 5y ago
    It's more that they've cheated a bit. 8 socket NUMA is a bit of a silly one, but it has been around in enterprise tech for a LONG time.
Joeri 5y ago
Stackoverflow can and has run on a single MS SQL server instance. Modern RDBMS’s on recent hardware can handle pretty ridiculous loads.

Yhippa 5y ago

The best part is when a company is building a line-of-business app which they know will get maybe a hundred users and end up throwing tools and processes at it that are complete overkill.

gocartStatue 5y ago

I would not be surprised, but it easily becomes a trap once you grow. With CRUDs / "Smart UI's" I'd always sign a contract in blood / note in ADR that we choose this to prototype quickly, but we'll stop and rebuild it before someone asks if he can "save his 20 fields grid filter he has to click through every day". Or (another Greg Young reference) - optimize for the ability to delete your code.