5 ms·

The tail wagging the dog that is the last 5 years of devops advancement. Constraints imposed on software development to achieve reproducibility. The most obvio

by latch 8y ago

The tail wagging the dog that is the last 5 years of devops advancement. Constraints imposed on software development to achieve reproducibility. The most obvious example of this is the need to prematurely scale horizontally and deal with ephemeraliy (even compute).

dilyevsky 8y ago
If you’re ever planing on running at scale (and I assume most startups/projects strive for it), then there’s no point (other than maybe a rough poc) at which sharding is “premature” because later on it most likely will be prohibitively expensive.
- latch 8y ago
  Picking sharding specifically is weird given the many problem spaces where it won't work. Also, whatever unnecessary scaling solution you come up with today for the problems you have today is not likely to be applicable to the problems you have years down the road; features will have changed, apps will be rewritten. I think people underestimate how much a handful of properly specd servers can achieve. My best experience with "at-scale" was 25+ million users, heavy API usage (and the API hits weren't trivial / hard to cache), and we could handle all the load on 2 beefy servers (but had more for HA and lower latency (they were distributed)). Quick search and I found a dual EPYC 7281 with 256GB DDR4, 2TB nvme, 2TB ssd, and 12TB HDD for $500/m in LA. Now it obviously depends on what you're doing, but for most startups to max this out, they're either extremely successful or extremely bad software engineers.
  - ori_b 8y ago
    > Quick search and I found a dual EPYC 7281 with 256GB DDR4, 2TB nvme, 2TB ssd, and 12TB HDD for $500/m in LA. And if you're willing to actually get a physical server, Dell happily sells single nodes with 6 terabytes of RAM for a few tens of thousands. Other vendors go much higher.
  - paulddraper 8y ago
    > I think people underestimate how much a handful of properly specd servers can achieve. Absolutely. > Quick search Even AWS (i.e. the very pricey one) has an x1e.2xlarge with 4.5 physical (9 hyperthreaded) Xenon E7 8880 v3 processors, 244GB DDR4, 240GB SSD, 25 Gbps network for $1200/month paid monthly or $700/month paid annually. And that scales linearly 16x. You've really got to be something amazing (maybe a ton of video?) to scale past what a single commodity box can handle.
- sbov 8y ago
  What do you mean by at scale? At what scale? In the past, I've supported a few million daily active users on a single MySQL database. Use cases obviously vary, your numbers may not line up. But I'm pretty sure there are lot of startups that would be fine starting with an architecture that can scale to 2 million users per day.
  - jjirsa 8y ago
    A few million DAU is not at scale. A few million concurrent active users might be, but really, “at scale” has no meaning other than “we got to the point where shit gets hard”. And that point is different for every app. I remember watching a big xbox/ps3 game launch crash and burn when 16 of the biggest MySQL servers we could buy couldn’t keep up. That was two jobs ago and embarrassing (and probably expensive). That was “at scale” for us. Last job we started at about 60 m4 ex2 instanced and were well into the thousands when I left. I suspect they’re approaching 10k instances now. And they’re pre-IPO startup, and I think that was at scale. Current job measures in the hundreds of thousands of database instances, and I only count one specific database engine. Probably counts as at scale.
    - ThePhysicist 8y ago
      Really curious what kind of problem requires 100.000s of DB servers, and what database system is actually capable of such massive horizontal scaling? Are you able to provide any specifics? I imagine that typically a single instance can serve at least 100-1000 concurrent users, meaning you have about 10 to 100.000.000 concurrent users on the system? Since you mentioned gaming I assume it's a massive multiplayer-game like Fortnite?
      - jjirsa 8y ago
        Gaming was two jobs ago. It’s apache Cassandra, and I work for a large tech company.

by latch 8y ago

dilyevsky 8y ago

If you’re ever planing on running at scale (and I assume most startups/projects strive for it), then there’s no point (other than maybe a rough poc) at which sharding is “premature” because later on it most likely will be prohibitively expensive.

latch 8y ago
Picking sharding specifically is weird given the many problem spaces where it won't work. Also, whatever unnecessary scaling solution you come up with today for the problems you have today is not likely to be applicable to the problems you have years down the road; features will have changed, apps will be rewritten. I think people underestimate how much a handful of properly specd servers can achieve. My best experience with "at-scale" was 25+ million users, heavy API usage (and the API hits weren't trivial / hard to cache), and we could handle all the load on 2 beefy servers (but had more for HA and lower latency (they were distributed)). Quick search and I found a dual EPYC 7281 with 256GB DDR4, 2TB nvme, 2TB ssd, and 12TB HDD for $500/m in LA. Now it obviously depends on what you're doing, but for most startups to max this out, they're either extremely successful or extremely bad software engineers.
- ori_b 8y ago
  > Quick search and I found a dual EPYC 7281 with 256GB DDR4, 2TB nvme, 2TB ssd, and 12TB HDD for $500/m in LA. And if you're willing to actually get a physical server, Dell happily sells single nodes with 6 terabytes of RAM for a few tens of thousands. Other vendors go much higher.
- paulddraper 8y ago
  > I think people underestimate how much a handful of properly specd servers can achieve. Absolutely. > Quick search Even AWS (i.e. the very pricey one) has an x1e.2xlarge with 4.5 physical (9 hyperthreaded) Xenon E7 8880 v3 processors, 244GB DDR4, 240GB SSD, 25 Gbps network for $1200/month paid monthly or $700/month paid annually. And that scales linearly 16x. You've really got to be something amazing (maybe a ton of video?) to scale past what a single commodity box can handle.
sbov 8y ago
What do you mean by at scale? At what scale? In the past, I've supported a few million daily active users on a single MySQL database. Use cases obviously vary, your numbers may not line up. But I'm pretty sure there are lot of startups that would be fine starting with an architecture that can scale to 2 million users per day.
- jjirsa 8y ago
  A few million DAU is not at scale. A few million concurrent active users might be, but really, “at scale” has no meaning other than “we got to the point where shit gets hard”. And that point is different for every app. I remember watching a big xbox/ps3 game launch crash and burn when 16 of the biggest MySQL servers we could buy couldn’t keep up. That was two jobs ago and embarrassing (and probably expensive). That was “at scale” for us. Last job we started at about 60 m4 ex2 instanced and were well into the thousands when I left. I suspect they’re approaching 10k instances now. And they’re pre-IPO startup, and I think that was at scale. Current job measures in the hundreds of thousands of database instances, and I only count one specific database engine. Probably counts as at scale.
  - ThePhysicist 8y ago
    Really curious what kind of problem requires 100.000s of DB servers, and what database system is actually capable of such massive horizontal scaling? Are you able to provide any specifics? I imagine that typically a single instance can serve at least 100-1000 concurrent users, meaning you have about 10 to 100.000.000 concurrent users on the system? Since you mentioned gaming I assume it's a massive multiplayer-game like Fortnite?
    - jjirsa 8y ago
      Gaming was two jobs ago. It’s apache Cassandra, and I work for a large tech company.