4 ms·

Here's the root of the problem. > Static provisioning also goes against the mindset of Kubernetes Then the mindset of Kubernetes is wrong. Or at least incompl

Here's the root of the problem. > Static provisioning also goes against the mindset of Kubernetes Then the mindset of Kubernetes is wrong. Or at least incomplete. Persistent data is an essential part of computing. In some ways it's the most important part. You could swap out every compute element in your system and be running exactly the way you were very quickly. Now try it with your storage elements. Oops, screwed. The data is the identity of your system. The problem is that storage is not trivially relocatable like compute is, and yet every single orchestration system I've seen seems to assume otherwise. The people who write them develop models to represent the easy case, then come back to the harder one as an afterthought. A car that's really a boat with wheels bolted on isn't going to have great handling, but that's pretty much where we are with storage in Kubernetes.

nemothekid 8y ago
AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc. The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potentially anywhere. Once you have that, then storage on k8s is "easy".
- candiodari 8y ago
  Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT: 1) It is per-cluster, not global (given Google's cluster sizes one imagine's that's not that much of a limitation, but then again, it seems like that is a clear limit) 2) you can only create or append to files (or delete them I guess) (and this is therefore non-posix, and does not support things like mysql or postgres) 3) this is very different from what GlusterFS, NFS, persistent volumes, etc provide. Therefore disks on google cloud are presumably very much not just files on this GFS/colossus thing. 4b) it was single master at least until 2004. Maybe until 2010. Apparently that can work. https://cloud.google.com/files/storage_architecture_and_challenges.pdf https://cloud.google.com/files/storage_architecture_and_chal... https://static.googleusercontent.com/media/research.google.com/en/us/archive/gfs-sosp2003.pdf https://static.googleusercontent.com/media/research.google.c...
  - philsnow 8y ago
    > GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. For the most part, google only has custom applications. Just about everything is written in-house, and takes advantage of things like being able to open a file from the local disk file just as easily as opening one in Colossus or their equivalent of Zookeeper.
  - steelframe 8y ago
    Would regional Persistent Disks be acceptable? https://cloud.google.com/compute/docs/disks/#repds https://cloud.google.com/compute/docs/disks/#repds Global might mean you're accessing blocks stored in Europe from a container running in Australia. If your workload is doing that, you might want to consider network costs and latency, and then reconsider whether you really want to architect things that way.
  - nemothekid 8y ago
    You are entirely correct, Colossus isn't POSIX. However, the solution Google has, is since they control all the software, essentially everything they write acts as if Colossus is their storage layer. This "solves" the persistent storage layer in a way Kubernetes cannot. Inside Google you might deploy your "Spanner" database container which knows how to interface with Colossus and doesn't require any special setup or colocation (below the cluster level). You cannot deploy MySQL on K8s and expect the same.
latch 8y ago
The tail wagging the dog that is the last 5 years of devops advancement. Constraints imposed on software development to achieve reproducibility. The most obvious example of this is the need to prematurely scale horizontally and deal with ephemeraliy (even compute).
- dilyevsky 8y ago
  If you’re ever planing on running at scale (and I assume most startups/projects strive for it), then there’s no point (other than maybe a rough poc) at which sharding is “premature” because later on it most likely will be prohibitively expensive.
  - latch 8y ago
    Picking sharding specifically is weird given the many problem spaces where it won't work. Also, whatever unnecessary scaling solution you come up with today for the problems you have today is not likely to be applicable to the problems you have years down the road; features will have changed, apps will be rewritten. I think people underestimate how much a handful of properly specd servers can achieve. My best experience with "at-scale" was 25+ million users, heavy API usage (and the API hits weren't trivial / hard to cache), and we could handle all the load on 2 beefy servers (but had more for HA and lower latency (they were distributed)). Quick search and I found a dual EPYC 7281 with 256GB DDR4, 2TB nvme, 2TB ssd, and 12TB HDD for $500/m in LA. Now it obviously depends on what you're doing, but for most startups to max this out, they're either extremely successful or extremely bad software engineers.
    - ori_b 8y ago
      > Quick search and I found a dual EPYC 7281 with 256GB DDR4, 2TB nvme, 2TB ssd, and 12TB HDD for $500/m in LA. And if you're willing to actually get a physical server, Dell happily sells single nodes with 6 terabytes of RAM for a few tens of thousands. Other vendors go much higher.
    - paulddraper 8y ago
      > I think people underestimate how much a handful of properly specd servers can achieve. Absolutely. > Quick search Even AWS (i.e. the very pricey one) has an x1e.2xlarge with 4.5 physical (9 hyperthreaded) Xenon E7 8880 v3 processors, 244GB DDR4, 240GB SSD, 25 Gbps network for $1200/month paid monthly or $700/month paid annually. And that scales linearly 16x. You've really got to be something amazing (maybe a ton of video?) to scale past what a single commodity box can handle.

nemothekid 8y ago

AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc. The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potentially anywhere. Once you have that, then storage on k8s is "easy".

candiodari 8y ago
Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT: 1) It is per-cluster, not global (given Google's cluster sizes one imagine's that's not that much of a limitation, but then again, it seems like that is a clear limit) 2) you can only create or append to files (or delete them I guess) (and this is therefore non-posix, and does not support things like mysql or postgres) 3) this is very different from what GlusterFS, NFS, persistent volumes, etc provide. Therefore disks on google cloud are presumably very much not just files on this GFS/colossus thing. 4b) it was single master at least until 2004. Maybe until 2010. Apparently that can work. https://cloud.google.com/files/storage_architecture_and_challenges.pdf https://cloud.google.com/files/storage_architecture_and_chal... https://static.googleusercontent.com/media/research.google.com/en/us/archive/gfs-sosp2003.pdf https://static.googleusercontent.com/media/research.google.c...
- philsnow 8y ago
  > GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. For the most part, google only has custom applications. Just about everything is written in-house, and takes advantage of things like being able to open a file from the local disk file just as easily as opening one in Colossus or their equivalent of Zookeeper.
- steelframe 8y ago
  Would regional Persistent Disks be acceptable? https://cloud.google.com/compute/docs/disks/#repds https://cloud.google.com/compute/docs/disks/#repds Global might mean you're accessing blocks stored in Europe from a container running in Australia. If your workload is doing that, you might want to consider network costs and latency, and then reconsider whether you really want to architect things that way.
- nemothekid 8y ago
  You are entirely correct, Colossus isn't POSIX. However, the solution Google has, is since they control all the software, essentially everything they write acts as if Colossus is their storage layer. This "solves" the persistent storage layer in a way Kubernetes cannot. Inside Google you might deploy your "Spanner" database container which knows how to interface with Colossus and doesn't require any special setup or colocation (below the cluster level). You cannot deploy MySQL on K8s and expect the same.

latch 8y ago

The tail wagging the dog that is the last 5 years of devops advancement. Constraints imposed on software development to achieve reproducibility. The most obvious example of this is the need to prematurely scale horizontally and deal with ephemeraliy (even compute).

dilyevsky 8y ago
If you’re ever planing on running at scale (and I assume most startups/projects strive for it), then there’s no point (other than maybe a rough poc) at which sharding is “premature” because later on it most likely will be prohibitively expensive.
- latch 8y ago
  Picking sharding specifically is weird given the many problem spaces where it won't work. Also, whatever unnecessary scaling solution you come up with today for the problems you have today is not likely to be applicable to the problems you have years down the road; features will have changed, apps will be rewritten. I think people underestimate how much a handful of properly specd servers can achieve. My best experience with "at-scale" was 25+ million users, heavy API usage (and the API hits weren't trivial / hard to cache), and we could handle all the load on 2 beefy servers (but had more for HA and lower latency (they were distributed)). Quick search and I found a dual EPYC 7281 with 256GB DDR4, 2TB nvme, 2TB ssd, and 12TB HDD for $500/m in LA. Now it obviously depends on what you're doing, but for most startups to max this out, they're either extremely successful or extremely bad software engineers.
  - ori_b 8y ago
    > Quick search and I found a dual EPYC 7281 with 256GB DDR4, 2TB nvme, 2TB ssd, and 12TB HDD for $500/m in LA. And if you're willing to actually get a physical server, Dell happily sells single nodes with 6 terabytes of RAM for a few tens of thousands. Other vendors go much higher.
  - paulddraper 8y ago
    > I think people underestimate how much a handful of properly specd servers can achieve. Absolutely. > Quick search Even AWS (i.e. the very pricey one) has an x1e.2xlarge with 4.5 physical (9 hyperthreaded) Xenon E7 8880 v3 processors, 244GB DDR4, 240GB SSD, 25 Gbps network for $1200/month paid monthly or $700/month paid annually. And that scales linearly 16x. You've really got to be something amazing (maybe a ton of video?) to scale past what a single commodity box can handle.