6 ms·

In places worked we usually had a vmware cluster, load balancer, NFS for shared data when necessary and DNS set up (e.g: through consul). This setup is very, v

by aprdm 8y ago

In places worked we usually had a vmware cluster, load balancer, NFS for shared data when necessary and DNS set up (e.g: through consul). This setup is very, very simple and scalable. There is very little to gain IMO on moving to Kubernetes. Consul, VSphere and load balancers have APIs and you can write tools to do everything that K8s does.

dilyevsky 8y ago
Scalable NFS, riiite.
- aprdm 8y ago
  If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the computer graphics you see in movies. At last job our NFS cluster did 300 000 IOPS with 82gb/s throughput
  - m0zg 8y ago
    82gb/s (assuming you mean gigabit) is _per-node_ throughput at Google (or FB, or I assume Amazon/Microsoft -- they all use 100GbE networks now). 300K IOPS is probably per-node, too, at this point. :-)
    - deleted 8y ago
      [deleted]
    - kortilla 8y ago
      Having a 100gbps nic in a node isn’t the same thing as doing storage at that speed in an HA cluster. Also, don’t confuse 100 gbe networks where spine links are 100 but the node links are only bonded 10s (much more common at $fang).
      - m0zg 8y ago
        Nope. It's all 100GbE throughout as far as I know. And people do work really hard to be able to saturate that bandwidth as it is by no means a trivial task to saturate it through the usual, naive means without the use of RDMA and Verbs. Years ago when I was there it was (IIRC) 40Gbps to each node straight up. It's a necessity really. All storage at Google has been remote and distributed for at least the past decade. That puts serious demands on network throughput if you want your CPUs to actually do work and not just sit there and wait for data. Here's some detail as of 2012: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43837.pdf https://storage.googleapis.com/pub-tools-public-publication-.... Note that host speed is 40Gbps. And here's FB talking about migrating from 40Gbps to 100Gbps in 2016: https://code.fb.com/data-center-engineering/introducing-backpack-our-second-generation-modular-open-switch/ https://code.fb.com/data-center-engineering/introducing-back...
  - dilyevsky 8y ago
    Sorry I don’t have to read it because i was borg sre for 6 years and i know how (the server part of) it works. You assume wrong. I know there are a lot of companies that try to put some lipstick on nfs pig and call it reliable/scalable/etc. so long their clients don’t actually try to run it at scale or don’t complain too publicly when they try and can’t, they are able to get away with it.
    - aprdm 8y ago
      Your concept of what is scale looks very different than mine, in my experience NFS does a very good job for in-datacenter workloads. CG rendering, oil/gas and others usually take this approach for HPC as far as I've seen. I consider this "scale". Close to 100k procs sharing the nfs is the biggest cluster I've worked at. Of course that over longer networks it isn't suitable as the roundtrips have too much latency, other than that, is your experience much different regarding nfs?
    - kortilla 8y ago
      What you consider ‘scale’ is a high watermark used by cloud providers that is irrelevant to 99.999% of the industry. Supporting all of a Fortune 500’s business operations is very reasonable to call ‘scale’ in the normal world. Your comment is a like a billionaire claiming that somebody that managed to hit 30 million isn’t rich.
- geggam 8y ago
  worked at a company with 4 Petabytes on NFS ... FWIW
merb 8y ago
How do you load balance? i mean load balance the "public ip" In some networks DNS failover is really not that great, so at least a virtual ip needs to be used.
- takeda 8y ago
  We use haproxy. I wrote code [1] that configures it based on Consul to do load balance. It has been running in production for 2 years without issue (tested with consul 1.0.6) For people wondering why not use consul template, this has benefit of understanding haproxy, and it minimizes numbers of restarts to make changes. Using haproxy over and has the benefit that if Winkle or consul goes down things continue to work, just not updates. [1] https://github.com/takeda/winkle https://github.com/takeda/winkle
  - merb 8y ago
    Haproxy only solves a single Part of the Problem. If you do DNS based failover you should really Check how clients behave, when one node goes down. Without a floating ip or a cloud lb, some stuff will be troublesome
    - takeda 8y ago
      The haproxy method doesn't rely on DNS at all so I'm a bit confused.
      - merb 8y ago
        well either it uses DNS for failover or you have ipvs (lvs, keepalived) enabled or worse if the machine with haproxy crashes your basically dead. Of course there is also bgp and anycast, but this is not "cheap"
        takeda 8y ago
        That's not how it works. It is very similar to sidecar approach which various service discovery solutions have. You have haproxy running locally on localhost and you communicate with it, the haproxy then routes the request to right nodes. No DNS, no LVS and no keepalived.

by aprdm 8y ago

dilyevsky 8y ago

Scalable NFS, riiite.

aprdm 8y ago
If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the computer graphics you see in movies. At last job our NFS cluster did 300 000 IOPS with 82gb/s throughput
- m0zg 8y ago
  82gb/s (assuming you mean gigabit) is _per-node_ throughput at Google (or FB, or I assume Amazon/Microsoft -- they all use 100GbE networks now). 300K IOPS is probably per-node, too, at this point. :-)
  - deleted 8y ago
    [deleted]
  - kortilla 8y ago
    Having a 100gbps nic in a node isn’t the same thing as doing storage at that speed in an HA cluster. Also, don’t confuse 100 gbe networks where spine links are 100 but the node links are only bonded 10s (much more common at $fang).
    - m0zg 8y ago
      Nope. It's all 100GbE throughout as far as I know. And people do work really hard to be able to saturate that bandwidth as it is by no means a trivial task to saturate it through the usual, naive means without the use of RDMA and Verbs. Years ago when I was there it was (IIRC) 40Gbps to each node straight up. It's a necessity really. All storage at Google has been remote and distributed for at least the past decade. That puts serious demands on network throughput if you want your CPUs to actually do work and not just sit there and wait for data. Here's some detail as of 2012: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43837.pdf https://storage.googleapis.com/pub-tools-public-publication-.... Note that host speed is 40Gbps. And here's FB talking about migrating from 40Gbps to 100Gbps in 2016: https://code.fb.com/data-center-engineering/introducing-backpack-our-second-generation-modular-open-switch/ https://code.fb.com/data-center-engineering/introducing-back...
- dilyevsky 8y ago
  Sorry I don’t have to read it because i was borg sre for 6 years and i know how (the server part of) it works. You assume wrong. I know there are a lot of companies that try to put some lipstick on nfs pig and call it reliable/scalable/etc. so long their clients don’t actually try to run it at scale or don’t complain too publicly when they try and can’t, they are able to get away with it.
  - aprdm 8y ago
    Your concept of what is scale looks very different than mine, in my experience NFS does a very good job for in-datacenter workloads. CG rendering, oil/gas and others usually take this approach for HPC as far as I've seen. I consider this "scale". Close to 100k procs sharing the nfs is the biggest cluster I've worked at. Of course that over longer networks it isn't suitable as the roundtrips have too much latency, other than that, is your experience much different regarding nfs?
  - kortilla 8y ago
    What you consider ‘scale’ is a high watermark used by cloud providers that is irrelevant to 99.999% of the industry. Supporting all of a Fortune 500’s business operations is very reasonable to call ‘scale’ in the normal world. Your comment is a like a billionaire claiming that somebody that managed to hit 30 million isn’t rich.
geggam 8y ago
worked at a company with 4 Petabytes on NFS ... FWIW

merb 8y ago

How do you load balance? i mean load balance the "public ip" In some networks DNS failover is really not that great, so at least a virtual ip needs to be used.

takeda 8y ago
We use haproxy. I wrote code [1] that configures it based on Consul to do load balance. It has been running in production for 2 years without issue (tested with consul 1.0.6) For people wondering why not use consul template, this has benefit of understanding haproxy, and it minimizes numbers of restarts to make changes. Using haproxy over and has the benefit that if Winkle or consul goes down things continue to work, just not updates. [1] https://github.com/takeda/winkle https://github.com/takeda/winkle
- merb 8y ago
  Haproxy only solves a single Part of the Problem. If you do DNS based failover you should really Check how clients behave, when one node goes down. Without a floating ip or a cloud lb, some stuff will be troublesome
  - takeda 8y ago
    The haproxy method doesn't rely on DNS at all so I'm a bit confused.
    - merb 8y ago
      well either it uses DNS for failover or you have ipvs (lvs, keepalived) enabled or worse if the machine with haproxy crashes your basically dead. Of course there is also bgp and anycast, but this is not "cheap"
      - takeda 8y ago
        That's not how it works. It is very similar to sidecar approach which various service discovery solutions have. You have haproxy running locally on localhost and you communicate with it, the haproxy then routes the request to right nodes. No DNS, no LVS and no keepalived.