3 ms·

I noticed something like this several times in the article: "Each machine has 2x 1gbit ethernet ports, in this case we’re only using one." It sounds like a sin

by mbell 14y ago

I noticed something like this several times in the article: "Each machine has 2x 1gbit ethernet ports, in this case we’re only using one." It sounds like a single switch failure could take down a large amount of their infrastructure. The seeming lack of geographic (or even datacenter) redundancy also seems a little dangerous at this scale.

avleenvig 14y ago
There is detail about multi-site redundancy which we're saving for a future post :-) You're right that a single switch failure would be impactful, but the impact of one switch going down would actually be very minimal. For example our web servers are spread out between many switches. The switches and network equipment have greater redundancy between them. - Avleen Vig, Staff Operations Engineer, Etsy
incision 14y ago
Yeah, that impresses me as a bit odd too. I've seen situations where folks have done this because they believed they would need VSS/VPC or similar on the switch side, but ALB on the server should avoid that. Perhaps they have a n array of access switches, each serving some subset to lessen the impact?
- brooksbp 14y ago
  VSS/VPC == MLAG... but what is ALB? I am guessing server uses one link until failure then uses the other? Even then, there are still network issues to overcome that may justify MLAG.
  - avleenvig 14y ago
    Honestly, we find it's much easier to spread the load between many switches, and have enough capacity that one switch failure is a non-event. Switches are expensive, it's true. After a certain point it does become less expensive and complex to just add more servers and switches. There's a strong advantage to keeping things as simple as possible. Bonding isn't really complicated, but how many not-complicated things can you add before things become complicated? :-)
maglos 14y ago
In fact while testing mrsync to synchronize search indexes among prod servers they overloaded their switches cpu and took down the site for a few minutes.
donavanm 14y ago
LAGs or redundant NICs is a mugs game. How deep do you go? Dual TORs up linked to dual aggs to dual transit routers, each with N+1 transit? all fed by A+B power infra, of course. Or keep it simple. One nic for prod traffic, one for private/control plane. Separate TORs are nice, or just don't oversubscribe them. Keep critical data hot across racks. Plan for failure of any given host, then for dc failure. You'll have site power or transit failure waaay more often than a bad tor.

by mbell 14y ago

avleenvig 14y ago

There is detail about multi-site redundancy which we're saving for a future post :-) You're right that a single switch failure would be impactful, but the impact of one switch going down would actually be very minimal. For example our web servers are spread out between many switches. The switches and network equipment have greater redundancy between them. - Avleen Vig, Staff Operations Engineer, Etsy

incision 14y ago

Yeah, that impresses me as a bit odd too. I've seen situations where folks have done this because they believed they would need VSS/VPC or similar on the switch side, but ALB on the server should avoid that. Perhaps they have a n array of access switches, each serving some subset to lessen the impact?

brooksbp 14y ago
VSS/VPC == MLAG... but what is ALB? I am guessing server uses one link until failure then uses the other? Even then, there are still network issues to overcome that may justify MLAG.
- avleenvig 14y ago
  Honestly, we find it's much easier to spread the load between many switches, and have enough capacity that one switch failure is a non-event. Switches are expensive, it's true. After a certain point it does become less expensive and complex to just add more servers and switches. There's a strong advantage to keeping things as simple as possible. Bonding isn't really complicated, but how many not-complicated things can you add before things become complicated? :-)

maglos 14y ago

In fact while testing mrsync to synchronize search indexes among prod servers they overloaded their switches cpu and took down the site for a few minutes.

donavanm 14y ago

LAGs or redundant NICs is a mugs game. How deep do you go? Dual TORs up linked to dual aggs to dual transit routers, each with N+1 transit? all fed by A+B power infra, of course. Or keep it simple. One nic for prod traffic, one for private/control plane. Separate TORs are nice, or just don't oversubscribe them. Keep critical data hot across racks. Plan for failure of any given host, then for dc failure. You'll have site power or transit failure waaay more often than a bad tor.