6 ms·

What powers Etsy

166 points by johnzimmerman 14y ago

druiid 14y ago
It's very cool to see these kinds of posts. I always learn something from them and it's always neat to see what others have built to deal with a specific problem. I wonder in this case why, if any specific reason Etsy is not utilizing virtualization of one kind or another? Basically they're building dumb-boxes that seem a much better fit to have a good number of slightly beefier VM systems and just rebuild bad instances from images (this would be especially good since they're already utilizing Chef). I know that as an e-commerce firm this has helped us substantially to move toward this kind of a structure.
- jcampbell1 14y ago
  Several years ago, 37 signals wrote a post showing that moving away from virtualization improved response times. http://37signals.com/svn/posts/1819-basecamp-now-with-more-vroom http://37signals.com/svn/posts/1819-basecamp-now-with-more-v...
  - druiid 14y ago
    Well at least one thing to note there, is that post is from several years ago... both virtualization technologies have gotten better since then, as well as processors. I think there are certainly specific web and db loads that aren't going to work as well under VM scenarios, but on the whole our experience has been positive. There will, I imagine, always be a slight 'slowdown' due to VM but with the gigantic push toward 'the cloud' right now, it seems to me one of the last things on the minds of most (and obviously Xen is highly used in that field, which is also what we use). The bigger question I think is if you are using a highly CPU bound task which VM isn't always very good at. According to Etsy they are very CPU bound, so it makes some amount of sense.
    - jbooth 14y ago
      I think, generally speaking, once you start having >10 or >50 servers, VMs just stop making sense. You're getting a lot of hardware because you have a problem that requires it, and for most cloud systems with built-in redundancy, you'd only be running a single VM per machine anyways. Why not just run the machine? You should be using puppet or chef or something so it's not much/any harder to refresh a raw machine than it would be to refresh a VM image.
      - druiid 14y ago
        I think this it where we get back to the CPU vs memory bound tasks. If the task allows you to be running 1/8th of a newer Intel/AMD cpu at any given time (also accounting for bursting), why not look if you can squish some instances together? CPU bound I fully agree with you (although I wonder how hyper-scale type companies like Zynga did an analysis at this level and decided upon their stack... if there's a writeup out their about their decisions I'd love to see it actually... sorry, off-topic slightly).
        jbooth 14y ago
        Most 'cloud' software like Hadoop, Cassandra, clustered Mongo or whatever makes it a point to solve that problem in userspace software, giving you a view of 'one big app' across a bunch of machines. Since that's your level of interaction with the system, VMs tend to not really make a lot of sense, they just give you new opportunities to make mistakes while imposing a small performance tax. Now, if you have 10 legacy apps on old P4 xeons and want to lump them all onto a sandy bridge server and save a bunch of power and rackspace, then separate VMs on one machine makes sense. Basically, a cluster of 30 servers, all running exactly one VM doing the same thing is on-its-face silly. Why even have the VM then?
        travem 14y ago
        The advantage I see with virtualization is that you could use those servers to run your Hadoop workloads on those 30 servers over the weekend and then repurpose those machines for other workloads during the week.
- runako 14y ago
  The OP says they can spin up a new bare metal machine in 10 minutes. It's likely the performance hit from running in a virtualized environment isn't worth shaving time off the 10 minute build.
  - druiid 14y ago
    I'd say there are other reasons behind using virtualization than simply quick system spin-up, such as easier/better integration with (Dev|Sys)Op tools, higher hardware->instance densities than you'll ever see from physical... etc. I think the performance hit for many people is overstated. Unless you're at the long-tail of CPU bound instances I imagine it won't become very apparent. That said, if I'm missing something/incorrect then someone please correct me!
    - tedunangst 14y ago
      It sounds like the reason they have so many machines is because they need a lot of CPU. Compressing those machines onto fewer CPUs would not be a win.
    - runako 14y ago
      I don't work at Etsy or have any specific knowledge, but it sounds like they already have the tool integration they need with their current setup. Increasing densities through virtualization allows you to perform more discrete functions, but obviously reduces the aggregate capacity to do so. Etsy specifically has a goal of minimizing instance types, but virtualization would enable them go in a direction counter to their goals. In my experience, the biggest weakness of virtualization is subpar I/O, and I would imagine that this would be an issue for Etsy as well. Virtualization has a very real cost. If the performance hit (CPU & I/O) is (say) 20%, that's 20% of your capacity you pay to keep operational but don't get to use. The benefits don't outweigh that cost for every situation.
    - avleenvig 14y ago
      So, this is somewhat untrue for us. For example, let's look at some common web stack infrastructure: * Web servers * Database servers * Monitoring servers * Asynchronous task processors * Scheduled task servers (eg, cron jobs) In all of these cases, we want the service on that hardware to be able to use 100% CPU. For example, it doesn't make sense to get N web servers, and then run M virtual instances with apache on them, when each instance will max out its available resources. Just run N dedicated web servers :-) The same is true for pretty much everything else too, sometimes for different reasons. Eg, you don't want your database servers or monitoring system to be impacted by multi-tenancy issues. - Avleen Vig, Staff Operations Engineer, Etsy
- jgoulah 14y ago
  FWIW we do use virtualization for our dev infrastructure (everyone gets a vm) which uses KVM/QEMU and our CI cluster which is LXC based A bit about the dev infrastructure here: http://codeascraft.etsy.com/2012/03/13/making-it-virtually-easy-to-deploy-on-day-one/ http://codeascraft.etsy.com/2012/03/13/making-it-virtually-e...
  - druiid 14y ago
    Thanks for the reply. Also interesting to see dev side... we run something similar although a bit more free-form with the internal developers, but that's being pushed into a more controlled environment ;). I should also say it's a bit heartening to know of other e-com groups out there benefiting from an in-house stack rather than attempting to go 'TO THE CLOUD!'.
- Dobbs 14y ago
  Your point about rebuilding bad instances is just as easy to do with PXE and Chef/Puppet on bare metal.
  - avleenvig 14y ago
    sudo koan --replace-self sudo reboot -f
  - e12e 14y ago
    "Just as easy" might be an overstatement. Equally doable might be closer to the truth. One example where virtualization is easier in practice is where you're not in control of the network intfrastructure -- ie: you subcontract management of switches etc to someone else. For someone the size of Etsy that wouldn't matter much -- they have an obvious need to manage the entire stack. But for smaller institutions that might not target "scaling up to serve the entire public web" -- it is a factor. Another example is if you have heterogeneous hardware -- it is much easier to just install a hypervisor that runs on every box and then manage that via your vm-toolset, that set up many different images. It also makes it easier if you need to run different os' -- do all your os images have drivers for all your hardware? Granted, in theory OS' should abstract away hardware -- in practice, if you need to change out a few of your mysql instances running GNU/Linux to Microsoft SQL servers for a new project/change in workload -- being able to just spin down the old vms and up the new ones helps a lot in terms of agility. But yes, it's absolutely possible to provision physical hardware as well.
lordlarm 14y ago
The Supermicro servers are really really great - one of the best investments I've done so far in 2012. I bought mine (a Supermicro 2U Dual 3GHz Xeon 4GB 12 Bay Storage Server) for 400$ on eBay, with great service and terms. Recommended if you are ever in the need (or just want) a private server rack unit (1U, 2U or more).
- druiid 14y ago
  I'm a big fan of Supermicro systems. They are probably one of the best bang-for-the-buck out there, although one has to purchase them with the understanding of the potential 'risk' of not having a giant support department behind them like Dell/HP. That's not a knock against Supermicro, just a note for the PHB type view.
- caseyf 14y ago
  Here's a new Supermicro form factor that I really like that wasn't mentioned in the post: 1U with 2 CPUs, 10 x 2.5" drive bays, redundant power http://www.supermicro.com/products/system/1U/1027/SYS-1027R-N3RF.cfm http://www.supermicro.com/products/system/1U/1027/SYS-1027R-...
- mbell 14y ago
  I've had a few supermicro systems, some rack-mounted servers and a two 4U workstation pedestals. I have one bad thing to say...they are loud, really _loud_. Several years ago I had a 1U SM server in a rack full of HP servers I could hear the supermicro over all off them. The 4U pedestal I have at home as a file server is the "super-quiet" model with 8 3.5" bays in front and it is still insanely loud. I had to to rig up a fan controller just to keep it livable to have in the apartment as the motherboard couldn't slow the fans down enough.
- ChuckMcM 14y ago
  https://plus.google.com/u/1/108703267897818623506/posts/GK2AauEtcHZ https://plus.google.com/u/1/108703267897818623506/posts/GK2A... One of the rows in our data center, lots of Supermicro goodness. We use the 12 bay 'jupiter' chassis for 2x SSD and 10X 2TB spinning rust per server.
mbell 14y ago
I noticed something like this several times in the article: "Each machine has 2x 1gbit ethernet ports, in this case we’re only using one." It sounds like a single switch failure could take down a large amount of their infrastructure. The seeming lack of geographic (or even datacenter) redundancy also seems a little dangerous at this scale.
- avleenvig 14y ago
  There is detail about multi-site redundancy which we're saving for a future post :-) You're right that a single switch failure would be impactful, but the impact of one switch going down would actually be very minimal. For example our web servers are spread out between many switches. The switches and network equipment have greater redundancy between them. - Avleen Vig, Staff Operations Engineer, Etsy
- incision 14y ago
  Yeah, that impresses me as a bit odd too. I've seen situations where folks have done this because they believed they would need VSS/VPC or similar on the switch side, but ALB on the server should avoid that. Perhaps they have a n array of access switches, each serving some subset to lessen the impact?
  - brooksbp 14y ago
    VSS/VPC == MLAG... but what is ALB? I am guessing server uses one link until failure then uses the other? Even then, there are still network issues to overcome that may justify MLAG.
    - avleenvig 14y ago
      Honestly, we find it's much easier to spread the load between many switches, and have enough capacity that one switch failure is a non-event. Switches are expensive, it's true. After a certain point it does become less expensive and complex to just add more servers and switches. There's a strong advantage to keeping things as simple as possible. Bonding isn't really complicated, but how many not-complicated things can you add before things become complicated? :-)
chrismealy 14y ago
Anybody know what they use hadoop for?
- mcfunley 14y ago
  Analytics, search indexing, recommendations, various datasets that support search ranking, and some other things.
- avleenvig 14y ago
  Tons of stuff! Stay tuned for that in a future post :-) We posted about this in 2010: http://codeascraft.etsy.com/2010/02/24/analyzing-etsys-data-with-hadoop-and-cascading/ http://codeascraft.etsy.com/2010/02/24/analyzing-etsys-data-... Probably time for a refresh! - Avleen Vig, Staff Operations Engineer, Etsy
  - chrismealy 14y ago
    Cool! Thanks.
lazyjones 14y ago
We use those 4-in-1 Supermicro systems too, but with AMD CPUs. They seemed a much better deal to us and have a few more DIMM slots (= 128MB per node in our ~2-3 years old boxes). The only issue was the almost-half-length PCIe slot, it's actually 6" instead of 6.6" so you have to be careful when buying PCIe cards (might not be an issue with the Intel version board layouts). We've also invested in a 10gbe infrastructure rather early (we're smaller than etsy traffic-wise) and found it to be very useful for avoiding congestion issues when we need to transfer large files or pull data from a busy Postgres server. As for HP servers, we avoid them since our DL360 D6 reported obscure fan failures when booting with an Intel 10gbe card and we had to buy the (slower) HP NetXen-based cards instead. Supermicro boxes are not that picky and you don't get the feeling that a vendor might be using dirty tricky to lock you in.
EliRivers 14y ago
It's powered by love of crafting stuff :p
- avleenvig 14y ago
  If I could hand-craft circuit boards and CPUs, I probably would :-)
- jonhendry 14y ago
  And lots of Chinese mass production.
donavanm 14y ago
"Nagios checks to let us know when the filesystem becomes un-writeable" Any specific reason you don't just panic? I've had RO fs edge cases bite me a few times. "In the future we may even run fiber to these machines" Anything against 10g copper? Per port prices are getting crazy cheap. "DNS servers, LDAP servers, and our Hadoop Namenodes ... need RAID for extra data safety" Anything you've had issues with? Host level redundancy with the first two has been pretty great IME, can't comment on name odes. The only issue I've run in to is poor LDAP clients not fast failing an unhealthy server when another is available.
- avleenvig 14y ago
  RO filesystems can be bad, but usually they're soft failures for us: * Memcache can still work just fine * Db servers stop responding (and the app handles that fairly gracefully) * Web servers serve files from a RAM disk, so they keep working No reason against 10G copper specifically - we haven't had to address the problem in detail yet. When we do, we might choose copper. Depends what happens when it happens :-) We had a very nasty incident a few months ago, where the drive in one of the LDAP servers died. Well, it sort-of died. It started to time out a lot but didn't go fully offline. openldap kept running, but when you connected to it, the TCP connection would open and hang. This meant that all of our servers saw the server as "OK", but LDAP stopped working and caused all kinds of brilliant havoc :-)
  - donavanm 14y ago
    Interesting. I was thinking failure to persist updates, inconsistent health check results, those types of nasty corner cases. Sure. Premature optimization and all. Huh. SSD from a major chip vendor perhaps? I had one of those bite me just a few weeks ago. in my case it would have been fine if the drive offlined itself. but the intermittent io pauses killed performance, but not health checks. Actually, same type of problem RO FSs make me scared of.

166 points by johnzimmerman 14y ago

It's very cool to see these kinds of posts. I always learn something from them and it's always neat to see what others have built to deal with a specific problem. I wonder in this case why, if any specific reason Etsy is not utilizing virtualization of one kind or another? Basically they're building dumb-boxes that seem a much better fit to have a good number of slightly beefier VM systems and just rebuild bad instances from images (this would be especially good since they're already utilizing Chef). I know that as an e-commerce firm this has helped us substantially to move toward this kind of a structure.

jcampbell1 14y ago
Several years ago, 37 signals wrote a post showing that moving away from virtualization improved response times. http://37signals.com/svn/posts/1819-basecamp-now-with-more-vroom http://37signals.com/svn/posts/1819-basecamp-now-with-more-v...
- druiid 14y ago
  Well at least one thing to note there, is that post is from several years ago... both virtualization technologies have gotten better since then, as well as processors. I think there are certainly specific web and db loads that aren't going to work as well under VM scenarios, but on the whole our experience has been positive. There will, I imagine, always be a slight 'slowdown' due to VM but with the gigantic push toward 'the cloud' right now, it seems to me one of the last things on the minds of most (and obviously Xen is highly used in that field, which is also what we use). The bigger question I think is if you are using a highly CPU bound task which VM isn't always very good at. According to Etsy they are very CPU bound, so it makes some amount of sense.
  - jbooth 14y ago
    I think, generally speaking, once you start having >10 or >50 servers, VMs just stop making sense. You're getting a lot of hardware because you have a problem that requires it, and for most cloud systems with built-in redundancy, you'd only be running a single VM per machine anyways. Why not just run the machine? You should be using puppet or chef or something so it's not much/any harder to refresh a raw machine than it would be to refresh a VM image.
    - druiid 14y ago
      I think this it where we get back to the CPU vs memory bound tasks. If the task allows you to be running 1/8th of a newer Intel/AMD cpu at any given time (also accounting for bursting), why not look if you can squish some instances together? CPU bound I fully agree with you (although I wonder how hyper-scale type companies like Zynga did an analysis at this level and decided upon their stack... if there's a writeup out their about their decisions I'd love to see it actually... sorry, off-topic slightly).
      - jbooth 14y ago
        Most 'cloud' software like Hadoop, Cassandra, clustered Mongo or whatever makes it a point to solve that problem in userspace software, giving you a view of 'one big app' across a bunch of machines. Since that's your level of interaction with the system, VMs tend to not really make a lot of sense, they just give you new opportunities to make mistakes while imposing a small performance tax. Now, if you have 10 legacy apps on old P4 xeons and want to lump them all onto a sandy bridge server and save a bunch of power and rackspace, then separate VMs on one machine makes sense. Basically, a cluster of 30 servers, all running exactly one VM doing the same thing is on-its-face silly. Why even have the VM then?
        travem 14y ago
        The advantage I see with virtualization is that you could use those servers to run your Hadoop workloads on those 30 servers over the weekend and then repurpose those machines for other workloads during the week.
runako 14y ago
The OP says they can spin up a new bare metal machine in 10 minutes. It's likely the performance hit from running in a virtualized environment isn't worth shaving time off the 10 minute build.
- druiid 14y ago
  I'd say there are other reasons behind using virtualization than simply quick system spin-up, such as easier/better integration with (Dev|Sys)Op tools, higher hardware->instance densities than you'll ever see from physical... etc. I think the performance hit for many people is overstated. Unless you're at the long-tail of CPU bound instances I imagine it won't become very apparent. That said, if I'm missing something/incorrect then someone please correct me!
  - tedunangst 14y ago
    It sounds like the reason they have so many machines is because they need a lot of CPU. Compressing those machines onto fewer CPUs would not be a win.
  - runako 14y ago
    I don't work at Etsy or have any specific knowledge, but it sounds like they already have the tool integration they need with their current setup. Increasing densities through virtualization allows you to perform more discrete functions, but obviously reduces the aggregate capacity to do so. Etsy specifically has a goal of minimizing instance types, but virtualization would enable them go in a direction counter to their goals. In my experience, the biggest weakness of virtualization is subpar I/O, and I would imagine that this would be an issue for Etsy as well. Virtualization has a very real cost. If the performance hit (CPU & I/O) is (say) 20%, that's 20% of your capacity you pay to keep operational but don't get to use. The benefits don't outweigh that cost for every situation.
  - avleenvig 14y ago
    So, this is somewhat untrue for us. For example, let's look at some common web stack infrastructure: * Web servers * Database servers * Monitoring servers * Asynchronous task processors * Scheduled task servers (eg, cron jobs) In all of these cases, we want the service on that hardware to be able to use 100% CPU. For example, it doesn't make sense to get N web servers, and then run M virtual instances with apache on them, when each instance will max out its available resources. Just run N dedicated web servers :-) The same is true for pretty much everything else too, sometimes for different reasons. Eg, you don't want your database servers or monitoring system to be impacted by multi-tenancy issues. - Avleen Vig, Staff Operations Engineer, Etsy
jgoulah 14y ago
FWIW we do use virtualization for our dev infrastructure (everyone gets a vm) which uses KVM/QEMU and our CI cluster which is LXC based A bit about the dev infrastructure here: http://codeascraft.etsy.com/2012/03/13/making-it-virtually-easy-to-deploy-on-day-one/ http://codeascraft.etsy.com/2012/03/13/making-it-virtually-e...
- druiid 14y ago
  Thanks for the reply. Also interesting to see dev side... we run something similar although a bit more free-form with the internal developers, but that's being pushed into a more controlled environment ;). I should also say it's a bit heartening to know of other e-com groups out there benefiting from an in-house stack rather than attempting to go 'TO THE CLOUD!'.
Dobbs 14y ago
Your point about rebuilding bad instances is just as easy to do with PXE and Chef/Puppet on bare metal.
- avleenvig 14y ago
  sudo koan --replace-self sudo reboot -f
- e12e 14y ago
  "Just as easy" might be an overstatement. Equally doable might be closer to the truth. One example where virtualization is easier in practice is where you're not in control of the network intfrastructure -- ie: you subcontract management of switches etc to someone else. For someone the size of Etsy that wouldn't matter much -- they have an obvious need to manage the entire stack. But for smaller institutions that might not target "scaling up to serve the entire public web" -- it is a factor. Another example is if you have heterogeneous hardware -- it is much easier to just install a hypervisor that runs on every box and then manage that via your vm-toolset, that set up many different images. It also makes it easier if you need to run different os' -- do all your os images have drivers for all your hardware? Granted, in theory OS' should abstract away hardware -- in practice, if you need to change out a few of your mysql instances running GNU/Linux to Microsoft SQL servers for a new project/change in workload -- being able to just spin down the old vms and up the new ones helps a lot in terms of agility. But yes, it's absolutely possible to provision physical hardware as well.

lordlarm 14y ago

The Supermicro servers are really really great - one of the best investments I've done so far in 2012. I bought mine (a Supermicro 2U Dual 3GHz Xeon 4GB 12 Bay Storage Server) for 400$ on eBay, with great service and terms. Recommended if you are ever in the need (or just want) a private server rack unit (1U, 2U or more).

druiid 14y ago
I'm a big fan of Supermicro systems. They are probably one of the best bang-for-the-buck out there, although one has to purchase them with the understanding of the potential 'risk' of not having a giant support department behind them like Dell/HP. That's not a knock against Supermicro, just a note for the PHB type view.
caseyf 14y ago
Here's a new Supermicro form factor that I really like that wasn't mentioned in the post: 1U with 2 CPUs, 10 x 2.5" drive bays, redundant power http://www.supermicro.com/products/system/1U/1027/SYS-1027R-N3RF.cfm http://www.supermicro.com/products/system/1U/1027/SYS-1027R-...
mbell 14y ago
I've had a few supermicro systems, some rack-mounted servers and a two 4U workstation pedestals. I have one bad thing to say...they are loud, really _loud_. Several years ago I had a 1U SM server in a rack full of HP servers I could hear the supermicro over all off them. The 4U pedestal I have at home as a file server is the "super-quiet" model with 8 3.5" bays in front and it is still insanely loud. I had to to rig up a fan controller just to keep it livable to have in the apartment as the motherboard couldn't slow the fans down enough.
ChuckMcM 14y ago
https://plus.google.com/u/1/108703267897818623506/posts/GK2AauEtcHZ https://plus.google.com/u/1/108703267897818623506/posts/GK2A... One of the rows in our data center, lots of Supermicro goodness. We use the 12 bay 'jupiter' chassis for 2x SSD and 10X 2TB spinning rust per server.

mbell 14y ago

I noticed something like this several times in the article: "Each machine has 2x 1gbit ethernet ports, in this case we’re only using one." It sounds like a single switch failure could take down a large amount of their infrastructure. The seeming lack of geographic (or even datacenter) redundancy also seems a little dangerous at this scale.

avleenvig 14y ago
There is detail about multi-site redundancy which we're saving for a future post :-) You're right that a single switch failure would be impactful, but the impact of one switch going down would actually be very minimal. For example our web servers are spread out between many switches. The switches and network equipment have greater redundancy between them. - Avleen Vig, Staff Operations Engineer, Etsy
incision 14y ago
Yeah, that impresses me as a bit odd too. I've seen situations where folks have done this because they believed they would need VSS/VPC or similar on the switch side, but ALB on the server should avoid that. Perhaps they have a n array of access switches, each serving some subset to lessen the impact?
- brooksbp 14y ago
  VSS/VPC == MLAG... but what is ALB? I am guessing server uses one link until failure then uses the other? Even then, there are still network issues to overcome that may justify MLAG.
  - avleenvig 14y ago
    Honestly, we find it's much easier to spread the load between many switches, and have enough capacity that one switch failure is a non-event. Switches are expensive, it's true. After a certain point it does become less expensive and complex to just add more servers and switches. There's a strong advantage to keeping things as simple as possible. Bonding isn't really complicated, but how many not-complicated things can you add before things become complicated? :-)

chrismealy 14y ago

Anybody know what they use hadoop for?

mcfunley 14y ago
Analytics, search indexing, recommendations, various datasets that support search ranking, and some other things.
avleenvig 14y ago
Tons of stuff! Stay tuned for that in a future post :-) We posted about this in 2010: http://codeascraft.etsy.com/2010/02/24/analyzing-etsys-data-with-hadoop-and-cascading/ http://codeascraft.etsy.com/2010/02/24/analyzing-etsys-data-... Probably time for a refresh! - Avleen Vig, Staff Operations Engineer, Etsy
- chrismealy 14y ago
  Cool! Thanks.

lazyjones 14y ago

We use those 4-in-1 Supermicro systems too, but with AMD CPUs. They seemed a much better deal to us and have a few more DIMM slots (= 128MB per node in our ~2-3 years old boxes). The only issue was the almost-half-length PCIe slot, it's actually 6" instead of 6.6" so you have to be careful when buying PCIe cards (might not be an issue with the Intel version board layouts). We've also invested in a 10gbe infrastructure rather early (we're smaller than etsy traffic-wise) and found it to be very useful for avoiding congestion issues when we need to transfer large files or pull data from a busy Postgres server. As for HP servers, we avoid them since our DL360 D6 reported obscure fan failures when booting with an Intel 10gbe card and we had to buy the (slower) HP NetXen-based cards instead. Supermicro boxes are not that picky and you don't get the feeling that a vendor might be using dirty tricky to lock you in.

EliRivers 14y ago

It's powered by love of crafting stuff :p

avleenvig 14y ago
If I could hand-craft circuit boards and CPUs, I probably would :-)
jonhendry 14y ago
And lots of Chinese mass production.

donavanm 14y ago

"Nagios checks to let us know when the filesystem becomes un-writeable" Any specific reason you don't just panic? I've had RO fs edge cases bite me a few times. "In the future we may even run fiber to these machines" Anything against 10g copper? Per port prices are getting crazy cheap. "DNS servers, LDAP servers, and our Hadoop Namenodes ... need RAID for extra data safety" Anything you've had issues with? Host level redundancy with the first two has been pretty great IME, can't comment on name odes. The only issue I've run in to is poor LDAP clients not fast failing an unhealthy server when another is available.

avleenvig 14y ago
RO filesystems can be bad, but usually they're soft failures for us: * Memcache can still work just fine * Db servers stop responding (and the app handles that fairly gracefully) * Web servers serve files from a RAM disk, so they keep working No reason against 10G copper specifically - we haven't had to address the problem in detail yet. When we do, we might choose copper. Depends what happens when it happens :-) We had a very nasty incident a few months ago, where the drive in one of the LDAP servers died. Well, it sort-of died. It started to time out a lot but didn't go fully offline. openldap kept running, but when you connected to it, the TCP connection would open and hang. This meant that all of our servers saw the server as "OK", but LDAP stopped working and caused all kinds of brilliant havoc :-)
- donavanm 14y ago
  Interesting. I was thinking failure to persist updates, inconsistent health check results, those types of nasty corner cases. Sure. Premature optimization and all. Huh. SSD from a major chip vendor perhaps? I had one of those bite me just a few weeks ago. in my case it would have been fine if the drive offlined itself. but the intermittent io pauses killed performance, but not health checks. Actually, same type of problem RO FSs make me scared of.