I'm always interested in the outcomes of these types of experiments. One thing that I find interesting is no mention of what network card you are using? or swi

I'm always interested in the outcomes of these types of experiments. One thing that I find interesting is no mention of what network card you are using? or switch setup, or ip config. I seem to remember in the dim and distant past that letting the storage layer handle the paths instead of LACP (which on some switches are active-passive) was better/faster/easier. (that could be bollocks though.) I used to work in VFX where performance and cost must be balanced. For the storage lumps we had a 1u dell with the fastest 10 core v3 processor in it, 384 gigs of ram (ram is a cheap and fast cache) tied to a dell 3060 (60 4tb disks in a 4u case, with up to 4 SAS connections out(well many other options it does iscsi too)) These topped out at about 2gigabytes/second (or about 1.3gigabytes with 100 concurrent fios from NFS clients). but there are 30+ of them so the aggregate was enough to dowse half the render farm in iops. A couple of things that stuck out for me: o 24/7 support contracts are amazing. o auto phone home disk replacements are also life savers o multipath iscsi can be painful unless you have decent switching (with ospf) and OS support o in short distances, SAS > FC >> iscsi for block storage. pretty much plug and play (multipath is easy too). o Replication needs a different path separate from the client o melenox or intel nics are the only 10gig cards worth paying for. (the only thing going for broadcom is that they are supported out of the box by vmware) o File systems always fail o for random IO; RAM > SSD by a massive factor. especially if your active dataset is less than ram cache. o drdb will be as slow as your slowest link in the network. o ethernet packet size really matters. (and depends on your workload) o synthetic benchmarks are the chocolate teapots of the storage world. o paying for decent support saved the (me and by extension the company(s)) at least twice. Yes, the dell/HP/whitebox route is more expensive in capital outlay. However you can if you so wish go the white box route. The dell raid enclosure is a rebadged netapp/lsi/enginio box. dothill do a similar one. The important thing to note though is the opex & RnD is far, far cheaper with prebuilt systems. With a click of a button you get replication. If you are using VMware (I'm sure KVM has a similar system) it can do it as well. The performance is predicatble, and the configuration and tuning has been done more or less for you (excluding the raid layout, FS tuning and network setup. but then you're a sysadmin who knows how to do all that right?) There is nothing worse than having to debug a storage system so unusual or highly configured that googling won't help. You feel terribly alone. Even more if you don't have decent backups. If you don't have those expertise, it'll cost you either time, or budget for a storage guy. (so just do some testing and buy the fastest prebuilt system for your budget.) For a previous place, I tested 7 different systems to get the best fit. (that included home rolled) A lot of the time was modelling the sotrage workload, and devising tests that would proxy that kind of load on the test systems. END_OF_RAMBLE