11 ms·

New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SSD-Backed Storage

196 points by jeffbarr 14y ago

beedogs 14y ago
Amazon will find some way to make this slower than shit and less reliable than a campaign promise.
- jedberg 14y ago
  I've used them and they are quite performant. Definitely lives up to its promise.
oomkiller 14y ago
Hopefully we will start to see some other providers offer SSD-backed storage since Amazon does it now. It would be nice if they offered it on some smaller instances too though.
- thegyppo 14y ago
  Storm on Demand have done SSD storage for a while now, pretty solid IOPS numbers too. I'll run a benchmark comparison tonight.
  - thegyppo 14y ago
    Ran some benchmarks, really insane IOPS on this plan. Pretty average CPU performance (they're using fairly old E5620's). http://blog.serverbear.com/post/27553311076/hi1-4xlarge-benchmarks http://blog.serverbear.com/post/27553311076/hi1-4xlarge-benc...
    - signifiers 14y ago
      I just ran the numbers on the new EC2 instance, and I'm pretty skeptical about the benchmarks above. I'm not sure that, for example, a half second of dd /dev/zero really tells us much. When interpreting any benchmarks on EC2, it's important to understand that there is a 5-10% read/write performance hit on first use because AWS uses lazy block wipes between customer instance launches. See http://www.youtube.com/watch?v=IedaYaKsb-4#t=29m49s http://www.youtube.com/watch?v=IedaYaKsb-4#t=29m49s (should pre-cue, if not, skip to 29:49). This is referenced in the docs, but it's easy to miss: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/InstanceStorage.html?r=409#disk-performance http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/In...) So here you go, for hi1.4xlarge: * Summary for the impatient - After initialization (i.e., second-write), quasi-realistic I/O on the new SSD EC2 instances sustains writes @ 420 MB/sec and reads @ 6 GB/sec. The entire 8.6GB / filesystem copied over to SSD in 21 seconds. Not bad. * # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 8.0G 1.1G 6.9G 14% / tmpfs 30G 0 30G 0% /dev/shm /dev/xvdf 1023G 16G 957G 2% /media/ephemeral0 (Note: /dev/xvdf and /dev/xvdg are just soft links to /dev/sdf and /dev/sdg respectively) Crude stats on first-use: # hdparm -tT /dev/xvdf /dev/xvdf: Timing cached reads: 14788 MB in 1.99 seconds = 7446.69 MB/sec Timing buffered disk reads: 1066 MB in 3.00 seconds = 355.04 MB/sec Wipe the device: dd if=/dev/zero of=/dev/xvdf bs=1M& pid=$! while true; do kill -USR1 $pid; sleep 4; done; [...] dd: writing `/dev/xvdf': No space left on device 1048567+17 records in 1048566+17 records out 1099511627776 bytes (1.1 TB) copied, 1955.42 s, 562 MB/s Stats after zero-wipe (dd /dev/zero) to device: hdparm -tT /dev/xvdf /dev/xvdf: Timing cached reads: 13260 MB in 1.99 seconds = 6673.05 MB/sec Timing buffered disk reads: 1124 MB in 3.01 seconds = 374.02 MB/sec hdparm -tT /dev/xvdf /dev/xvdf: Timing cached reads: 11188 MB in 1.99 seconds = 5624.17 MB/sec Timing buffered disk reads: 1122 MB in 3.00 seconds = 373.99 MB/sec hdparm -tT /dev/xvdf /dev/xvdf: Timing cached reads: 12930 MB in 1.99 seconds = 6505.78 MB/sec Timing buffered disk reads: 1124 MB in 3.00 seconds = 374.15 MB/sec Confirming Effect Of Pre-wiped I/O: hdparm -tT /dev/xvdg Timing cached reads: 11796 MB in 1.99 seconds = 5931.68 MB/sec Timing buffered disk reads: 1038 MB in 3.00 seconds = 345.87 MB/sec hdparm -tT /dev/xvdg /dev/xvdg: Timing cached reads: 12658 MB in 1.99 seconds = 6367.41 MB/sec Timing buffered disk reads: 1050 MB in 3.00 seconds = 349.47 MB/sec hdparm -tT /dev/xvdg /dev/xvdg: Timing cached reads: 12856 MB in 1.99 seconds = 6468.39 MB/sec Timing buffered disk reads: 1066 MB in 3.00 seconds = 354.80 MB/sec Pre- Vs. Post-wipe performance: 373.6 MB/sec vs. 349.3 MB/sec (6-7% speed improvement) Somewhat more real-world numbers: dd if=/dev/sda1 of=/dev/xvdf bs=1M 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 19.7876 s, 434 MB/s dd if=/dev/sda1 of=/dev/xvdf bs=1M 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 20.0365 s, 429 MB/s dd if=/dev/sda1 of=/dev/xvdf bs=1M 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 21.4193 s, 401 MB/s *Edit: formatting
- j-kidd 14y ago
  Check out this long thread on WHT, started 2 months ago: http://www.webhostingtalk.com/showthread.php?t=1151090 http://www.webhostingtalk.com/showthread.php?t=1151090 From my limited testing, a 1 Node instance from GridVirt at $30/month can compile gcc-4.6.3 within 15 minutes (not exactly an IO-intensive example...)
ehsanu1 14y ago
This has been a long time coming, but AWS has consistently been improving their service (as long as you can ignore the particularly bad reliability as of late). It's telling that they have only enabled this for a huge (quadruple extra large) instance type. It's probably hard to make this work for someone who just wants a 10GB disk with great IO. The problem at the low end is that disks are larger and would thus have to be divided up to make proper use of them, leading to IO contention.. The high IO options will probably only ever be available for pretty large instances.
- oomkiller 14y ago
  It seems like they could spread a bunch of smaller instances over an array of SSDs, maybe just offer less space at a higher price? SSDs are naturally better at concurrent access, so while you might not get 150k+ IOPS, you could get at least 10k or so, with the expected low random access times. Imagine an SSD-backed EBS.
- cperciva 14y ago
  It's telling that they have only enabled this for a huge (quadruple extra large) instance type. My guess (not based on any knowledge of EC2 internals) is that they don't have any way to do fair I/O sharing between guests. If they did, they could split these boxes into 32 small instances with 1 ECU, 1.7 GB RAM, and a 60 GB disk with 2500 random reads / 250-4000 random writes per second.
  - XERQ 14y ago
    Xen offers easy ways of doing fair I/O sharing between guests. These servers they're using are most likely multi-tenant systems with 256-512GB of RAM and 6-12TB of SSD storage. Providers don't like keeping expensive systems around that aren't making money, especially when demand changes every hour, so I expect that they have at least 4 instances sharing the I/O of each host (especially when they mention broad ranges of expected I/O). The most likely reason for not slicing these systems up to smaller instances is they want to maintain consistent, high performance I/O.
    - 14y ago
bravura 14y ago
What are good use-cases for on-demand high I/O servers? At $3.10/hr, these instances work out to $2k/mo. There are probably many more cost-effective options if you want a 2TB SSD server. Since the benefit of using EC2 is that you can provision instances elastically, what are the sorts of scenarios in which one needs to provision high I/O servers elastically? [edit: A few minutes of Googling, and I can't find any dedicated servers with 2 TB of SSD.]
- saurik 14y ago
  That is only one of the benefits of EC2. If you are not using elasticity, then you have to factor in the reserved instance pricing, which drops the prices down by 71% (as in, to 29% of the list price; and I mean even if you include the up-front cost: that's overall savings). You, like most people who comment on the price of EC2, do not seem to be taking this into consideration. :(
  - joe_bloggs 14y ago
    What are some of the other benefits, other than elasticity or on-demand ness, and not having to spend a huge amount upfront?
    - saurik 14y ago
      One example, which stems from "on-demand ness" (which you added: I was only responding to "elasticity"), is that you can do "test runs" of migrations and deployments without even thinking about it: you can rent, for just an hour, a setup identical to your existing one, often based on a consistent and atomic snapshot of your production machine, so you can try something "likely correct but possibly horribly wrong"; then, if it works, rather than replicating the change on your "real" machine, you can just cut over to the new one and shut down the old. Way too many people seem to believe that the only benefit of "on-demand" is "elasticity", and then make bogus arguments here that "if you can plan your traffic you shouldn't be using EC2": EC2 is cheaper than people like to claim (and is in fact quite price competitive) and your ability to turn on/off machines on a whim changes the way you look at hardware so drastically that, in all honesty, it makes traditional ways of dealing with hardware seem draconian and only worth putting up with if you are dealing with some weird corner case or have horribly special requirements.
damian2000 14y ago
The pricing in the blog post is a bit unclear - the prices on http://aws.amazon.com/ec2/pricing/ http://aws.amazon.com/ec2/pricing/ are ... US East $3.10 for linux and $3.58 for windows. EU West $3.41 for linux and $3.58 for windows. (Reserved instance prices are cheaper)
jbarham 14y ago
FWIW it costs $27,156 a year on-demand or $12,720 as a reserved heavy utilization instance. For heavy analytics workloads I'd bet that Google BigQuery (https://developers.google.com/bigquery/ https://developers.google.com/bigquery/) would be cheaper and faster and more reliable.
amnigos 14y ago
This would be very useful for elastic EMR workloads, will be good for killing I/O bottlenecks.
Fizzer 14y ago
This is a game changer for big sites on EC2. The key word here is local: 2 TB of local SSD-backed storage. In this video [1], Foursquare says the biggest problem they're facing with EC2 is consistency in I/O performance. They say that the instance storage simply isn't fast enough for them, and while EBS is fast enough when RAIDed, it isn't consistent since it isn't local (EBS is traffic goes over the network). Reddit has also complained about EBS, but they've been able to move onto the instance storage. If you're willing to reserve the instance for 3 years, the average monthly cost becomes only $656. That's quite a good deal. Foursquare says in that video they're planning to migrate off of EC2, in part due to I/O performance. I'll be interested to hear whether or not this instance type changes their minds. [1] http://www.10gen.com/presentations/MongoNYC-2012/MongoDB-at-foursquare http://www.10gen.com/presentations/MongoNYC-2012/MongoDB-at-...
- jedberg 14y ago
  Netflix got a huge performance boost for Cassandra using the SSD instances: http://news.ycombinator.com/item?id=4264888 http://news.ycombinator.com/item?id=4264888
- XERQ 14y ago
  If you're willing to reserve the instance for 3 years, the average monthly cost becomes only $656. That's quite a good deal. The only problem with reserving that instance for 3 years is that better hardware always comes along, especially with the cost of SSDs coming down significantly every year. Usually if you're in the big-data space, your hardware is likely retired after 24 months (12 months if you're well funded) so locking yourself in for 36 months might be a bad investment.
  - jaylevitt 14y ago
    Has Amazon ever bumped the specs on existing hardware types? Or do they just create new hardware types? e.g. is it possible that if you get a 3-year reservation for an h1.xlarge, by 2015 h1.xlarge might have newer specs? I had thought that EC2 reservations were upgradeable, but a quick check on the forums shows you're right, they're not. Of course, you can play your own "tiered usage" game, like laptops in IT departments, where the old h1.xlarge becomes cheap enough to use as a second-tier machine and you go reserve the h1.xxlarge for Cassandra.
btb 14y ago
Would these be suitable to run a single big SQL Server on? I mean specwise they seem perfect for our size/use. They say data will survive a reboot, but what are the chances I would some day have to wake up in the middle of the night and having to restore the database to a new server? Would some of the recent amazon downtimes be one of those cases where that could happen?
- jedberg 14y ago
  That would be a really bad idea unless you have real-time replication to another server. SSD or not, you don't want to write data to the ephemeral store without a real-time backup, unless you're willing to lose it all.
Loic 14y ago
Another approach is something like OVH SSD servers (24GB ECC memory, 2x300GB SSD, £210/month) https://www.ovh.co.uk/dedicated_servers/mg_ssd_max.xml https://www.ovh.co.uk/dedicated_servers/mg_ssd_max.xml If you are using MongoDB, you take 3 or 4 of them and shard and you backup with "conventional" storage for the replica set. You end up with a 6 node cluster for less than the price of this Amazon instance. Lesson: You need to have a business which can benefit from a lot of start/stop of your instances for them to make sense from a pure financial point of view.
- cbg0 14y ago
  Some notes: 1. You can't order from OVH if you're from outside their list of approved countries. 2. You're not using any RAID and those are desktop grade SSD drives, and they tend to die out, sometimes without a clear warning, as they're not really intended for 24/7 server use.
  - Loic 14y ago
    1. The list of countries from where you can order servers is expanding on a regular basis. If your country is not there yet, it should come in the future. 2. You have 2 300GB with a RAID card (battery powered). So, you can put in RAID. For the reliability, I keep my fingers crossed (I have some of these servers) but no failures yet and an interview with the operators said that basically they an extremely good reliability. This is not marketing in this case, this is because they need it to be financially sustainable. By the way, do you know what kind of SSD (SLC, MLC, real disks or cards?) are used by Amazon?
    - caw 14y ago
      Being a storage admin, when you start to have gobs of hard drives, there's always one failing. You'll have dry spells where nothing fails, and then all of a sudden you're looking at 2,3+ failures on a system, although not necessarily on the same RAID group. If your hardware provider needs to cut down on the warranties to be financially sustainable, I'd be concerned. It looks like these are rentals and not purchases, so why wouldn't these guys be warrantying to Dell/HP or the drive manufacturer directly? Are they buying gray market to reduce the cost, trying to pass that savings off to you, but then in turn run out of recourse when they need to replace a drive? I'm just speculating; I have no idea if this company is good or not. I'm just concerned about the statement you made about the company, whether it's from your understanding or what they actually said.
mrb 14y ago
Anyone can report on the model of SSD they use (via ATA IDENTIFY)? My guess based on perf characteristics is each instance has 2 x 960GB OCZ Talos 2 C Series SSDs: http://www.oczenterprise.com/ssd-products/talos-2-c-sas-2.5-mlc.html http://www.oczenterprise.com/ssd-products/talos-2-c-sas-2.5-...
- spartango 14y ago
  It's very likely that they are using Intel SSDs (likely 320/330, maybe 510+) for a variety of reasons: Intel has a good drive reliability history, and is very enterprise friendly in bulk purchasing. Intel has had excellent firmware for their drives, which at the datacenter scale is valuable--people who have dealt with raid controller firmware(including Amazon) know all about this. Traditionally, Amazon has not used SAS drives in EC2, opting for lower cost SATA drives. It's also unlikely that Amazon is using small numbers of high capacity (>=500GB) drives because they still aren't perfectly price effective; price per gb is ok, but replacing a failed drive is more costly. Also keep in mind that to get to today, Amazon has been rolling these drives in huge numbers out through two enormous data centers, so it's unlikely that Amazon has picked a brand new drive (say the latest OCZ Vertex 4). There are other factors that Amazon has bumped into while testing drives, but they remain unreported and internal.
rit 14y ago
It seems we all may be missing the "backed" part - which I did on my first read through. They don't seem to be revealing how much of the logical volume is actual SSD, which I think is why they're instead putting down IOPS numbers. Still, a huge and significant improvement over anything previously available. I'm looking forward to playing with it.
- spartango 14y ago
  It's very likely that the entire volume is SSD, otherwise the tp99 would be atrocious (and if you look at the netflix numbers its actually quite good). The reason for the broad ranges is more likely that the SSDs have some inherent performance variability due to wear leveling and GC processes. Additionally, as time goes on, multi-tenancy will start to come into play (if Amazon has smaller SSD instances or larger machines), which will stay within that range comfortably.
- jedberg 14y ago
  The whole volume is SSD.
josephcp 14y ago
Reminder since it's a pair of SSDs and most people will probably look into using this for their DB store: If you use current generation controllers/software & SSDs, you're going to have a bad time if you turn on RAID and don't know exactly what you're doing. TRIM ( https://en.wikipedia.org/wiki/TRIM https://en.wikipedia.org/wiki/TRIM ) isn't supported with RAID on SSD today on hardware controllers and most distributions of linux don't support TRIM on RAID out of the box if you're doing software RAID, so you're going to see performance plummet like a rock after you do one pass of writes on the disk. In many RAID configurations, you're going to zero-write the entire disk when formatting it, so performance is going to suck from the get-go. For this reason, even if you have a tiny database and don't expect to write 1TB worth of data, your performance might still suck. Personally, I haven't tried linux software md TRIM in production, the patch is pretty recent, so you're on your own here (if possible, scaling out horizontally may be a solution to consider for redundancy, I have no idea what Amazon using for SSDs, but recent Sandforce generations fail all the time, so plan for that). If you don't know to look for this issue, you're going to be scratching your head when your RAID10 SSD configuration write throughput is worse than a single 7200rpm drive. On the other hand, IOPS on SSDs are AMAZING for databases/datastores. Amazon may have solved this for you already behind their visualization instance, and they might be running their own software striping behind whatever raid you're doing, so be sure to test it out fully first.
- nirvdrum 14y ago
  Interesting. Do you happen to know if using LVM instead of mdadm is susceptible to the same issues?
  - js2 14y ago
    In addition, keep in mind that MD (software raid) does not support discards. In contrast, the logical volume manager (LVM) and the device-mapper (DM) targets that LVM uses do support discards. The only DM targets that do not support discards are dm-snapshot, dm-crypt, and dm-raid45. Discard support for the dm-mirror was added in Red Hat Enterprise Linux 6.1. Red Hat also warns that software RAID levels 1, 4, 5, and 6 are not recommended for use on SSDs. During the initialization stage of these RAID levels, some RAID management utilities (such as mdadm) write to all of the blocks on the storage device to ensure that checksums operate properly. This will cause the performance of the SSD to degrade quickly. http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/newmds-ssdtuning.html#ssddeploy http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6...
jaredstenquist 14y ago
Since I'm spending more than $1,000/month for an RDS master, which is backed by EBS, i'm intrigued at the idea of running our database off of these. Of course I'd lose all the awesome automated features of RDS, but worth considering. Maybe AWS will release a High Performance RDS option that runs off of them. Wishful thinking.
drudru11 14y ago
This is nice, but they need a smaller instance that costs less that also has SSD. Until then, I'm not going to blow $5k a month on a system like this.
level09 14y ago
wow, would like to do a redis benchmark on one of those !

196 points by jeffbarr 14y ago

beedogs 14y ago

Amazon will find some way to make this slower than shit and less reliable than a campaign promise.

jedberg 14y ago
I've used them and they are quite performant. Definitely lives up to its promise.

oomkiller 14y ago

Hopefully we will start to see some other providers offer SSD-backed storage since Amazon does it now. It would be nice if they offered it on some smaller instances too though.

thegyppo 14y ago
Storm on Demand have done SSD storage for a while now, pretty solid IOPS numbers too. I'll run a benchmark comparison tonight.
- thegyppo 14y ago
  Ran some benchmarks, really insane IOPS on this plan. Pretty average CPU performance (they're using fairly old E5620's). http://blog.serverbear.com/post/27553311076/hi1-4xlarge-benchmarks http://blog.serverbear.com/post/27553311076/hi1-4xlarge-benc...
  - signifiers 14y ago
    I just ran the numbers on the new EC2 instance, and I'm pretty skeptical about the benchmarks above. I'm not sure that, for example, a half second of dd /dev/zero really tells us much. When interpreting any benchmarks on EC2, it's important to understand that there is a 5-10% read/write performance hit on first use because AWS uses lazy block wipes between customer instance launches. See http://www.youtube.com/watch?v=IedaYaKsb-4#t=29m49s http://www.youtube.com/watch?v=IedaYaKsb-4#t=29m49s (should pre-cue, if not, skip to 29:49). This is referenced in the docs, but it's easy to miss: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/InstanceStorage.html?r=409#disk-performance http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/In...) So here you go, for hi1.4xlarge: * Summary for the impatient - After initialization (i.e., second-write), quasi-realistic I/O on the new SSD EC2 instances sustains writes @ 420 MB/sec and reads @ 6 GB/sec. The entire 8.6GB / filesystem copied over to SSD in 21 seconds. Not bad. * # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 8.0G 1.1G 6.9G 14% / tmpfs 30G 0 30G 0% /dev/shm /dev/xvdf 1023G 16G 957G 2% /media/ephemeral0 (Note: /dev/xvdf and /dev/xvdg are just soft links to /dev/sdf and /dev/sdg respectively) Crude stats on first-use: # hdparm -tT /dev/xvdf /dev/xvdf: Timing cached reads: 14788 MB in 1.99 seconds = 7446.69 MB/sec Timing buffered disk reads: 1066 MB in 3.00 seconds = 355.04 MB/sec Wipe the device: dd if=/dev/zero of=/dev/xvdf bs=1M& pid=$! while true; do kill -USR1 $pid; sleep 4; done; [...] dd: writing `/dev/xvdf': No space left on device 1048567+17 records in 1048566+17 records out 1099511627776 bytes (1.1 TB) copied, 1955.42 s, 562 MB/s Stats after zero-wipe (dd /dev/zero) to device: hdparm -tT /dev/xvdf /dev/xvdf: Timing cached reads: 13260 MB in 1.99 seconds = 6673.05 MB/sec Timing buffered disk reads: 1124 MB in 3.01 seconds = 374.02 MB/sec hdparm -tT /dev/xvdf /dev/xvdf: Timing cached reads: 11188 MB in 1.99 seconds = 5624.17 MB/sec Timing buffered disk reads: 1122 MB in 3.00 seconds = 373.99 MB/sec hdparm -tT /dev/xvdf /dev/xvdf: Timing cached reads: 12930 MB in 1.99 seconds = 6505.78 MB/sec Timing buffered disk reads: 1124 MB in 3.00 seconds = 374.15 MB/sec Confirming Effect Of Pre-wiped I/O: hdparm -tT /dev/xvdg Timing cached reads: 11796 MB in 1.99 seconds = 5931.68 MB/sec Timing buffered disk reads: 1038 MB in 3.00 seconds = 345.87 MB/sec hdparm -tT /dev/xvdg /dev/xvdg: Timing cached reads: 12658 MB in 1.99 seconds = 6367.41 MB/sec Timing buffered disk reads: 1050 MB in 3.00 seconds = 349.47 MB/sec hdparm -tT /dev/xvdg /dev/xvdg: Timing cached reads: 12856 MB in 1.99 seconds = 6468.39 MB/sec Timing buffered disk reads: 1066 MB in 3.00 seconds = 354.80 MB/sec Pre- Vs. Post-wipe performance: 373.6 MB/sec vs. 349.3 MB/sec (6-7% speed improvement) Somewhat more real-world numbers: dd if=/dev/sda1 of=/dev/xvdf bs=1M 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 19.7876 s, 434 MB/s dd if=/dev/sda1 of=/dev/xvdf bs=1M 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 20.0365 s, 429 MB/s dd if=/dev/sda1 of=/dev/xvdf bs=1M 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 21.4193 s, 401 MB/s *Edit: formatting
j-kidd 14y ago
Check out this long thread on WHT, started 2 months ago: http://www.webhostingtalk.com/showthread.php?t=1151090 http://www.webhostingtalk.com/showthread.php?t=1151090 From my limited testing, a 1 Node instance from GridVirt at $30/month can compile gcc-4.6.3 within 15 minutes (not exactly an IO-intensive example...)

ehsanu1 14y ago

This has been a long time coming, but AWS has consistently been improving their service (as long as you can ignore the particularly bad reliability as of late). It's telling that they have only enabled this for a huge (quadruple extra large) instance type. It's probably hard to make this work for someone who just wants a 10GB disk with great IO. The problem at the low end is that disks are larger and would thus have to be divided up to make proper use of them, leading to IO contention.. The high IO options will probably only ever be available for pretty large instances.

oomkiller 14y ago
It seems like they could spread a bunch of smaller instances over an array of SSDs, maybe just offer less space at a higher price? SSDs are naturally better at concurrent access, so while you might not get 150k+ IOPS, you could get at least 10k or so, with the expected low random access times. Imagine an SSD-backed EBS.
cperciva 14y ago
It's telling that they have only enabled this for a huge (quadruple extra large) instance type. My guess (not based on any knowledge of EC2 internals) is that they don't have any way to do fair I/O sharing between guests. If they did, they could split these boxes into 32 small instances with 1 ECU, 1.7 GB RAM, and a 60 GB disk with 2500 random reads / 250-4000 random writes per second.
- XERQ 14y ago
  Xen offers easy ways of doing fair I/O sharing between guests. These servers they're using are most likely multi-tenant systems with 256-512GB of RAM and 6-12TB of SSD storage. Providers don't like keeping expensive systems around that aren't making money, especially when demand changes every hour, so I expect that they have at least 4 instances sharing the I/O of each host (especially when they mention broad ranges of expected I/O). The most likely reason for not slicing these systems up to smaller instances is they want to maintain consistent, high performance I/O.
  - 14y ago

bravura 14y ago

What are good use-cases for on-demand high I/O servers? At $3.10/hr, these instances work out to $2k/mo. There are probably many more cost-effective options if you want a 2TB SSD server. Since the benefit of using EC2 is that you can provision instances elastically, what are the sorts of scenarios in which one needs to provision high I/O servers elastically? [edit: A few minutes of Googling, and I can't find any dedicated servers with 2 TB of SSD.]

saurik 14y ago
That is only one of the benefits of EC2. If you are not using elasticity, then you have to factor in the reserved instance pricing, which drops the prices down by 71% (as in, to 29% of the list price; and I mean even if you include the up-front cost: that's overall savings). You, like most people who comment on the price of EC2, do not seem to be taking this into consideration. :(
- joe_bloggs 14y ago
  What are some of the other benefits, other than elasticity or on-demand ness, and not having to spend a huge amount upfront?
  - saurik 14y ago
    One example, which stems from "on-demand ness" (which you added: I was only responding to "elasticity"), is that you can do "test runs" of migrations and deployments without even thinking about it: you can rent, for just an hour, a setup identical to your existing one, often based on a consistent and atomic snapshot of your production machine, so you can try something "likely correct but possibly horribly wrong"; then, if it works, rather than replicating the change on your "real" machine, you can just cut over to the new one and shut down the old. Way too many people seem to believe that the only benefit of "on-demand" is "elasticity", and then make bogus arguments here that "if you can plan your traffic you shouldn't be using EC2": EC2 is cheaper than people like to claim (and is in fact quite price competitive) and your ability to turn on/off machines on a whim changes the way you look at hardware so drastically that, in all honesty, it makes traditional ways of dealing with hardware seem draconian and only worth putting up with if you are dealing with some weird corner case or have horribly special requirements.

damian2000 14y ago

The pricing in the blog post is a bit unclear - the prices on http://aws.amazon.com/ec2/pricing/ http://aws.amazon.com/ec2/pricing/ are ... US East $3.10 for linux and $3.58 for windows. EU West $3.41 for linux and $3.58 for windows. (Reserved instance prices are cheaper)

jbarham 14y ago

FWIW it costs $27,156 a year on-demand or $12,720 as a reserved heavy utilization instance. For heavy analytics workloads I'd bet that Google BigQuery (https://developers.google.com/bigquery/ https://developers.google.com/bigquery/) would be cheaper and faster and more reliable.

amnigos 14y ago

This would be very useful for elastic EMR workloads, will be good for killing I/O bottlenecks.

Fizzer 14y ago

This is a game changer for big sites on EC2. The key word here is local: 2 TB of local SSD-backed storage. In this video [1], Foursquare says the biggest problem they're facing with EC2 is consistency in I/O performance. They say that the instance storage simply isn't fast enough for them, and while EBS is fast enough when RAIDed, it isn't consistent since it isn't local (EBS is traffic goes over the network). Reddit has also complained about EBS, but they've been able to move onto the instance storage. If you're willing to reserve the instance for 3 years, the average monthly cost becomes only $656. That's quite a good deal. Foursquare says in that video they're planning to migrate off of EC2, in part due to I/O performance. I'll be interested to hear whether or not this instance type changes their minds. [1] http://www.10gen.com/presentations/MongoNYC-2012/MongoDB-at-foursquare http://www.10gen.com/presentations/MongoNYC-2012/MongoDB-at-...

jedberg 14y ago
Netflix got a huge performance boost for Cassandra using the SSD instances: http://news.ycombinator.com/item?id=4264888 http://news.ycombinator.com/item?id=4264888
XERQ 14y ago
If you're willing to reserve the instance for 3 years, the average monthly cost becomes only $656. That's quite a good deal. The only problem with reserving that instance for 3 years is that better hardware always comes along, especially with the cost of SSDs coming down significantly every year. Usually if you're in the big-data space, your hardware is likely retired after 24 months (12 months if you're well funded) so locking yourself in for 36 months might be a bad investment.
- jaylevitt 14y ago
  Has Amazon ever bumped the specs on existing hardware types? Or do they just create new hardware types? e.g. is it possible that if you get a 3-year reservation for an h1.xlarge, by 2015 h1.xlarge might have newer specs? I had thought that EC2 reservations were upgradeable, but a quick check on the forums shows you're right, they're not. Of course, you can play your own "tiered usage" game, like laptops in IT departments, where the old h1.xlarge becomes cheap enough to use as a second-tier machine and you go reserve the h1.xxlarge for Cassandra.

btb 14y ago

Would these be suitable to run a single big SQL Server on? I mean specwise they seem perfect for our size/use. They say data will survive a reboot, but what are the chances I would some day have to wake up in the middle of the night and having to restore the database to a new server? Would some of the recent amazon downtimes be one of those cases where that could happen?

jedberg 14y ago
That would be a really bad idea unless you have real-time replication to another server. SSD or not, you don't want to write data to the ephemeral store without a real-time backup, unless you're willing to lose it all.

Loic 14y ago

Another approach is something like OVH SSD servers (24GB ECC memory, 2x300GB SSD, £210/month) https://www.ovh.co.uk/dedicated_servers/mg_ssd_max.xml https://www.ovh.co.uk/dedicated_servers/mg_ssd_max.xml If you are using MongoDB, you take 3 or 4 of them and shard and you backup with "conventional" storage for the replica set. You end up with a 6 node cluster for less than the price of this Amazon instance. Lesson: You need to have a business which can benefit from a lot of start/stop of your instances for them to make sense from a pure financial point of view.

cbg0 14y ago
Some notes: 1. You can't order from OVH if you're from outside their list of approved countries. 2. You're not using any RAID and those are desktop grade SSD drives, and they tend to die out, sometimes without a clear warning, as they're not really intended for 24/7 server use.
- Loic 14y ago
  1. The list of countries from where you can order servers is expanding on a regular basis. If your country is not there yet, it should come in the future. 2. You have 2 300GB with a RAID card (battery powered). So, you can put in RAID. For the reliability, I keep my fingers crossed (I have some of these servers) but no failures yet and an interview with the operators said that basically they an extremely good reliability. This is not marketing in this case, this is because they need it to be financially sustainable. By the way, do you know what kind of SSD (SLC, MLC, real disks or cards?) are used by Amazon?
  - caw 14y ago
    Being a storage admin, when you start to have gobs of hard drives, there's always one failing. You'll have dry spells where nothing fails, and then all of a sudden you're looking at 2,3+ failures on a system, although not necessarily on the same RAID group. If your hardware provider needs to cut down on the warranties to be financially sustainable, I'd be concerned. It looks like these are rentals and not purchases, so why wouldn't these guys be warrantying to Dell/HP or the drive manufacturer directly? Are they buying gray market to reduce the cost, trying to pass that savings off to you, but then in turn run out of recourse when they need to replace a drive? I'm just speculating; I have no idea if this company is good or not. I'm just concerned about the statement you made about the company, whether it's from your understanding or what they actually said.

mrb 14y ago

Anyone can report on the model of SSD they use (via ATA IDENTIFY)? My guess based on perf characteristics is each instance has 2 x 960GB OCZ Talos 2 C Series SSDs: http://www.oczenterprise.com/ssd-products/talos-2-c-sas-2.5-mlc.html http://www.oczenterprise.com/ssd-products/talos-2-c-sas-2.5-...

spartango 14y ago
It's very likely that they are using Intel SSDs (likely 320/330, maybe 510+) for a variety of reasons: Intel has a good drive reliability history, and is very enterprise friendly in bulk purchasing. Intel has had excellent firmware for their drives, which at the datacenter scale is valuable--people who have dealt with raid controller firmware(including Amazon) know all about this. Traditionally, Amazon has not used SAS drives in EC2, opting for lower cost SATA drives. It's also unlikely that Amazon is using small numbers of high capacity (>=500GB) drives because they still aren't perfectly price effective; price per gb is ok, but replacing a failed drive is more costly. Also keep in mind that to get to today, Amazon has been rolling these drives in huge numbers out through two enormous data centers, so it's unlikely that Amazon has picked a brand new drive (say the latest OCZ Vertex 4). There are other factors that Amazon has bumped into while testing drives, but they remain unreported and internal.

rit 14y ago

It seems we all may be missing the "backed" part - which I did on my first read through. They don't seem to be revealing how much of the logical volume is actual SSD, which I think is why they're instead putting down IOPS numbers. Still, a huge and significant improvement over anything previously available. I'm looking forward to playing with it.

spartango 14y ago
It's very likely that the entire volume is SSD, otherwise the tp99 would be atrocious (and if you look at the netflix numbers its actually quite good). The reason for the broad ranges is more likely that the SSDs have some inherent performance variability due to wear leveling and GC processes. Additionally, as time goes on, multi-tenancy will start to come into play (if Amazon has smaller SSD instances or larger machines), which will stay within that range comfortably.
jedberg 14y ago
The whole volume is SSD.

josephcp 14y ago

Reminder since it's a pair of SSDs and most people will probably look into using this for their DB store: If you use current generation controllers/software & SSDs, you're going to have a bad time if you turn on RAID and don't know exactly what you're doing. TRIM ( https://en.wikipedia.org/wiki/TRIM https://en.wikipedia.org/wiki/TRIM ) isn't supported with RAID on SSD today on hardware controllers and most distributions of linux don't support TRIM on RAID out of the box if you're doing software RAID, so you're going to see performance plummet like a rock after you do one pass of writes on the disk. In many RAID configurations, you're going to zero-write the entire disk when formatting it, so performance is going to suck from the get-go. For this reason, even if you have a tiny database and don't expect to write 1TB worth of data, your performance might still suck. Personally, I haven't tried linux software md TRIM in production, the patch is pretty recent, so you're on your own here (if possible, scaling out horizontally may be a solution to consider for redundancy, I have no idea what Amazon using for SSDs, but recent Sandforce generations fail all the time, so plan for that). If you don't know to look for this issue, you're going to be scratching your head when your RAID10 SSD configuration write throughput is worse than a single 7200rpm drive. On the other hand, IOPS on SSDs are AMAZING for databases/datastores. Amazon may have solved this for you already behind their visualization instance, and they might be running their own software striping behind whatever raid you're doing, so be sure to test it out fully first.

nirvdrum 14y ago
Interesting. Do you happen to know if using LVM instead of mdadm is susceptible to the same issues?
- js2 14y ago
  In addition, keep in mind that MD (software raid) does not support discards. In contrast, the logical volume manager (LVM) and the device-mapper (DM) targets that LVM uses do support discards. The only DM targets that do not support discards are dm-snapshot, dm-crypt, and dm-raid45. Discard support for the dm-mirror was added in Red Hat Enterprise Linux 6.1. Red Hat also warns that software RAID levels 1, 4, 5, and 6 are not recommended for use on SSDs. During the initialization stage of these RAID levels, some RAID management utilities (such as mdadm) write to all of the blocks on the storage device to ensure that checksums operate properly. This will cause the performance of the SSD to degrade quickly. http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/newmds-ssdtuning.html#ssddeploy http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6...

jaredstenquist 14y ago

Since I'm spending more than $1,000/month for an RDS master, which is backed by EBS, i'm intrigued at the idea of running our database off of these. Of course I'd lose all the awesome automated features of RDS, but worth considering. Maybe AWS will release a High Performance RDS option that runs off of them. Wishful thinking.

drudru11 14y ago

This is nice, but they need a smaller instance that costs less that also has SSD. Until then, I'm not going to blow $5k a month on a system like this.

level09 14y ago

wow, would like to do a redis benchmark on one of those !