6 ms·

EC2 High Memory Cluster Eight Extra Large Instance (244 GiB RAM)

65 points by jeffbarr 14y ago

pieter 14y ago
This instance was announced at RE:invent 2 months ago, and is now available. You can try one out for $0.34 / hour with spot instances.
- tiernano 14y ago
  It does not seem to be possible to launch a spot instance of one of these... the web interface wont allow it, anyway... Still, about $3 per hour to play with a machine of that size should be interesting...
  - res0nat0r 14y ago
    You might have to use the ec2-api-tools cli package to launch spot requests for this instance type right now. The AWS Console doesn't always seem to be updated at exactly the same time as new instance releases.
michaelt 14y ago
Almost all the big EC2 outages have been due to EBS, and you have to pay extra for it as well. Netflix steers clear of it [1]. Does anyone know why Cluster Compute and Cluster GPU Instances [2] (including this new one) make using EBS mandatory? [1] http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html http://techblog.netflix.com/2011/04/lessons-netflix-learned-... [2] http://aws.amazon.com/ec2/faqs/#Does_use_of_Cluster_Compute_Instances_differ_from_other_Amazon_EC2_instance_types http://aws.amazon.com/ec2/faqs/#Does_use_of_Cluster_Compute_...
- lucian1900 14y ago
  Likely because they are considered abstracted, throw away machines.
- cbsmith 14y ago
  I believe it is a function of the fact that you are "owning" the machine.
- tiernano 14y ago
  I dont think it uses EBS, since the storage is using SSDs. they are "in" the machine, not remote, like EBS disks. If the machine dies, or moves instance, your storage is gone... As for if your boot disk is on EBS, thats a different question and one i cant answer...
  - michaelt 14y ago
    Well, the article says "Support for EBS-backed AMIs only." i.e. you can only boot from EBS, you can't boot from instance storage.
- crb 14y ago
  Two reasons come to mind: * the failure rate of an EBS volume (assuming a bug-free EBS software stack!) should be less than any one spinning disk * these machines don't have disk space set aside for copying the root volumes to them from S3 to boot. (There's a 10GB limit, and to allow for this to be somewhere, you'd have to have another SSD, or partition the existing ones.) There's a good blog post about it on Rightscale: http://blog.rightscale.com/2012/11/02/new-ec2-instance-types-and-coordinated-failures-in-the-cloud/ http://blog.rightscale.com/2012/11/02/new-ec2-instance-types... esh's "You Should Use EBS Boot Instances" also lays out a number of good reasons to use EBS for boot: http://alestic.com/2012/01/ec2-ebs-boot-recommended http://alestic.com/2012/01/ec2-ebs-boot-recommended You at least have two 120GB ephemeral SSDs to play with when you're running - a luxury you don't have on the new M3 classes.
  - Dylan16807 14y ago
    It's not really the rate that's the issue here, it's correlation between failures. The chance of losing a third or more of your machines at once is enormously higher with EBS. Partitioning a disk should take seconds; I don't see why it would be an impediment. As it is, someone can do it themselves but now it'll be a custom job and less reliable.
jahewson 14y ago
I'd like to see a machine with an SSD from Amazon that wasn't so big and expensive.
- devopstom 14y ago
  I don't think that's possible. Big|Fast|Cheap, Choose two. (or one, under some circumstances).
  - Dylan16807 14y ago
    I think you misread the parent. jahewson asked for a machine that is Fast and Cheap and Not Big. More specifically they want to upgrade a weaker machine with an SSD instead of getting a high end machine with an SSD.
- signifiers 14y ago
  Nothing has been announced, but fwiw, I've been running benchmarks comparing the SSD block storage at Rackspace (http://www.rackspace.com/cloud/blockstorage/pricing/ http://www.rackspace.com/cloud/blockstorage/pricing/), as well as the all-SSD VMs from Digital Ocean (http://www.digitalocean.com/ http://www.digitalocean.com/), and frankly, the CPU & I/O blow away EC2 so far. Compared to Google Compute Engine (https://cloud.google.com/pricing/compute-engine https://cloud.google.com/pricing/compute-engine), CPU is on par, but obviously SSD is going to best even zippy local instance spindle storage. For example, calculating 8th order Fermat prime factors (single thread/core intensive, so ymmv), EC2 XLs are getting smoked by 20-50% vs. Digital Ocean's 2 core 2GB RAM/40GB SSD. Replicated in multiple data centers/zones a dozen times. Not bad for $20/month! Also, for the low end, remember "core" on EC2 is throttled at 50%-75% for the small & medium instances, translating to lots of stolen cpu time. Still working out the details on the Rackspace SSD EBS-equivalents, but will post update soon. But, agreed, given across-the-board I/O benefits of SSD, surprised Amazon isn't offering something in the non-cluster category. And at $2,500+ /mo, the new instances are definitely targeting a very niche use case for burst-heavy batching.
mariuz 14y ago
One way to scale Firebird Database is to put it on memory virtual disk or ramfs On linux i can mount a partition in memory (Install ubuntu/debian) sudo mkdir /mnt/ram sudo mount -t ramfs -o size=200G ramfs /mnt/ram mount to show you the partitions mounted and then move your database into the ram partition cp -rp /var/lib/firebird/2.5/data /mnt/ram That large instance could help you with the Firebird cache settings also the extra SSD could do wonders Inspired by the stack overflow big fat server architecture http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html http://highscalability.com/blog/2009/8/5/stack-overflow-arch...
- jacques_chester 14y ago
  Well, two things with that. Three really. First, does AWS allow ramdisks? It's been a few years since I tried to mount a ram disk on Xen, but it was not possible at the time. Second, most databases are smart enough to shuffle stuff into RAM for you, starting with indexes and commonly-accessed tables. Third, the "D" in ACID stands for "Durable". If you want an in-memory store, use an in-memory store. RDBMSes do sorta kinda rely on the assumption that stuff written to disk is actually written to a disk.
  - mariuz 14y ago
    Yes it AWS allows ramdisks , here is the output from a machine root@ip-amazon-ip:/home/ubuntu# mkdir /mnt/ram mount -t ramfs -o size=2G ramfs /mnt/ram mount ramfs on /mnt/ram type ramfs (rw,size=2G) There are the cache and memory settings Fore Firebird but is better to put the main full database on the fastest storage and that is RAM for the moment In Firebird we have a nifty feature called Shadow that creates a life snapshots of the active database (Think of it as Replication in real time) so you can activate it and keep a safe database on the SSD/HDD http://ibexpert.net/ibe/index.php?n=Doc.DatabaseShadow http://ibexpert.net/ibe/index.php?n=Doc.DatabaseShadow
profquail 14y ago
Colin Percival (cperciva) already has FreeBSD 9.1-RELEASE running on these! http://pastebin.com/SCMUJNDt http://pastebin.com/SCMUJNDt
zippie 14y ago
RAM isn't a cure all for big data applications and presents its own set of challenges. The largest issue is with traversal and data access latencies simply because of the virtual memory management overhead that is required. I've run into this problem with our btree search indexes on bare metal (~96GB), some more reading: https://github.com/johnj/llds https://github.com/johnj/llds
politician 14y ago
Recently, antirez pointed out that redis was essentially memtest86; consequently, he has had to spend a lot of time chasing down phantom bugs due to memory modules with stuck/flipped bits. At that time, I think HN was left wondering whether Amazon uses ECC memory in its EC2 machines. Do we know whether that's the case now?
- wmf 14y ago
  All Xeons and Opterons (in other words, all EC2) use ECC.
StefanKarpinski 14y ago
Some time ago (mid 2011), I was trying to use an AWS 64 GB machine for some largish work and discovered that I couldn't actually allocate anywhere near all of the 64 GB of RAM I was supposed to have access to. As I recall, writing a simple C program to do binary search to pinpoint exactly what the max amount of memory I could malloc was, it failed somewhere around 34 GB – so just over half of the memory I was supposed to have access to. Makes me wonder how much of this 244 GiB you can actually use when you spin up one of these machines.

65 points by jeffbarr 14y ago

This instance was announced at RE:invent 2 months ago, and is now available. You can try one out for $0.34 / hour with spot instances.

tiernano 14y ago
It does not seem to be possible to launch a spot instance of one of these... the web interface wont allow it, anyway... Still, about $3 per hour to play with a machine of that size should be interesting...
- res0nat0r 14y ago
  You might have to use the ec2-api-tools cli package to launch spot requests for this instance type right now. The AWS Console doesn't always seem to be updated at exactly the same time as new instance releases.

michaelt 14y ago

Almost all the big EC2 outages have been due to EBS, and you have to pay extra for it as well. Netflix steers clear of it [1]. Does anyone know why Cluster Compute and Cluster GPU Instances [2] (including this new one) make using EBS mandatory? [1] http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html http://techblog.netflix.com/2011/04/lessons-netflix-learned-... [2] http://aws.amazon.com/ec2/faqs/#Does_use_of_Cluster_Compute_Instances_differ_from_other_Amazon_EC2_instance_types http://aws.amazon.com/ec2/faqs/#Does_use_of_Cluster_Compute_...

lucian1900 14y ago
Likely because they are considered abstracted, throw away machines.
cbsmith 14y ago
I believe it is a function of the fact that you are "owning" the machine.
tiernano 14y ago
I dont think it uses EBS, since the storage is using SSDs. they are "in" the machine, not remote, like EBS disks. If the machine dies, or moves instance, your storage is gone... As for if your boot disk is on EBS, thats a different question and one i cant answer...
- michaelt 14y ago
  Well, the article says "Support for EBS-backed AMIs only." i.e. you can only boot from EBS, you can't boot from instance storage.
crb 14y ago
Two reasons come to mind: * the failure rate of an EBS volume (assuming a bug-free EBS software stack!) should be less than any one spinning disk * these machines don't have disk space set aside for copying the root volumes to them from S3 to boot. (There's a 10GB limit, and to allow for this to be somewhere, you'd have to have another SSD, or partition the existing ones.) There's a good blog post about it on Rightscale: http://blog.rightscale.com/2012/11/02/new-ec2-instance-types-and-coordinated-failures-in-the-cloud/ http://blog.rightscale.com/2012/11/02/new-ec2-instance-types... esh's "You Should Use EBS Boot Instances" also lays out a number of good reasons to use EBS for boot: http://alestic.com/2012/01/ec2-ebs-boot-recommended http://alestic.com/2012/01/ec2-ebs-boot-recommended You at least have two 120GB ephemeral SSDs to play with when you're running - a luxury you don't have on the new M3 classes.
- Dylan16807 14y ago
  It's not really the rate that's the issue here, it's correlation between failures. The chance of losing a third or more of your machines at once is enormously higher with EBS. Partitioning a disk should take seconds; I don't see why it would be an impediment. As it is, someone can do it themselves but now it'll be a custom job and less reliable.

jahewson 14y ago

I'd like to see a machine with an SSD from Amazon that wasn't so big and expensive.

devopstom 14y ago
I don't think that's possible. Big|Fast|Cheap, Choose two. (or one, under some circumstances).
- Dylan16807 14y ago
  I think you misread the parent. jahewson asked for a machine that is Fast and Cheap and Not Big. More specifically they want to upgrade a weaker machine with an SSD instead of getting a high end machine with an SSD.
signifiers 14y ago
Nothing has been announced, but fwiw, I've been running benchmarks comparing the SSD block storage at Rackspace (http://www.rackspace.com/cloud/blockstorage/pricing/ http://www.rackspace.com/cloud/blockstorage/pricing/), as well as the all-SSD VMs from Digital Ocean (http://www.digitalocean.com/ http://www.digitalocean.com/), and frankly, the CPU & I/O blow away EC2 so far. Compared to Google Compute Engine (https://cloud.google.com/pricing/compute-engine https://cloud.google.com/pricing/compute-engine), CPU is on par, but obviously SSD is going to best even zippy local instance spindle storage. For example, calculating 8th order Fermat prime factors (single thread/core intensive, so ymmv), EC2 XLs are getting smoked by 20-50% vs. Digital Ocean's 2 core 2GB RAM/40GB SSD. Replicated in multiple data centers/zones a dozen times. Not bad for $20/month! Also, for the low end, remember "core" on EC2 is throttled at 50%-75% for the small & medium instances, translating to lots of stolen cpu time. Still working out the details on the Rackspace SSD EBS-equivalents, but will post update soon. But, agreed, given across-the-board I/O benefits of SSD, surprised Amazon isn't offering something in the non-cluster category. And at $2,500+ /mo, the new instances are definitely targeting a very niche use case for burst-heavy batching.

mariuz 14y ago

One way to scale Firebird Database is to put it on memory virtual disk or ramfs On linux i can mount a partition in memory (Install ubuntu/debian) sudo mkdir /mnt/ram sudo mount -t ramfs -o size=200G ramfs /mnt/ram mount to show you the partitions mounted and then move your database into the ram partition cp -rp /var/lib/firebird/2.5/data /mnt/ram That large instance could help you with the Firebird cache settings also the extra SSD could do wonders Inspired by the stack overflow big fat server architecture http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html http://highscalability.com/blog/2009/8/5/stack-overflow-arch...

jacques_chester 14y ago
Well, two things with that. Three really. First, does AWS allow ramdisks? It's been a few years since I tried to mount a ram disk on Xen, but it was not possible at the time. Second, most databases are smart enough to shuffle stuff into RAM for you, starting with indexes and commonly-accessed tables. Third, the "D" in ACID stands for "Durable". If you want an in-memory store, use an in-memory store. RDBMSes do sorta kinda rely on the assumption that stuff written to disk is actually written to a disk.
- mariuz 14y ago
  Yes it AWS allows ramdisks , here is the output from a machine root@ip-amazon-ip:/home/ubuntu# mkdir /mnt/ram mount -t ramfs -o size=2G ramfs /mnt/ram mount ramfs on /mnt/ram type ramfs (rw,size=2G) There are the cache and memory settings Fore Firebird but is better to put the main full database on the fastest storage and that is RAM for the moment In Firebird we have a nifty feature called Shadow that creates a life snapshots of the active database (Think of it as Replication in real time) so you can activate it and keep a safe database on the SSD/HDD http://ibexpert.net/ibe/index.php?n=Doc.DatabaseShadow http://ibexpert.net/ibe/index.php?n=Doc.DatabaseShadow

profquail 14y ago

Colin Percival (cperciva) already has FreeBSD 9.1-RELEASE running on these! http://pastebin.com/SCMUJNDt http://pastebin.com/SCMUJNDt

zippie 14y ago

RAM isn't a cure all for big data applications and presents its own set of challenges. The largest issue is with traversal and data access latencies simply because of the virtual memory management overhead that is required. I've run into this problem with our btree search indexes on bare metal (~96GB), some more reading: https://github.com/johnj/llds https://github.com/johnj/llds

politician 14y ago

Recently, antirez pointed out that redis was essentially memtest86; consequently, he has had to spend a lot of time chasing down phantom bugs due to memory modules with stuck/flipped bits. At that time, I think HN was left wondering whether Amazon uses ECC memory in its EC2 machines. Do we know whether that's the case now?

wmf 14y ago
All Xeons and Opterons (in other words, all EC2) use ECC.

StefanKarpinski 14y ago

Some time ago (mid 2011), I was trying to use an AWS 64 GB machine for some largish work and discovered that I couldn't actually allocate anywhere near all of the 64 GB of RAM I was supposed to have access to. As I recall, writing a simple C program to do binary search to pinpoint exactly what the max amount of memory I could malloc was, it failed somewhere around 34 GB – so just over half of the memory I was supposed to have access to. Makes me wonder how much of this 244 GiB you can actually use when you spin up one of these machines.