4 ms·

One of my favorite parts of my job is the ability to grab thousands of machines at low priority for the hell of it.

• packetslave 15y ago
  ...and without asking permission. "Gee, I wonder how much faster this would be on 5000 machines instead of 2000?" clickicky-click ... "yep, quite a bit faster."
  • jsnell 15y ago
    Not permission, but occasionally forgiveness. Task isolation isn't perfect, and backends don't scale infinitely.
• boyter 15y ago
  As someone at a large corp who just had a 3 day battle to get a new VM with 2 gig of ram and a shared CPU I suddenly feel like crying. I guess I have no real use for something of this magnitude, but I would still like to get my hands on a seriously large cluster just for the sake of expanding my experience. I don't suppose anyone knows of a cost effective way of doing so? I have spun up my own Hadoop clusters before but with only 4 machines its not exactly exciting.
  • jonburs 15y ago
    Amazon -- you can use Elastic Map Reduce [1] (the easiest way to run a transient Hadoop cluster) or spin up a bunch of EC2 nodes and deploy your own stack. Both methods can provide you access to very powerful hardware [2]. (Disclaimer: I work for Amazon, but not in AWS, and these are my own personal opinions.) [1] http://aws.amazon.com/elasticmapreduce/ http://aws.amazon.com/elasticmapreduce/ [2] http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/instance-types/
    • epistasis 15y ago
      I don't work for Amazon, have hundreds of non-AWS CPUs at my disposal, and still prefer AWS, as long as you don't have to do much I/O. Once there's a few terabytes of data or more, it's best to be off of AWS, because you'll never be able to saturate your CPUs. Even for smaller datasets of 10s of gigabytes, it's kind of iffy.