4 ms·
That is pretty remarkable. One of the limitations of doing one's own version of mass analytics is the cost of acquiring, installing, configuring, and then maint
by ChuckMcM 11y ago
That is pretty remarkable. One of the limitations of doing one's own version of mass analytics is the cost of acquiring, installing, configuring, and then maintaining the hardware. Generally I've found AWS to be more expensive but you get to "turn it on, turn it off" which is not something you can do when you have to pay monthly for data center space.
It makes for an interesting exercise to load in your data, do your analytics, and then store out the meta data. I wonder if the oil and gas people are looking at this for pre-processing their seismic data dumps.
- dastbe 11y agoAre you using seismic to describe what the data is, how big the data is, or both?
- ChuckMcM 11y agoHere is a good link: http://www.seismicsurvey.com.au/ http://www.seismicsurvey.com.au/ Basically you take waves that are transiting the area of interest and do transforms on them to ascertain the structure underground. Dave Hitz of NetApp used to joke these guys have great compression algorithm, they can convert a terabyte of data into 1 bit (oil/no-oil). One of the challenges is that the algorithms are running in a volume of space, so 'nearest neigbor' in terms of samples has more than 8 vectors. In the early 2000's they would stream their raw data off tape cartridges into a beowulf type cluster, process it, and then store the post processed (and smaller) data to storage arrays. Then that post processed data would go through its own round of processing. One of their challenges was that they ended up duplicating the data on multiple nodes because they needed it for their algorithm and it was too slow to fetch it across the network. A single system image with a TB of memory would let them go back to some of their old mainframe algorithms which, I'm told, were much easier to maintain.
- samstave 11y agoJust curious - but wouldnt the GPU based instances be more efficient for oil and gas people? Or load a data set in this monster and then use GPU workers to hit it?
- koolba 11y agoGPUs work when the data is small and the calculation can be parallelized. Random access to memory from a GPU would be slow. It's more like a separate computer (or lots of separate computers) that you can send a small program to execute and get the result.
- ddorian43 11y agoWhy does everyone(really!) compare aws to colocation ? I've never head an aws-believer ever mention dedicated servers. Why don't you compare aws to building your own cpu?
- ChuckMcM 11y agoI suspect it is because "everyone" (which is to abscond with your definition) believes that colocation is an alternative to AWS (well the EC2 part anyway). I would be interested to hear how you seem them as not being comparable. On your definition of "aws-believer" is that someone who feels that AWS is a superior solution to deploying a web facing application in all cases? Does your definition include economics? (like $/month vs request/month vs latency?) Can I assume that you consider comparing AWS to building your own CPU as an apples to oranges comparison? I certainly do, because I define a CPU to be a small component part of a distributed system hosting a web facing application.
- ddorian43 11y agoby "everyone" I mean >90% of comments on this forum comparing aws to an alternative (90% of cases it's straight to colocation) Building your own datacenters is also an alternative to aws-ec2, but you go to that step by step (I think?) (dedicated > colocation > datacenter). In some cases when you have crazy growth you can skip a/some steps (ex: dropbox going from aws to their own datacenter) They don't even compare them in whatever $/request/latency/$metric since they don't even mention them. And dedicated has also many options from low-price+no-support-shitty-network to high price/good-network/support etc.
- biot 11y agoMore food for thought: how many neurons + synapses can one model with that amount of RAM?