5 ms·

What is the use case for something like this versus a regular RDS service?

by rpicard 14y ago

amock 14y ago
This can scale up much more than a single RDS database since it spreads the data across multiple machines, but it's not exactly a replacement for MySQL database. It's also possible that this doesn't make use of EBS, which could make it perform more predictably and protect it from failure when EBS fails.
- semiquaver 14y ago
  > It's also possible that this doesn't make use of EBS This quote from the product page seems to indicate that EBS is not used for primary data storage: "it runs on hardware that is optimized for data warehousing, with local attached storage and 10GigE network connections between nodes."
- bvdbijl 14y ago
  Why isn't it a replacement? From what I read it seems it's like a very large and transparantly scalable SQL database
ceejayoz 14y ago
RDS has a maximum capacity of a terabyte, you'd need to shard to go beyond that.
dgreensp 14y ago
The answer is in the term "data warehousing" -- http://en.wikipedia.org/wiki/Data_warehouse http://en.wikipedia.org/wiki/Data_warehouse -- which has implication that you're going to be doing data mining on vast amounts of data, often historical data like logs or transaction histories. Google has systems like this for analyzing its request logs. Think of how many HTTP requests hit Google's front-end servers per second or hour or day. Each one has a few dozen pieces of data associated with it -- URL, client IP, headers, etc. Suppose I want to make a bar chart of how many requests came from France containing a certain header, each day for the last year. The system can do this query quickly if the requests are already bucketed by time interval, organized by column, compressed, and stored so that exactly the information needed can be brought into RAM quickly. It is a little funny, when you step back, that "storing," "archiving," and "warehousing" are different things and Amazon has services for each. Try explaining the difference between S3, RDS, EBS, Glacier, and Redshift to a layperson.
- ohashi 14y ago
  Would you mind explaining what you think each service is designed for and the use case(s) that they would be most appropriate for?
- rpicard 14y ago
  Thanks for the response. Would it make sense to say that this is more likely to be used for metadata (i.e. analytics, logs, etc.) while a normal RDB (or NoSQL DB) would be used for application data (i.e. users, settings, etc.)?

What is the use case for something like this versus a regular RDS service?

by rpicard 14y ago

What is the use case for something like this versus a regular RDS service?

amock 14y ago

This can scale up much more than a single RDS database since it spreads the data across multiple machines, but it's not exactly a replacement for MySQL database. It's also possible that this doesn't make use of EBS, which could make it perform more predictably and protect it from failure when EBS fails.

semiquaver 14y ago
> It's also possible that this doesn't make use of EBS This quote from the product page seems to indicate that EBS is not used for primary data storage: "it runs on hardware that is optimized for data warehousing, with local attached storage and 10GigE network connections between nodes."
bvdbijl 14y ago
Why isn't it a replacement? From what I read it seems it's like a very large and transparantly scalable SQL database

ceejayoz 14y ago

RDS has a maximum capacity of a terabyte, you'd need to shard to go beyond that.

dgreensp 14y ago

The answer is in the term "data warehousing" -- http://en.wikipedia.org/wiki/Data_warehouse http://en.wikipedia.org/wiki/Data_warehouse -- which has implication that you're going to be doing data mining on vast amounts of data, often historical data like logs or transaction histories. Google has systems like this for analyzing its request logs. Think of how many HTTP requests hit Google's front-end servers per second or hour or day. Each one has a few dozen pieces of data associated with it -- URL, client IP, headers, etc. Suppose I want to make a bar chart of how many requests came from France containing a certain header, each day for the last year. The system can do this query quickly if the requests are already bucketed by time interval, organized by column, compressed, and stored so that exactly the information needed can be brought into RAM quickly. It is a little funny, when you step back, that "storing," "archiving," and "warehousing" are different things and Amazon has services for each. Try explaining the difference between S3, RDS, EBS, Glacier, and Redshift to a layperson.

ohashi 14y ago
Would you mind explaining what you think each service is designed for and the use case(s) that they would be most appropriate for?
rpicard 14y ago
Thanks for the response. Would it make sense to say that this is more likely to be used for metadata (i.e. analytics, logs, etc.) while a normal RDB (or NoSQL DB) would be used for application data (i.e. users, settings, etc.)?