4 ms·

I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..

m4tthumphrey 13y ago
Me too, except I skipped straight to the end :)
DoubleCluster 13y ago
Just a plain old RDMS?
rubinelli 13y ago
Look into Cassandra. Very fast writes and scales linearly (although there is some elbow grease involved in keyspace distribution). Our analytics platform guys are very happy with it, after flirting with several other options (including big, beefy RDBMS). Great for large, flat, denormalized tables.
monstrado 13y ago
Based on their use case, I'd expect they went with either HBase or Cassandra. I'm quite partial to HBase, it's insanely scalable and has a lot of pretty amazing features, but at the cost of knowing exactly what you want to do with your application beforehand. I'm not too familiar with Cassandra, but the scalability of an HBase table is almost entirely dependent on your key design. Judging from their use case and requirements, they would likely use a incremental key design which would allow for super fast range scans, of course, this leads to region server hotspotting, which may or not may not be a big deal to them.
pablohoffman 13y ago
We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.
- monstrado 13y ago
  Clouderan here! Glad to hear you guys went with HBase, I'm looking forward to your follow up post. Will you detail your key design / architectural setup? Did you guys roll your own HBase environment or did you go with the CDH? If you're using the CDH version and have any questions, feel free to shoot an email to cdh-user.
  - pablohoffman 13y ago
    We are using CDH4.2 and have had a very positive experience so far. Cloudera has in fact been an inspiration for us to follow, you guys have really struck the right balance between open source and commercial support. We follow the same philosophy with Scrapy (an open source web crawling framework), as you do with Hadoop and its ecosystem.
    - monstrado 13y ago
      That's really awesome to hear, thanks for your kind words. I'm looking forward to the follow up blog, depending on your key design you may be able to take advantage of Impala for ad-hoc queries using SQL.

m4tthumphrey 13y ago

Me too, except I skipped straight to the end :)

DoubleCluster 13y ago

Just a plain old RDMS?

rubinelli 13y ago

Look into Cassandra. Very fast writes and scales linearly (although there is some elbow grease involved in keyspace distribution). Our analytics platform guys are very happy with it, after flirting with several other options (including big, beefy RDBMS). Great for large, flat, denormalized tables.

monstrado 13y ago

Based on their use case, I'd expect they went with either HBase or Cassandra. I'm quite partial to HBase, it's insanely scalable and has a lot of pretty amazing features, but at the cost of knowing exactly what you want to do with your application beforehand. I'm not too familiar with Cassandra, but the scalability of an HBase table is almost entirely dependent on your key design. Judging from their use case and requirements, they would likely use a incremental key design which would allow for super fast range scans, of course, this leads to region server hotspotting, which may or not may not be a big deal to them.

pablohoffman 13y ago

We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.

monstrado 13y ago
Clouderan here! Glad to hear you guys went with HBase, I'm looking forward to your follow up post. Will you detail your key design / architectural setup? Did you guys roll your own HBase environment or did you go with the CDH? If you're using the CDH version and have any questions, feel free to shoot an email to cdh-user.
- pablohoffman 13y ago
  We are using CDH4.2 and have had a very positive experience so far. Cloudera has in fact been an inspiration for us to follow, you guys have really struck the right balance between open source and commercial support. We follow the same philosophy with Scrapy (an open source web crawling framework), as you do with Hadoop and its ecosystem.
  - monstrado 13y ago
    That's really awesome to hear, thanks for your kind words. I'm looking forward to the follow up blog, depending on your key design you may be able to take advantage of Impala for ad-hoc queries using SQL.