3 ms·

Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction

by jhugg 16y ago

Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.

charrington 16y ago
I think that at the rates Twitter is writing counter data (Many TBs per day denormalized, ~0.5TB normalized), a RAM-based solution like VoltDB would be prohibitively expensive. Rainbird allows Twitter to use cheap disk-based storage but still get acceptable (sub-second) latency.
- jhugg 16y ago
  Perhaps, but it really depends on what you want to do exactly. You can probably store a lot less and get the same throughput. - VoltDB isn't log-structured, so you really only have to store the state. How fast you can mutate it isn't limited by RAM amounts. We see use cases with utter firehoses of data that update just tens or hundreds of gigabytes of state. - Beyond normalization, you can probably reduce the number of redundant counters, e.g. use SQL to count which URLs start with "amazon". This would be painful in many systems, but depending on the query, can often be done at scale in VoltDB. - The byte overhead per counter is also likely much lower in an ACID/Relational store. Finally, VoltDB is designed to migrate data to disk based stores (such as Hadoop or an OLAP store) as memory fills up. This is a feature we're working very hard on see as a big differentiator. It adds complexity if you need to query across stores, but you get a best-of-both-worlds feature set.
JoachimSchipper 16y ago
I didn't know VoltDB was open source, actually. Cool! Is the database really entirely in-memory, though?
- jhugg 16y ago
  VoltDB relies on syncronous replication to protect data from hardware and network failures. It also supports continuous and transactionally-isolated snapshotting to disk.
lsb 16y ago
VoltDB looks pretty awesome, but I'm pretty concerned by its lack of ability to join a table to itself, to have over six tables in a join, to aggregate or select distinct arbitrary values. http://community.voltdb.com/docs/ReleaseNotes/index http://community.voltdb.com/docs/ReleaseNotes/index
- jhugg 16y ago
  At VoltDB, we're working on improving our SQL support. Initially, we focused on core SQL useful for OLTP. We've added functionality with every release and we plan to continue in 2011. For example, the March release of VoltDB will support more than 6 table joins and has some improvements to aggregation and distinct code. It also has a much more usable explain plan feature. We feel that VoltDB offers one of the richer query interfaces of systems that scale to its level, but we don't plan to sit still.

by jhugg 16y ago

Shameless self-promotion, but important point nonetheless: VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.

charrington 16y ago

I think that at the rates Twitter is writing counter data (Many TBs per day denormalized, ~0.5TB normalized), a RAM-based solution like VoltDB would be prohibitively expensive. Rainbird allows Twitter to use cheap disk-based storage but still get acceptable (sub-second) latency.

jhugg 16y ago
Perhaps, but it really depends on what you want to do exactly. You can probably store a lot less and get the same throughput. - VoltDB isn't log-structured, so you really only have to store the state. How fast you can mutate it isn't limited by RAM amounts. We see use cases with utter firehoses of data that update just tens or hundreds of gigabytes of state. - Beyond normalization, you can probably reduce the number of redundant counters, e.g. use SQL to count which URLs start with "amazon". This would be painful in many systems, but depending on the query, can often be done at scale in VoltDB. - The byte overhead per counter is also likely much lower in an ACID/Relational store. Finally, VoltDB is designed to migrate data to disk based stores (such as Hadoop or an OLAP store) as memory fills up. This is a feature we're working very hard on see as a big differentiator. It adds complexity if you need to query across stores, but you get a best-of-both-worlds feature set.

JoachimSchipper 16y ago

I didn't know VoltDB was open source, actually. Cool! Is the database really entirely in-memory, though?

jhugg 16y ago
VoltDB relies on syncronous replication to protect data from hardware and network failures. It also supports continuous and transactionally-isolated snapshotting to disk.

lsb 16y ago

VoltDB looks pretty awesome, but I'm pretty concerned by its lack of ability to join a table to itself, to have over six tables in a join, to aggregate or select distinct arbitrary values. http://community.voltdb.com/docs/ReleaseNotes/index http://community.voltdb.com/docs/ReleaseNotes/index

jhugg 16y ago
At VoltDB, we're working on improving our SQL support. Initially, we focused on core SQL useful for OLTP. We've added functionality with every release and we plan to continue in 2011. For example, the March release of VoltDB will support more than 6 table joins and has some improvements to aggregation and distinct code. It also has a much more usable explain plan feature. We feel that VoltDB offers one of the richer query interfaces of systems that scale to its level, but we don't plan to sit still.