4 ms·
I'm a little confused by the schema he describes. He says represent "periods" in columns and "records" in rows. Are these two different units of time? For say
by abalone 13y ago
I'm a little confused by the schema he describes. He says represent "periods" in columns and "records" in rows. Are these two different units of time?
For say stock data that is sampled every second, is he saying there'd be one row per symbol per minute (a "record"), with 60 columns holding the value for each second (a "period")?
If so, does that mean the data is buffered in memory for 1 minute before getting written to the DB?
"Cassandra is really good for time-series data because you can write one column for each period in your series and then query across a range of time using sub-string matching. This is best done using columns for each period rather than rows, as you get huge IO efficiency wins from loading only a single row per query. Cassandra then has to at worst do one seek and then read for all the remaining data as it’s written in sequence to disk.
We designed our schema to use IDs that begin with timestamps so that they can be range queried over arbitrary periods like this, with each row representing one record and each column representing one period in the series. All data is then available to be queried on a row key and start and end times."
- marshray 13y agoMy dim understanding is that the term "column" in Cassandra doesn't mean what we think it means.
- nemothekid 13y agoNo, OP's wording can be confusing if you have never used Cassandra before. You are right. There is one row per symbol. However with Cassandra any given row can have any number of columns, so when you want to write a new value, you just create a new column for that second (/period). The writes are not buffered in memory.
- abalone 13y agoI see. From what I know about Cassandra, this is a much more expensive write than doing it as a new row. To do this he has to be using dynamic columns, and those are stored as one serialized blob per row. So the more data you have in the row, the more expensive the deserialization/reserialization is with each column you add. For very large series this could be an issue. But it sounds like this is tolerable for his app because the writes are distributed over time in a predictable fashion. I am a little surprised though at the author's claim that fetching a single big row results in "huge IO efficiency" over a range of small rows. I'd expect a small amount of overhead, but isn't it more or less the same amount of data being retrieved? What am I missing? EDIT: I see the author mentioned that it reduces disk seeks because it's all serialized together already. Sort of like you're defragging the series data on every write. I guess that makes sense. Personally I would probably look at using SSDs and keep the schema more "sane" and have more scalable writes, but that's just me.
- nemothekid 13y ago1.) You do not have to use dynamic columns for this. Unfortunately I've found in my own experience, as Cassandra has matured over the last year, alot of terminology has fallen in and out of fashion and its hard to recognize what is actually current. Dynamic columns in CQL3 has nothing to do with the behavior OP is talking about and in dynamic columns are sort-of a deprecated feature in Cassandra 1.2. In CQL3, OP's use-pattern in actually hidden if you didn't know any better. In short, there is no deserialization/reserializaion. OP's writes are append only. I have a similar use pattern to OP, and I haven't seen any performance issues with 100,000s of columns (on SSDs) 2.) The "huge IO efficiency" is similar to what you would see in any columnar data store. Wikipedia has a good walkthrough of it (http://en.wikipedia.org/wiki/Column-oriented_DBMS http://en.wikipedia.org/wiki/Column-oriented_DBMS). The short story is now there is fewer meta data between his values. -- In any case, it works out because Cassandra is far more well suited for this type use pattern than Mongo is. We migrated from MongoDB (on SSDs) to Cassandra for similar reasons. The perf-killer on Mongo in this scenario is the write lock.