3 ms·
Depending on what you are doing you can even try to write it yourself. It would be a good exercise. Here is a toy hand crafted time series storage design: Say
by rdtsc 12y ago
Depending on what you are doing you can even try to write it yourself. It would be a good exercise.
Here is a toy hand crafted time series storage design:
Say you are storing tuples of {<timestamp>,<datablob>}. Then querying it by timestamp.
Writer can store it in two files,open in append only mode only. One is the data file one is the index file. Data might look like:
<datablob1><datablob2>...
And an index file, it stores timestamps and offsets into the data files where the blobs are:
<timestamp1><offset1><timestamp2><offset2>...
If you need rolling fall-off. Then create new pairs of files every day (hour, week, month). And delete old ones as you go.
Then if you can ensure that your have time synchronization set up and timestamp are in increasing order (this might be hard). You can do binary searching. If you use rolling fall-offs. Then you can discard whole files periods based on the query range when you search.
All this would go into a directory. Reader and writer could be different processes. Your timestamp and offset sizes should be fixed length. Writer first appends to the data file and then writes the index. Reader knows how to find the last valid record by looking at the size of the file.
- ibebrett 12y agoI wouldn't recommend trying to do this yourself. Of course you can make something that kind of works, but making a resilient production ready database that is fault tolerant and scales is a lot harder than writing to a file.
- rdtsc 12y agoWell it was just a toy example I came up with in a couple of minutes. But sometimes depending on the requirements a file is enough. If you intimately know the and control the bytes that get written it is easier to understand and reason about your systems (that means optimizing it, scaling it, making it fault tolerant). Also one way to make a resilient and fault tolerant database is to have less code running. Sometimes the base libc and unix offer a good and stable base on which it is easy to build. If you append the file in read or append only mode. You can rely on certain behavior now. People in the past have bought into marketing crap and got stuff like MongoDB which would throw data over the fence and pray that it would be synced eventually (by default!). But heck it was WebScale(tm).
- ibebrett 12y agoThat is why you seriously audit your tools and why many in the industry avoid Mongo like the plague. Controlling the byes that gets written to a file is actually not simple at all, and its a huge research problem as far as file systems of databases. I'm just saying, I don't think writing your own db is every a very good idea, unless it is SO simple that you would barely call it a DB.