3 ms·
100,000 points per day? If a point is just a 64bit time stamp, and 64 bit float value. That results in ~122Gb per day of data. (or 44.5Tb per year). I'd sugges
by valarauca1 12y ago
100,000 points per day? If a point is just a 64bit time stamp, and 64 bit float value. That results in ~122Gb per day of data. (or 44.5Tb per year).
I'd suggest looking into a horizontally scaling solution if that is actually every day, day to day use.
:.:.:
The traditional solution to this problem when your working with raw DAQ communication/values is you store your data logs on a cold storage cluster (lots of slow cheap HDD's).
While you store 'meta-data', when data was collected, by who, what device, what day, a pointer/path to orginal data. In a more active SQL data base.
:.:.:
I'm speaking from live data acquisition experience. May not 100% align with your use case. Feel free to email if you need more info.
- bsmith 12y agoWe actually see more on the order of a few 10s of MB per day (the data points are fairly small, and we've done some optimization to de-duplicate timestamps). Our entire database is currently less than 10GB.
- valarauca1 12y agoIf you de-duplicate time stamps how can you dynamically re-generate the data? I.E.: Receive a sine wave, and give back the same sine wave?
- bsmith 12y agoWe collect data from a number of sensors at once, at an interval of 15 seconds (nothing super high speed or precision, here), so a row contains only one timestamp (UNIX epoch), but several sensor readings. It's for an electricity metering application.
- linuxhansl 12y agoHuh? 100.000/day*128bits = 12.8mbit/day = 1.6mb/day. You read it as per second? That would indeed lead to 128gb/day.