3 ms·

Hmm, how many bits are you allowing for the 'number'? Also, you need to identify which user it belongs to, as well as - in our case - which device and which sen

by samhw 4y ago

Hmm, how many bits are you allowing for the 'number'? Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Also, is that ordinary NVRAM? How do you protect against bit flips? Google's well-known paper[0] found a 0.22% average incidence of uncorrectable errors per DIMM - that's corruption of more bits than ECC can fix (and I'm not sure that your pricing is even for ECC RAM). Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust (i.e. redundancy against total loss, as well as handling the 'freak errors' in data correctness that become the norm when you're dealing with vast amounts of hardware). I'm a fan of the kind of minimalism that you're advocating, don't get me wrong, but it's from pushing that limit that I've learned what the hard limits are. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c...

Dylan16807 4y ago
> Hmm, how many bits are you allowing for the 'number'? 32. Which I would expect to be overkill compared to the precision of your average sensor. > Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Which is the same for big chunks of readings, so I'm assuming a system that's able to store that metadata once per several readings. > How do you protect against bit flips? I dunno, I was just saying the cost of memory. You can have bit flips on any kind of server no matter how you're doing it, and they might get persisted, so I'd say that's out of scope. > average incidence of uncorrectable errors per DIMM That average is highly skewed by broken DIMMs though. They found a mean of a few thousand correctable errors per DIMM, but they also found that 80% of DIMMs on one platform and >96% of DIMMs on the other platform had zero correctable errors. > Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust Sure, disk for backup sounds lovely and would be extremely cheap compared to the RAM. I wasn't advocating having only the RAM copy, just saying that depending on other factors it might be reasonable for a RAM copy to be the main analysis database even for sensor data. The post that started this thread directly says you should be using persistent files as write-only dumps/logs.

by samhw 4y ago

Dylan16807 4y ago

> Hmm, how many bits are you allowing for the 'number'? 32. Which I would expect to be overkill compared to the precision of your average sensor. > Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Which is the same for big chunks of readings, so I'm assuming a system that's able to store that metadata once per several readings. > How do you protect against bit flips? I dunno, I was just saying the cost of memory. You can have bit flips on any kind of server no matter how you're doing it, and they might get persisted, so I'd say that's out of scope. > average incidence of uncorrectable errors per DIMM That average is highly skewed by broken DIMMs though. They found a mean of a few thousand correctable errors per DIMM, but they also found that 80% of DIMMs on one platform and >96% of DIMMs on the other platform had zero correctable errors. > Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust Sure, disk for backup sounds lovely and would be extremely cheap compared to the RAM. I wasn't advocating having only the RAM copy, just saying that depending on other factors it might be reasonable for a RAM copy to be the main analysis database even for sensor data. The post that started this thread directly says you should be using persistent files as write-only dumps/logs.