3 ms·

I assume they mean "relative to [my idea of how much a company will make when serving that amount of traffic]" - which does depend on the industry they are in,

by samhw 4y ago

I assume they mean "relative to [my idea of how much a company will make when serving that amount of traffic]" - which does depend on the industry they are in, not just in terms of how profitable it is but also its 'user load coefficient', so to speak. (The first job I had was at a 10-person startup in a coworking space, barely making enough revenue to break even, and still consuming a vast, vast amount of data, because the product involved constant streams of sensor data from a vast number of tiny cheap devices. People tend to forget that not every single business is a CRUD web/mobile app whose average user accounts for $20/month revenue against at most a few hundred HTTP requests and a couple megabytes of disk.)

Dylan16807 4y ago
> a vast, vast amount of data, because the product involved constant streams of sensor data from a vast number of tiny cheap devices Even that depends on a lot of factors. Just as an example, if it takes $10 to build and deploy a sensor, and it returns one number per second, then $1 of RAM can hold 2+ years of data before archiving it.
- marginalia_nu 4y ago
  That ignores any type of structure to the data (such as timestamps and which sensor it is). It would also probably need to be indexed somehow.
  - Dylan16807 4y ago
    I left spare space for that. Especially if you store a minute of samples at a time.
- samhw 4y ago
  Hmm, how many bits are you allowing for the 'number'? Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Also, is that ordinary NVRAM? How do you protect against bit flips? Google's well-known paper[0] found a 0.22% average incidence of uncorrectable errors per DIMM - that's corruption of more bits than ECC can fix (and I'm not sure that your pricing is even for ECC RAM). Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust (i.e. redundancy against total loss, as well as handling the 'freak errors' in data correctness that become the norm when you're dealing with vast amounts of hardware). I'm a fan of the kind of minimalism that you're advocating, don't get me wrong, but it's from pushing that limit that I've learned what the hard limits are. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c...
  - Dylan16807 4y ago
    > Hmm, how many bits are you allowing for the 'number'? 32. Which I would expect to be overkill compared to the precision of your average sensor. > Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Which is the same for big chunks of readings, so I'm assuming a system that's able to store that metadata once per several readings. > How do you protect against bit flips? I dunno, I was just saying the cost of memory. You can have bit flips on any kind of server no matter how you're doing it, and they might get persisted, so I'd say that's out of scope. > average incidence of uncorrectable errors per DIMM That average is highly skewed by broken DIMMs though. They found a mean of a few thousand correctable errors per DIMM, but they also found that 80% of DIMMs on one platform and >96% of DIMMs on the other platform had zero correctable errors. > Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust Sure, disk for backup sounds lovely and would be extremely cheap compared to the RAM. I wasn't advocating having only the RAM copy, just saying that depending on other factors it might be reasonable for a RAM copy to be the main analysis database even for sensor data. The post that started this thread directly says you should be using persistent files as write-only dumps/logs.

by samhw 4y ago

Dylan16807 4y ago

> a vast, vast amount of data, because the product involved constant streams of sensor data from a vast number of tiny cheap devices Even that depends on a lot of factors. Just as an example, if it takes $10 to build and deploy a sensor, and it returns one number per second, then $1 of RAM can hold 2+ years of data before archiving it.

marginalia_nu 4y ago
That ignores any type of structure to the data (such as timestamps and which sensor it is). It would also probably need to be indexed somehow.
- Dylan16807 4y ago
  I left spare space for that. Especially if you store a minute of samples at a time.
samhw 4y ago
Hmm, how many bits are you allowing for the 'number'? Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Also, is that ordinary NVRAM? How do you protect against bit flips? Google's well-known paper[0] found a 0.22% average incidence of uncorrectable errors per DIMM - that's corruption of more bits than ECC can fix (and I'm not sure that your pricing is even for ECC RAM). Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust (i.e. redundancy against total loss, as well as handling the 'freak errors' in data correctness that become the norm when you're dealing with vast amounts of hardware). I'm a fan of the kind of minimalism that you're advocating, don't get me wrong, but it's from pushing that limit that I've learned what the hard limits are. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c...
- Dylan16807 4y ago
  > Hmm, how many bits are you allowing for the 'number'? 32. Which I would expect to be overkill compared to the precision of your average sensor. > Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Which is the same for big chunks of readings, so I'm assuming a system that's able to store that metadata once per several readings. > How do you protect against bit flips? I dunno, I was just saying the cost of memory. You can have bit flips on any kind of server no matter how you're doing it, and they might get persisted, so I'd say that's out of scope. > average incidence of uncorrectable errors per DIMM That average is highly skewed by broken DIMMs though. They found a mean of a few thousand correctable errors per DIMM, but they also found that 80% of DIMMs on one platform and >96% of DIMMs on the other platform had zero correctable errors. > Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust Sure, disk for backup sounds lovely and would be extremely cheap compared to the RAM. I wasn't advocating having only the RAM copy, just saying that depending on other factors it might be reasonable for a RAM copy to be the main analysis database even for sensor data. The post that started this thread directly says you should be using persistent files as write-only dumps/logs.