3 ms·
with the risk of sounding cynical here, but it feels this is a solution from people that aren't familiar with what the problem is about, ending up in optimizing
by halayli 10y ago
with the risk of sounding cynical here, but it feels this is a solution from people that aren't familiar with what the problem is about, ending up in optimizing the wrong areas.
Few observations here:
* using udp means you don't care if you lose data and that's bad. if your buffers are full your data will be silently dropped, aside from network glitches and metric sizes that can easily exceed 512 bytes. Your buffers can get full when you wait on disk io etc..
Optimizing your network stack when your bottleneck is disk io doesn't buy you much. But the real issue that they haven't tackled is data distribution. If you can horizontally scale your data optimally, then when you hit a bottleneck (whatever it is) you can always increase your cluster size and solve your bottlenecks. In the current solution, If the machine dies, you will lose tons of data.
cow pipelining exist for a very long time now. There's nothing new in what they are introducing.
The persistent data they hold in memory is not guaranteed to be recovered if they crash before an fsync/msync.
Secondary index on time column is not enough. Queries often need particular keys only and you'll end up doing full table scans.
- yeukhon 10y agoI think udp vs tcp is debatable depending on the kind of data. If you are collecting continuous telemetry, losing a few may not be end of the world. Of course, YMMD so you should pick a database based on your objective. > Optimizing your network stack when your bottleneck is disk io doesn't buy you much. Agree but also the famous "depends." If you are collecting A LOT of data over the network, first makes sure your network card is capable of handling the incoming traffic. An anology would be a 100M vs 1G port. Also, if you are running on embedded system device like RPie or Arundio, I think disk will definitely be the first bottleneck before you even have a chance to respond to your network saturation.