4 ms·
Concider what a database does. It provides ACID properties and ability to query data. If all you need is writing data, the fastest you can do it write it direct
by hvidgaard 9y ago
Concider what a database does. It provides ACID properties and ability to query data. If all you need is writing data, the fastest you can do it write it directly to the disk, without the overhead a database comes with.
Using a loadbalancer in front of a farm of cheap logging machines, and aggreate the data you need for analysis to a suitable machine.
- LoSboccacc 9y agothey'd still need to query their stuff, I guess, so you'd need to trow in there somewhere something to aggregate logs and get the metrics they're tracking out of it - which can totally be done in streaming, without the need of going trough the logs every time, for most metrics.
- hvidgaard 9y agoWith that amount of data, streaming and only saving aggregated data is the only sane way. With 1.2TB/hour there is a limit to how much historical data that can be saved anyway, and we're talking about 30% utilization of a 10gbps network interface, so it's beyond using single machines for most usecases.
- tehlike 9y agoQuery? Most likely not. At least not in the traditional "lets on the fly create a dashboard" sense.
- LoSboccacc 9y agoquery as in 'how much I bill this guy for it's click' - doesn't have to be sql nor on the fly of course