4 ms·
Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), whe
by akavel 8y ago
Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...
- cedricvg 8y agoThese numbers really depend on the compressibility of the content, compression scheme and the type of batching used. The metadata overhead is fairly minimal. LogDevice allows you to configure this on either the client, sequencer or rocksdb level.
- akavel 8y agoIs some form of compression enabled by default, without having to tweak options?
- sh00s 8y agoNo, compression is disabled by default. You can enable compression either on the storage layer, or by enabling batching and compression on the sequencer, or by using the buffered write API with compression on the client.
- SirMonkey 8y agohave you had a look at https://github.com/oklog/oklog https://github.com/oklog/oklog https://www.youtube.com/watch?v=gWWK2eyZ-sc https://www.youtube.com/watch?v=gWWK2eyZ-sc I think it's fairly simple and might be enough. Can't comment on storage requirements thou.
- are595 8y agoDang, I was looking at oklog earlier, but looks like it is archived now...