4 ms·
Good question. I thought it would be a no brainer to put it on s3 or similiar but thats already way to expensive at 2m/month without api requests. Backplace st
by AJSDfljff 2y ago
Good question. I thought it would be a no brainer to put it on s3 or similiar but thats already way to expensive at 2m/month without api requests.
Backplace storage pods are an initial investment of 5 Million, thats probably the best bet you could do and on that savings level, having 1-3 good people dedicated to this is probably still cheaper.
But you could / should start talking to the big cloud providers to see if they are flexible enough going lower on the price.
I have seen enough companies, including big ones, being absolut shitty in optimizing these types of things. At this level of data, i would optimize everyting including encoding, date format etc.
But i said it in my other comment: the interesting questions are not answered :D
- orf 2y agoThe compressed size is 20pb, so it’s about 500k per month in S3 fees
- francoismassot 2y agoIndeed. They benefit from a discount, but we don't know the discount figure. To further reduce the storage costs, you can use S3 Storage Classes or cheaper object storage like Alibaba for longer retention. Quickwit does not handle that, so you need to handle this yourself, though.
- AJSDfljff 2y agoI would probably build my own storage pods, keep a day or a week on cloud and move everything over every night.
- jcgrillo 2y agoLogs should compress better than that, though, right? 5:1 compression is only about half as good as you'd expect even naive gzipped json to achieve, and even that is an order of magnitude worse than the state of the art for logs[1]. What's the story there? [1] https://news.ycombinator.com/item?id=40938112 https://news.ycombinator.com/item?id=40938112