5 ms·
Show HN: Infreqdb – S3 backed key/value database for infrequent read access
- TheDong 9y agoPlease don't post your own garbage with only a half-dozen commits and zero users. Wait for something to be used by more than yourself before you assume anyone else will want to see it. Anyways, the whole thing is utterly broken because S3 has no consistency guarantee on read-after-update, only read-after-write-untouched-key, so each get of a partition ID can be arbitrarily stale.
- nunez 9y agoThis is terrible feedback. Have you used it? Do you plan on using it? If the answer to both of those questions is "no," then you (a) can't call it garbage, and (b) haven't provided anything useful. The whole point of showing something on HN is to get more feedback and, in turn, more users.
- sctb 9y agoThe Show HN guidelines make it clear that early work is welcome, and that rude dismissals are not: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html.
- sajal83 9y agoI can't view the original comment since its flagged, but in hindsight I should have used the Show HN tag.
- bryanlarsen 9y agoYou can turn on showdead to see it, but here's the core of the comment phrased more charitably: How did you deal with stale reads caused by S3 lacking consistency guarantees on read-after-update?
- sajal83 9y agoThanks for the tip, just enabled it. In the current form, I don't directly deal with stale reads. But there is a CheckExpiry [1] method that iterates thru each cached partition, does a HEAD request to the corresponding S3 object and compares Last-Modified. If the cached object is older than the one in S3, the cache will be invalidated. Currently the user needs to invoke this by them-self. I also plan on having a user configurable TTL per partition, so user could use low TTL for objects they expect to change, i.e. One would think data for today might change soon, but data from 5 years ago wont. [1] https://godoc.org/github.com/turbobytes/infreqdb#DB.CheckExpiry https://godoc.org/github.com/turbobytes/infreqdb#DB.CheckExp...
- sctb 9y agoNo problem, we've updated the title for you.
- sajal83 9y agoThanks a lot
- spullara 9y agoBe careful using S3 for lots of small writes. The price is utterly dominated by the PUT calls which cost $0.01/1000.
- sajal83 9y agoYes. I should include this info in the README calcs. For my usecase I intend to do 4 - 10 PUTs per hour, since they are batched. 10 PUTS/h = 7300 PUTs/month = $0.073/month. The key here is infrequent
- spullara 9y agoBe most careful with retries. Saw a case where the bucket wasn't writable and the code just retried in a loop. You get charged for the API call, successful or not.
- saurik 9y agoOne day, many years ago, in a panic of "how the hell do I make this scale: I need it to work right now" I replaced an infrequently read data logging setup I had with S3... I ended up spending >$100k storing keys into that thing and momentarily was 2% of all objects stored in S3 (based on the stats they occasionally provided) before I eventually (two or three years later?) retired it; and to be extremely honest and to add insult to injury, my read access was so "infrequent" that I think I just never ended up reading it again after that day :/.
- ddxv 9y agoWhat do you think about this being able to work with Gzip compressed JSONs stored on S3? Cool project, thank you.
- sajal83 9y agoThanks. I think flat JSON files wont be efficient. My goal is to have the cache on disk, and each cached file would be big with lots of keys on it. In order to use JSON files, I would either have to keep the whole parsed data in memory, or parse the whole JSON each time I want to lookup a key. If the data fits in memory then sure JSON is more convenient.
- udkl 9y agoHow is this different than AWS Athena ? - https://aws.amazon.com/blogs/aws/amazon-athena-interactive-sql-queries-for-data-in-amazon-s3/ https://aws.amazon.com/blogs/aws/amazon-athena-interactive-s... Also why don't you dump the log data into a NoSQL like dynamoDB instead of S3 ?
- sajal83 9y agoAthena looks cool. Didn't know about it. It probably describes what I'm trying to do. > Also why don't you dump the log data into a NoSQL like dynamoDB instead of S3 ? Price.
- eropple 9y agoMaybe you should take another look at pricing? DynamoDB is like $0.002/GB/month more expensive than S3 for storage. Requests are more expensive, but you're also not reinventing every wheel and if they're as infrequent a set of requests as this seems designed for it's still only a few bucks a month.
- sajal83 9y agoMy infrequent reads and writes are huge, and possibly spikey. > Write Throughput: $0.0065 per hour for every 10 units of Write Capacity (enough capacity to do up to 36,000 writes per hour)* > A unit of Write Capacity enables you to perform one write per second for items of up to 1KB in size As I understand it, for $4.68/month I can only add 36 MB/hour, and thats assuming my objects are in exact multiple of 1KB.
- eropple 9y agoAhh, gotcha. That makes sense, this setup makes a lot more sense if you're dealing with very spikey reads.
- udkl 9y agoIf your data storage requirement is semi-constant at 1.5TB and is ephemeral, have you looked at just using your own replicated servers ? That would probably turn out to be cheaper than using S3 - https://github.com/minio/minio https://github.com/minio/minio
- marknadal 9y agoStuff like this is great! And you said you mentioned you are primarily doing it for timeseries data - then you can easily batch writes to handle high volume throughput too! We did this with S3 as the storage engine, 100M+ records for $10/day: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 And discord had a very nice article on this as well: https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7 https://blog.discordapp.com/how-discord-stores-billions-of-m... Great work, I think there is a lot of exciting stuff you can add to it!
- sajal83 9y agoThat's cool. Last week I tried googling for similar stuff but all I could find was people asking "How to run postgres on S3"...
- derefr 9y agoSo, a database that's has a file format of write-once content-addressed "shards" and is persisted/distributed by "offlining" those shards into object storage, and then "onlining" them back into a given DB node's MRU cache at query time; and is mostly read-only, but can update by downloading the relevant shards, modifying them locally, re-hashing the modified shards to get new names for them, and then uploading them again under those new names. Is this basically equivalent to Datomic, then? (Not that that's a bad thing. The world needs an open-source, non-JVM-targeted Datomic.)
- reubano 9y ago> The world needs an open-source, non-JVM-targeted Datomic.) Couchdb?
- vtuulos 9y agoWe use a similar approach with TrailDB. You can make S3 access totally seamless with user-space page fault handling, which is pretty cool: http://tech.adroll.com/blog/data/2016/11/29/traildb-mmap-s3.html http://tech.adroll.com/blog/data/2016/11/29/traildb-mmap-s3....
- sajal83 9y agoThis is pretty cool. Do you invalidate all pages if the file changes upstream on S3?
- vtuulos 9y agoIn our case all blobs in S3 are immutable, so no need to invalidate anything.
- rakoo 9y agoNext level, same idea but with SQLite: https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net