7 ms·
Why Tarsnap won't use DynamoDB
- encoderer 15y agoI only skimmed this, and I'll go back and read it later, but I think the issue he's mentioning is price? It's true that on Dynamo, large item sizes get very expensive, very quick. That stems from your data being stored on SSD--and with a replication factor of 3 (iirc). Cloud or no cloud, Amazon or not, SSD storage is still a lil pricey.
- philjohn 15y agoLots and lots of small items also costs though, because there's a 100byte per k/v pair overhead. So if you're storing 2GB of 100 byte k/v pairs, you have to pay for 4GB of storage. If you store lots and lots, you probably also have either high read, high write, or high read/write rates, in which case you'll need to pay more per hour as well.
- TeamAqua 15y agoGood article. In many ways, I think the author is echoing the same frustrations as with Google App Engine's Datastore ( http://news.ycombinator.com/item?id=3431132 http://news.ycombinator.com/item?id=3431132 App Engine charges $6,500 to update a ListProperty on 14.1 million entities ). Both Google Datastore and the new AWS DynamoDB are great, fast services, but they are a bit too expensive at the medium/high end. Frankly, I wish App Engine/AWS/Heroku/etc would introduce a database that traded speed for cheaper costs. I'd be fine with certain writes being delayed if it meant lower costs.
- tyw 15y agoIf Amazon had an option to use HDD instead of SSD storage for a certain DynamoDB table and adjusted the pricing accordingly, would that pretty much take care of users wanting basically what DyanamoDB offers except for cheaper with slightly worse performance?
- ceejayoz 15y agoTL;DR version: "The custom NoSQL store (that lacks one of DynamoDB's major selling points) I wrote for my unusual use case works better for my unusual use case than DynamoDB."
- cperciva 15y agoYep. Or as I put it, DynamoDB is a great hammer, but sometimes you need a screwdriver.
- mkup 15y agoMy TL;DR version is "DynamoDB is much more expensive than custom NoSQL store at thousand-writes-per-second scale"
- encoderer 15y agoI don't mean to hammer this comment page with this point, but just to clarify this since the product is so new: It really has very little to do with writes per second. High volume writes are Dynamo's bread and butter. His cost issues come from the large item size.
- cperciva 15y agoNo, I have very small items. The cost issues come from the large number of items.
- encoderer 15y agoI thought I read 33kb somewhere in there. If so, by dynamodb standards, that's not a small item. If I mis-attributed that 33kb number, then I kindly retract what I said :D
- cperciva 15y agoI have 33kB blocks which are stored on S3 (after aggregating them into objects of up to 8 MB). I have key-value pairs which are 24 bytes and 53 bytes long.
- drewcrawford 15y ago> $44 per month. That's 14.6% of Tarsnap's gross revenues; Hi Cpervia, Please, please, please, charge more for Tarsnap. Unbeknownst to you, I used you as an example in a blog post a year ago: > If cpervia tripled the cost of tarsnap tomorrow I would not bat an eyelash.
- cperciva 15y agoSince your quote missed some important context: That's $44/month per TB of data stored on Tarsnap. Tarsnap's gross revenues are considerably more than $300/month. :-) > Please, please, please, charge more for Tarsnap. Funny thing is, I get lots of people (especially large users) saying exactly the opposite. The fact that there are lots of people who think I'm very wrong in both directions suggests to me that Tarsnap's pricing is about right.
- drx 15y agoNot sure if it's a question you'd answer publicly, but have you considered having pricing plans to capture both pies? I'm genuinely curious, not suggesting you should do it.
- cperciva 15y agoYou mean to offer a lower price, but say "if you want to pay more, feel free"? I have a feeling that wouldn't work very well.
- underwater 15y agoHe means offering tiered pricing.
- qeorge 15y agoThe reverse. Offer a high price, but make it easy to get a discount (i.e., by contacting sales and asking for one, or Googling for a coupon). Or offer an Enterprise plan and charge a ton for it. You might be surprised. Some customers may actually prefer it if you charged more, and might not consider Tarsnap if you don't. (Enterprise software is "supposed to be" expensive). At the very least its worth testing.
- tom_b 15y agoColin, since you also wrote the Tarsnap back-end data engine yourself, how would you weigh a cost vs loss-of-control of that functionality in Tarsnap? From my perspective, the flexibility and control you have over kivaloo generates some value to you as the developer of Tarsnap. At what cost point/function-level would you actually make a switch to a SaaS back-end? My interest in this question goes beyond this specific case to the consideration of when service/library use in a new SaaS production is contraindicated.
- cperciva 15y agoIf Amazon had a service which gave me the cost and performance I expect to get from kivaloo I'd start using it without any hesitation. I assign a negative cost to loss-of-control since the testing provided by their large user base and their experience with scalability far outweighs the advantages of greater control.
- res0nat0r 15y agoNot to denigrate Colin's work, nor Tarsnap at all, as I love these CLI based tools, but is there anything better about Tarsnap than just using Duplicity as I do now? All I have to do is a ' duplicity $DIR s3+http://bucket http://bucket ' and I will get a gpg encrypted full/diff of my data, which looks to be a bit cheaper and GPG encrypted stored on S3. Plus if I really want to save some cash I could mark every object as Reduced Redundancy to save even more money. Thoughts?