5 ms·

While S3 storage costs are fairly reasonable, request costs are unrelated to file size, which makes it an absolutely terrible choice for filesystem-like activit

by meritt 9y ago

While S3 storage costs are fairly reasonable, request costs are unrelated to file size, which makes it an absolutely terrible choice for filesystem-like activities. I say this from experience: S3 PUT requests make up a very significant portion of our AWS bill as we write a massive number of very small files to S3. Does anyone know of a middle-layer solution that automatically concatenates files into larger chunks (similar to HDFS) prior to persisting to S3 and for retrieval utilizes HTTP Range requests to fetch the individual file? I'm going to build it if it doesn't exist. 100MB chunks would reduce our S3 PUT costs by ~1000x.

deleted 9y ago
[deleted]
toomuchtodo 9y ago
Couldn’t find anything from a google search, but seems you could buffer files for writes, tar them, PUT the tar, and then use the tar index to perform range reads against the tar file. Edit: “dar”, “zip”, or “dump/restore” might be able to do this: https://serverfault.com/questions/59795/is-there-a-smarter-tar-or-cpio-out-there-for-efficiently-retrieving-a-file-store https://serverfault.com/questions/59795/is-there-a-smarter-t... (will create an archive with an index file for random access, vs tar streaming requiring reading the entire file) If you come up with another solution, I’d love to hear it!
shortj 9y ago
Entirely untested and just spitballing... use Firehose to concat a bunch of small files and store based on size or time passage as a larger file to s3. Then use Lambda transform in firehose to index the byte ranges for each record (file)? (Biggest q in the process would be here... you might have to index after it hits s3) S3 also allows the Range request header out of the box as well. Interesting problem, and I can totally related with s3 put costs be astronomical.
lars_francke 9y ago
HDFS does no such thing. If you store a million small files it'll store a million small files on disk as well.
- dvfjsdhgfv 9y ago
  I think the parent might mean HDF5.
  - deleted 9y ago
    [deleted]
idunno246 9y ago
I POCd dynamodb(+api gateway+lambda for reads) for a project that was tons of tiny files dominated by puts. was a pretty trivial switch(writes was one extra line of code, reads was like 10 total lines), estimated cost savings was large(though we didnt fully calculate it). it didnt exist at the time, but i think lambda@edge might work really well for reads.
sigil 9y ago
cperciva’s tarsnap server implements a log structured filesystem on S3. [1] This enables bundling of multiple writes into large objects, with the explicit goal of reducing per-request costs. It’s not open source, but maybe some of the high level ideas can be cribbed? [1] http://www.daemonology.net/blog/2008-12-14-how-tarsnap-uses-aws.html http://www.daemonology.net/blog/2008-12-14-how-tarsnap-uses-...
RantyDave 9y ago
Dunno if it would work in your case but ... I've had fun mounting a ZFS file system using an entire EFS volume as the "physical" back end. You have to give up on writes actually being on disk when you think they are, but the payback is that's it's really kinda fast.
daviesliu 9y ago
My friend also got a real big bill for S3 PUT requests, they are just too expensive. When we built JuiceFS [1], a POSIX filesystem on top of S3, there is feature in the roadmap to combine multiple small files together before uploading into S3 (each file is a slice of chunks in S3). We will let you know when it's ready. [1] https://juicefs.io/ https://juicefs.io/
- idunno246 9y ago
  heh, when s3 first came out they didnt have per request charges. then someone built a fuse fs on top of s3, and adding those charges was largely to convince people not to use s3 as a filesystem
dvfjsdhgfv 9y ago
Do yourself a favor and use a Hetzner Storage Box: https://www.hetzner.com/storage-box https://www.hetzner.com/storage-box Even if you don't include any extra costs, just keeping 10 TB on Hetzner will cost you 40€ max, whereas with S3 it will cost you at least $230.
- blfr 9y ago
  A single Hetzner box is not really comparable to S3 in terms of durability and availability.
  - dvfjsdhgfv 9y ago
    My experience is that they go down every 2-3 years, usually the problem is with the hard rive, occasionally with some other component. Under normal circumstances it takes them up to one hour to replace the hard drive, and then the array gets rebuilt for a few hours with the system being online. That means something like 2 hours downtime every 2-3 years, that's quite acceptable for my usage scenarios. In terms of money spent during that time, the difference is enormous.
- meritt 9y ago
  Like most businesses on a cloud provider, we're fairly locked-in to AWS without major changes. Going offsite would replace S3 PUT costs with significant bandwidth costs.
nolite 9y ago
Depends on how you’re writing. You could send everything to a kinesis or firehouse stream and use that to batch inputs, or you could write to an Elastic File System and batch there with a scheduled function

by meritt 9y ago

deleted 9y ago

[deleted]

toomuchtodo 9y ago

Couldn’t find anything from a google search, but seems you could buffer files for writes, tar them, PUT the tar, and then use the tar index to perform range reads against the tar file. Edit: “dar”, “zip”, or “dump/restore” might be able to do this: https://serverfault.com/questions/59795/is-there-a-smarter-tar-or-cpio-out-there-for-efficiently-retrieving-a-file-store https://serverfault.com/questions/59795/is-there-a-smarter-t... (will create an archive with an index file for random access, vs tar streaming requiring reading the entire file) If you come up with another solution, I’d love to hear it!

shortj 9y ago

Entirely untested and just spitballing... use Firehose to concat a bunch of small files and store based on size or time passage as a larger file to s3. Then use Lambda transform in firehose to index the byte ranges for each record (file)? (Biggest q in the process would be here... you might have to index after it hits s3) S3 also allows the Range request header out of the box as well. Interesting problem, and I can totally related with s3 put costs be astronomical.

lars_francke 9y ago

HDFS does no such thing. If you store a million small files it'll store a million small files on disk as well.

dvfjsdhgfv 9y ago
I think the parent might mean HDF5.
- deleted 9y ago
  [deleted]

idunno246 9y ago

I POCd dynamodb(+api gateway+lambda for reads) for a project that was tons of tiny files dominated by puts. was a pretty trivial switch(writes was one extra line of code, reads was like 10 total lines), estimated cost savings was large(though we didnt fully calculate it). it didnt exist at the time, but i think lambda@edge might work really well for reads.

sigil 9y ago

cperciva’s tarsnap server implements a log structured filesystem on S3. [1] This enables bundling of multiple writes into large objects, with the explicit goal of reducing per-request costs. It’s not open source, but maybe some of the high level ideas can be cribbed? [1] http://www.daemonology.net/blog/2008-12-14-how-tarsnap-uses-aws.html http://www.daemonology.net/blog/2008-12-14-how-tarsnap-uses-...

RantyDave 9y ago

Dunno if it would work in your case but ... I've had fun mounting a ZFS file system using an entire EFS volume as the "physical" back end. You have to give up on writes actually being on disk when you think they are, but the payback is that's it's really kinda fast.

daviesliu 9y ago

My friend also got a real big bill for S3 PUT requests, they are just too expensive. When we built JuiceFS [1], a POSIX filesystem on top of S3, there is feature in the roadmap to combine multiple small files together before uploading into S3 (each file is a slice of chunks in S3). We will let you know when it's ready. [1] https://juicefs.io/ https://juicefs.io/

idunno246 9y ago
heh, when s3 first came out they didnt have per request charges. then someone built a fuse fs on top of s3, and adding those charges was largely to convince people not to use s3 as a filesystem

dvfjsdhgfv 9y ago

Do yourself a favor and use a Hetzner Storage Box: https://www.hetzner.com/storage-box https://www.hetzner.com/storage-box Even if you don't include any extra costs, just keeping 10 TB on Hetzner will cost you 40€ max, whereas with S3 it will cost you at least $230.

blfr 9y ago
A single Hetzner box is not really comparable to S3 in terms of durability and availability.
- dvfjsdhgfv 9y ago
  My experience is that they go down every 2-3 years, usually the problem is with the hard rive, occasionally with some other component. Under normal circumstances it takes them up to one hour to replace the hard drive, and then the array gets rebuilt for a few hours with the system being online. That means something like 2 hours downtime every 2-3 years, that's quite acceptable for my usage scenarios. In terms of money spent during that time, the difference is enormous.
meritt 9y ago
Like most businesses on a cloud provider, we're fairly locked-in to AWS without major changes. Going offsite would replace S3 PUT costs with significant bandwidth costs.

nolite 9y ago

Depends on how you’re writing. You could send everything to a kinesis or firehouse stream and use that to batch inputs, or you could write to an Elastic File System and batch there with a scheduled function