3 ms·

File storage in 2016? Why not just use S3? If file storage even a problem anymore?

Nux 11y ago
Yes, it's still a big problem.
ssmoot 11y ago
Because if you have apps that weren't written at a time that S3 pricing made any sense at all (it wasn't that long ago you could afford to buy a new file-server and JBOD every month for what it would've cost to store a few dozen terabytes on S3) migration is a huge burden and expense. Really looking forward to EFS to take a big bite out of that commitment since it has reasonable pricing for projects with smaller storage requirements. Just switching a portion (photos) of the project with the most files to S3 took on the order of 800 (mostly developer) hours. At some point you have to decide what you can do with the hand you're dealt and wether you want to distribute those tens of thousands of dollars to your employees, pursuing new business or your own pocket; or reinvest it into an existing project despite having no mandate to do so. It's not an easy or cut and dry a decision to make (even today) for legacy projects IME. If you're starting a new project. Yes. S3 is the most reliable service in the AWS stack (AFAIK) by a big margin. The pricing is very reasonable today unless you're in the petabyte range maybe, and you have no operations overhead. Agreed it's a no-brainer.
epistasis 11y ago
Anytime you use the word "just", you're sweeping a huge number of assumptions under the rug. For example, what if you have to run code that depends on a POSIX interface? That "just" becomes a massive rewrite project.
milesward 11y ago
Software that talks POSIX tends to dislike object storage. Disclaimer: I work on GCP.
boulos 11y ago
Not mentioned yet is that nearly all object stores (S3, GCS, etc.) are pretty awful for small, random reads and writes. For workloads like rendering, the renderer is often reading a small tile of a texture (an EXR or tiff) with about 32x32 pixels (so anywhere between 1 KiB and 16 KiB depending on precision and compression). Object stores on the other hand are tuned for big files and so often work in 1 to 4 MiB chunks at best. Disclaimer: I work on Compute Engine.

Nux 11y ago

Yes, it's still a big problem.

Because if you have apps that weren't written at a time that S3 pricing made any sense at all (it wasn't that long ago you could afford to buy a new file-server and JBOD every month for what it would've cost to store a few dozen terabytes on S3) migration is a huge burden and expense. Really looking forward to EFS to take a big bite out of that commitment since it has reasonable pricing for projects with smaller storage requirements. Just switching a portion (photos) of the project with the most files to S3 took on the order of 800 (mostly developer) hours. At some point you have to decide what you can do with the hand you're dealt and wether you want to distribute those tens of thousands of dollars to your employees, pursuing new business or your own pocket; or reinvest it into an existing project despite having no mandate to do so. It's not an easy or cut and dry a decision to make (even today) for legacy projects IME. If you're starting a new project. Yes. S3 is the most reliable service in the AWS stack (AFAIK) by a big margin. The pricing is very reasonable today unless you're in the petabyte range maybe, and you have no operations overhead. Agreed it's a no-brainer.

epistasis 11y ago

Anytime you use the word "just", you're sweeping a huge number of assumptions under the rug. For example, what if you have to run code that depends on a POSIX interface? That "just" becomes a massive rewrite project.

milesward 11y ago

Software that talks POSIX tends to dislike object storage. Disclaimer: I work on GCP.

boulos 11y ago

Not mentioned yet is that nearly all object stores (S3, GCS, etc.) are pretty awful for small, random reads and writes. For workloads like rendering, the renderer is often reading a small tile of a texture (an EXR or tiff) with about 32x32 pixels (so anywhere between 1 KiB and 16 KiB depending on precision and compression). Object stores on the other hand are tuned for big files and so often work in 1 to 4 MiB chunks at best. Disclaimer: I work on Compute Engine.