4 ms·

This does sound nice, but the article barely mentions a pretty major limitation from the release announcement[0]: > it doesn’t support writes in this first rel

This does sound nice, but the article barely mentions a pretty major limitation from the release announcement[0]: > it doesn’t support writes in this first release, and in the future will only support sequential writes to new objects. [0]: https://aws.amazon.com/blogs/storage/the-inside-story-on-mountpoint-for-amazon-s3-a-high-performance-open-source-file-client/ https://aws.amazon.com/blogs/storage/the-inside-story-on-mou...

tyingq 4y ago
>only support sequential writes to new objects. "Doesn't supports writes in the first release" is a letdown, but the latter part above seems expected. I suppose you could abstract what would have to happen to fake arbitrary seek() and overwriting portions of a file, appending, etc, but it would encourage things that wouldn't work well.
- sebazzz 4y ago
  Partial writes would need to buffered and flush()ed. That is not unlike other file systems, until you fully know it is persisted - or am I missing something?
  - tyingq 4y ago
    That you can't update an S3 object. There is no equivalent to open() on an existing file, seek() to some position, and modify a few bytes.
    - rbanffy 4y ago
      That S3 storage is almost immediately consistent on a planetary scale is pretty impressive already. This would make it a lot more complicated to do that. OTOH, you can always engineer around such limitations by storing smaller objects you can completely ingest and store on a single operation.
  - stingraycharles 4y ago
    It’s not supported by S3 to partially update an object; you would have to completely replace the entire object. While they could do some magic under the hood to make it seem like it works, it’s better to just reject these operations and let the application developer decide what’s best.
    - bilekas 4y ago
      Maybe I'm missing some context here, but can't you generate X number of presigned URL's with your chunk size and if you provide the chunk ordering it's automatically done on S3 side, but I though you could manually implement the chunk order assignment also.
      - ldoughty 4y ago
        I would suspect the team making this tool is not exactly the same team maintaining S3... So adding "server side operations" would require work from the other team to be added.. which certainly could be done if they decided it was needed. As a first MVP release this project seems neat for read use cases.. and feedback from the community can help ensure read operations are solid while they are also working on the write side -- which they admitted was lacking in this initial release, but want to improve.
- rbanffy 4y ago
  For that use case, they have EFS. If you are reading from one of more S3 blobs and gradually building another blob, you can keep that blob on EFS and, when finished, dump it back to S3.
varunjain99 4y ago
My understanding it goofys and s3fs support these types of operations? Still, a win for reliability / performance of whatever aws will be supporting
jjoonathan 4y ago
It wouldn't be an AWS announcement without a whopper of a caveat, it's like the Amazon equivalent of Google canceling things.
- quickthrower2 4y ago
  To be fair it sounds like this isn’t for that use case.
- kjuulh 4y ago
  AWS really does have its own quirks as a company. - The tools feel fragmented, not aligned and often don't compose well - Always sharp corners (Billing, ops, you name it) - Products in various state of usability / maturity, sometimes a product is launched by barely usable. I've never worked for Amazon, but if what is said about amazon is true, then the culture is reflected in their products, teams siloed owning their own services, then it does make sense that their services have those quirks.
- sudhirj 4y ago
  That comparison seems unfair... AWS has a habit of launching early and iterating, and they almost never shut stuff down. In this case it's a filesystem implementation of a HTTP API, so the capabilities reflect the main API. Deciding to build and launch reads before writes isn't really a bad decision, it helps those who have the read use case try this out and give feedback immediately.
  - ochoseis 4y ago
    > In this case it's a filesystem implementation of a HTTP API When you put it like that, it makes me wonder why they don't support WebDAV.
    - sudhirj 4y ago
      I think there’s a bunch of gateways available already. The example use case for this seems to be mounting a 10 petabyte “hard drive” with CSVs to a reporting machine.
908B64B197 4y ago
> it doesn’t support writes in this first release, and in the future will only support sequential writes to new objects. To me it just sounds like Azure Files with less features. You can just mount a share on any machine supporting NFS or SMB or just use the REST interface. [0] [0] https://learn.microsoft.com/en-us/rest/api/storageservices/file-service-rest-api https://learn.microsoft.com/en-us/rest/api/storageservices/f...
hazbot 4y ago
Fair point, defs should be mentioned by any serious article. I have use cases where forbidding writes is a feature, and if it means more stability and performance it’s a trade-off that’s worth it to me.

tyingq 4y ago

>only support sequential writes to new objects. "Doesn't supports writes in the first release" is a letdown, but the latter part above seems expected. I suppose you could abstract what would have to happen to fake arbitrary seek() and overwriting portions of a file, appending, etc, but it would encourage things that wouldn't work well.

sebazzz 4y ago
Partial writes would need to buffered and flush()ed. That is not unlike other file systems, until you fully know it is persisted - or am I missing something?
- tyingq 4y ago
  That you can't update an S3 object. There is no equivalent to open() on an existing file, seek() to some position, and modify a few bytes.
  - rbanffy 4y ago
    That S3 storage is almost immediately consistent on a planetary scale is pretty impressive already. This would make it a lot more complicated to do that. OTOH, you can always engineer around such limitations by storing smaller objects you can completely ingest and store on a single operation.
- stingraycharles 4y ago
  It’s not supported by S3 to partially update an object; you would have to completely replace the entire object. While they could do some magic under the hood to make it seem like it works, it’s better to just reject these operations and let the application developer decide what’s best.
  - bilekas 4y ago
    Maybe I'm missing some context here, but can't you generate X number of presigned URL's with your chunk size and if you provide the chunk ordering it's automatically done on S3 side, but I though you could manually implement the chunk order assignment also.
    - ldoughty 4y ago
      I would suspect the team making this tool is not exactly the same team maintaining S3... So adding "server side operations" would require work from the other team to be added.. which certainly could be done if they decided it was needed. As a first MVP release this project seems neat for read use cases.. and feedback from the community can help ensure read operations are solid while they are also working on the write side -- which they admitted was lacking in this initial release, but want to improve.
rbanffy 4y ago
For that use case, they have EFS. If you are reading from one of more S3 blobs and gradually building another blob, you can keep that blob on EFS and, when finished, dump it back to S3.

varunjain99 4y ago

My understanding it goofys and s3fs support these types of operations? Still, a win for reliability / performance of whatever aws will be supporting

jjoonathan 4y ago

It wouldn't be an AWS announcement without a whopper of a caveat, it's like the Amazon equivalent of Google canceling things.

quickthrower2 4y ago
To be fair it sounds like this isn’t for that use case.
kjuulh 4y ago
AWS really does have its own quirks as a company. - The tools feel fragmented, not aligned and often don't compose well - Always sharp corners (Billing, ops, you name it) - Products in various state of usability / maturity, sometimes a product is launched by barely usable. I've never worked for Amazon, but if what is said about amazon is true, then the culture is reflected in their products, teams siloed owning their own services, then it does make sense that their services have those quirks.
sudhirj 4y ago
That comparison seems unfair... AWS has a habit of launching early and iterating, and they almost never shut stuff down. In this case it's a filesystem implementation of a HTTP API, so the capabilities reflect the main API. Deciding to build and launch reads before writes isn't really a bad decision, it helps those who have the read use case try this out and give feedback immediately.
- ochoseis 4y ago
  > In this case it's a filesystem implementation of a HTTP API When you put it like that, it makes me wonder why they don't support WebDAV.
  - sudhirj 4y ago
    I think there’s a bunch of gateways available already. The example use case for this seems to be mounting a 10 petabyte “hard drive” with CSVs to a reporting machine.

908B64B197 4y ago

> it doesn’t support writes in this first release, and in the future will only support sequential writes to new objects. To me it just sounds like Azure Files with less features. You can just mount a share on any machine supporting NFS or SMB or just use the REST interface. [0] [0] https://learn.microsoft.com/en-us/rest/api/storageservices/file-service-rest-api https://learn.microsoft.com/en-us/rest/api/storageservices/f...

hazbot 4y ago

Fair point, defs should be mentioned by any serious article. I have use cases where forbidding writes is a feature, and if it means more stability and performance it’s a trade-off that’s worth it to me.