8 ms·

Mountpoint – file client for S3 written in Rust, from AWS

257 points by ranman 4y ago

ranman 4y ago
A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.
toomuchtodo 4y ago
> This is an alpha release and not yet ready for production use. We're especially interested in early feedback on features, performance, and compatibility. Please send feedback by opening a GitHub issue. See Current status for more limitations.
deleted 4y ago
[deleted]
metadat 4y ago
How does this compare to rclone, performance wise? https://rclone.org https://rclone.org
- e12e 4y ago
  And s4cmd... https://github.com/bloomreach/s4cmd https://github.com/bloomreach/s4cmd
dimatura 4y ago
It would be interesting to see how this compares to other solutions in this space, such as s3fs (the FUSE driver, not the python package), goofys, and the rclone mount feature, among others. This certainly has less features (notably, mounts are read-only!).
klodolph 4y ago
I want a better client for Google Cloud Storage, too, while we’re at it. The Python gcloud / gsutil stuff is mediocre on the best of days.
- thangngoc89 4y ago
  In theory, you can just use this library since GC Storage supports S3 protocol. But in practice, I’m not sure
  - e12e 4y ago
    Or https://clone.org https://clone.org ?
    - kwk1 4y ago
      Presumably you forgot the 'r': https://rclone.org/ https://rclone.org/
      - e12e 4y ago
        Autocorrect strikes again! Yes, I meant rclone - thank you.
- seabrookmx 4y ago
  `gsutil ...` was pretty bad (it is python like you say, and based on a very outdated fork of boto2). I've had really good luck with `gcloud storage ...` though, which takes essentially the same CLI args. It's much faster and IIRC written in golang.
- dotancohen 4y ago
  I'm just about to start using it. I'd love to know what issues you've encountered. Thanks!
- anshumankmr 4y ago
  https://github.com/GoogleCloudPlatform/gcsfuse https://github.com/GoogleCloudPlatform/gcsfuse?
  - klodolph 4y ago
    FUSE makes everything worse, not better. The Unix file API is awful in general, and a terrible mismatch for key-values storage systems.
bravura 4y ago
How does this compare to goofys?
favourable 4y ago
Something similar that I've been using for a while now for an S3 filesystem: Cyberduck[0] [0] https://cyberduck.io/s3/ https://cyberduck.io/s3/
- mNovak 4y ago
  In a similar vein, I've been using ExpanDrive [0] for a while. Though admittedly it's only suitable for infrequent access / long term storage type use. [0] https://www.expandrive.com/desktop/ https://www.expandrive.com/desktop/
- lightlyused 4y ago
  I just wish cyberduck would get a more standard UI. It is so win95 vb. Otherwise works great!
supriyo-biswas 4y ago
After teaching customers for years that S3 shouldn't be mounted as a filesystem because of its whole object-or-nothing semantics, and even offering a paid solution named "storage gateway" to prevent issues between FS and S3 semantics, it's rather interesting they'd release a product like this. Amazon should really just fix the underlying issue of semantics by providing a PatchObjectPart API call that overwrites a particular multipart upload chunk with a new chunk uploaded from the client. CopyObjectPart+CompleteMultipartUpload still requires the client to issue CopyObjectPart calls for the entire object.
- nostrebored 4y ago
  Distributed patching becomes hell. You need transactional semantics and files are not laid out well to help you define invariants that should reject the transaction.
  - supriyo-biswas 4y ago
    There is no reason why the descriptor of objects can’t be updated with a new value that has all of the old chunks and a new one, since S3 doesn’t do deduplication anyway the other chunks may be resized internally with an asynchronous process that gets rid of the excess data corresponding to the now overridden chunk.
- wmf 4y ago
  Eventually AWS always gives customers what they want even if it's a "bad idea".
  - mlhpdx 4y ago
    Indeed.
  - znpy 4y ago
    Bad ideas are very relative. Some bad ideas work extremely well if they fit your use case, you understand very well the tradeoffs and you’re building safeguards (disaster recovery). Some other companies try to convince (force?) you into a workflow or into a specific solution. Aws just gives you the tools and some guidance on how to use them best.
tolmasky 4y ago
Couldn't tell from the README, does this do any sort of cache management or LRU type thing? In other words, does it fetch the underlying S3 object in real time, and then eventually eject them from memory and/or the backing FS when they haven't been used for a while?
- renewiltord 4y ago
  `catfs` is a FUSE FS that can do this for you. You'll need some changes to make it work well. I'll have a friend upstream them soon, but they're easy to make yourself. Could replace `goofys` with this and then stick `catfs` in front.
sfritz 4y ago
There's a similar project under awslabs for using S3 as a FileSystem within the Java JVM: https://github.com/awslabs/aws-java-nio-spi-for-s3 https://github.com/awslabs/aws-java-nio-spi-for-s3
ary 4y ago
This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new files, and must be done sequentially. Modifying existing files will not be supported. Truncation will not be supported. The sequential requirement for writes is the part that I've been mulling over whether or not it's actually required in S3. Last year I discovered that S3 can do transactional I/O via multipart upload[2] operations combined with the CopyObject[3] operation. This should, in theory, allow for out of order writes, existing partial object re-use, and file appends. [1] https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMAN... [2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuove... [3] https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje...
- deleted 4y ago
  [deleted]
- FullyFunctional 4y ago
  Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.
  - Scarbutt 4y ago
    S3 is slow but at the same time low cost, if you want fast AWS has other alternatives but pricier.
    - mlhpdx 4y ago
      This is misleading. S3 is also incredibly fast. The former when you’re sequentially writing (or reading) objects and the latter when concurrently writing (or reading) vast numbers of them.
IceWreck 4y ago
They should benchmark it against rclone
BrianHenryIE 4y ago
JungleDisk was backup software I used ~2009 that allowed mounting S3. They were bought by Rackspace and the product wasn't updated. Seems to be called/part of Cyberfortress now. Later I used Panic's Transmit Disk but they removed the feature. Recently I'd been looking at s3fs-fuse to use with gocryptfs but haven't actually installed it yet! https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse https://github.com/rfjakob/gocryptfs https://github.com/rfjakob/gocryptfs
- CharlesW 4y ago
  > Later I used Panic's Transmit Disk but they removed the feature. BTW, Panic seemingly intends to re-build Transmit Disk. Hopefully it'll be part of Transmit 6: https://help.panic.com/transmit/transmit5/transmit-disk/#technical-details-and-the-future-of-transmit-disk https://help.panic.com/transmit/transmit5/transmit-disk/#tec... A supported macOS option appears to be Mountain Duck: https://mountainduck.io/ https://mountainduck.io/
  - drcongo 4y ago
    ForkLift also lets you mount S3 as a drive. https://binarynights.com https://binarynights.com
- legorobot 4y ago
  We've used the s3fs-fuse library for a while at work for SFTP/FTP server alternatives (AWS wants you to pay $150+/server/month last I checked!) and it's worked like a dream. We scripted the setup of new users via a simple bash script and the S3 CloudWatch events for file uploads is a dream. Its been pretty seamless and hasn't caused many headaches. We've had to perform occasional maintenance but its operated for years with no major issues. 99% are solved with a server restart + a startup script to auto-re-mount s3fs-fuse in all the appropriate places. Give them a try, I recommend it!
mlindner 4y ago
There's some really confusing use of unsafe going on. For example I'm not sure what they're doing here: https://github.com/awslabs/mountpoint-s3/blob/main/mountpoint-s3-crt/src/io/stream.rs#L111 https://github.com/awslabs/mountpoint-s3/blob/main/mountpoin...
Dinux 4y ago
This is exactly what I need. The current python scrips are good enough but a rust utility would be preferable
hkgjjgjfjfjfjf 4y ago
[dead]
wanderingmind 4y ago
For anyone looking to mount S3 as file system, I will suggest giving rclone a shot. It can mount, copy and do all file operations not just on s3 but on a wide range of cloud providers, you can also declare a remote as encrypted so it does client side encryption
hawski 4y ago
I would suggest changing the title to "Mountpoint-S3 - ..." as that's the project name to avoid confusion with mountpoint(1): https://man7.org/linux/man-pages/man1/mountpoint.1.html https://man7.org/linux/man-pages/man1/mountpoint.1.html
arretevad 4y ago
I find all the “written in Rust” qualifier in all these post title to just be a distraction. It doesn’t feel like it’s telling me anything.
- grogenaut 4y ago
  Amazon is starting to invest in rust internally, strategically, since joining some of the leadership (https://aws.amazon.com/blogs/opensource/why-aws-loves-rust-and-how-wed-like-to-help/ https://aws.amazon.com/blogs/opensource/why-aws-loves-rust-a...). It's considered a good replacement for C++, and like go, is really good for releasing tools. Tools like the AWS CLI... that work great when you can plop a single exe down as your install story, as opposed to say a python install and app (aws cli). But it's also still new. Releasing a tool like this is likely a big deal in the area, and they're likely quite prod of it due to the effort of things like getting legal approval, marketing, etc, let alone the cool nerd factor of a filesystem, who doesn't want to show off by having written a filesystem or hell a fuse plugin... file system over dns anyone?
zsy056 4y ago
If you are looking for something supports atomic rename, you can checkout Blobfuse2[0] + ADLS Gen2[1] Disclaimer: work for MSFT [0] https://github.com/Azure/azure-storage-fuse https://github.com/Azure/azure-storage-fuse [1] https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace https://learn.microsoft.com/en-us/azure/storage/blobs/data-l...
baggiponte 4y ago
I don’t understand whether this is just a higher level abstraction of boto’s s3 client (à la s3fs)
- biorach 4y ago
  It looks to be a completely different codebase from boto/s3fs. Not having used s3fs I'm going to guess that s3fs is limited due to the limits of the underlying language - Python - namely poor performance overall and poor multi-thread story. I'd imagine s3fs is useful for stuff like backing up personal projects, quickly sharing files between developers etc. For operating at any kind of scale - in terms of concurrent requests, number or size of files etc - I'd guess that Mountpoint would be the only viable solution.
rdxm 4y ago
[dead]

257 points by ranman 4y ago

A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.

> This is an alpha release and not yet ready for production use. We're especially interested in early feedback on features, performance, and compatibility. Please send feedback by opening a GitHub issue. See Current status for more limitations.

deleted 4y ago

[deleted]

metadat 4y ago

How does this compare to rclone, performance wise? https://rclone.org https://rclone.org

e12e 4y ago
And s4cmd... https://github.com/bloomreach/s4cmd https://github.com/bloomreach/s4cmd

dimatura 4y ago

It would be interesting to see how this compares to other solutions in this space, such as s3fs (the FUSE driver, not the python package), goofys, and the rclone mount feature, among others. This certainly has less features (notably, mounts are read-only!).

klodolph 4y ago

I want a better client for Google Cloud Storage, too, while we’re at it. The Python gcloud / gsutil stuff is mediocre on the best of days.

thangngoc89 4y ago
In theory, you can just use this library since GC Storage supports S3 protocol. But in practice, I’m not sure
- e12e 4y ago
  Or https://clone.org https://clone.org ?
  - kwk1 4y ago
    Presumably you forgot the 'r': https://rclone.org/ https://rclone.org/
    - e12e 4y ago
      Autocorrect strikes again! Yes, I meant rclone - thank you.
seabrookmx 4y ago
`gsutil ...` was pretty bad (it is python like you say, and based on a very outdated fork of boto2). I've had really good luck with `gcloud storage ...` though, which takes essentially the same CLI args. It's much faster and IIRC written in golang.
dotancohen 4y ago
I'm just about to start using it. I'd love to know what issues you've encountered. Thanks!
anshumankmr 4y ago
https://github.com/GoogleCloudPlatform/gcsfuse https://github.com/GoogleCloudPlatform/gcsfuse?
- klodolph 4y ago
  FUSE makes everything worse, not better. The Unix file API is awful in general, and a terrible mismatch for key-values storage systems.

bravura 4y ago

How does this compare to goofys?

favourable 4y ago

Something similar that I've been using for a while now for an S3 filesystem: Cyberduck[0] [0] https://cyberduck.io/s3/ https://cyberduck.io/s3/

mNovak 4y ago
In a similar vein, I've been using ExpanDrive [0] for a while. Though admittedly it's only suitable for infrequent access / long term storage type use. [0] https://www.expandrive.com/desktop/ https://www.expandrive.com/desktop/
lightlyused 4y ago
I just wish cyberduck would get a more standard UI. It is so win95 vb. Otherwise works great!

supriyo-biswas 4y ago

After teaching customers for years that S3 shouldn't be mounted as a filesystem because of its whole object-or-nothing semantics, and even offering a paid solution named "storage gateway" to prevent issues between FS and S3 semantics, it's rather interesting they'd release a product like this. Amazon should really just fix the underlying issue of semantics by providing a PatchObjectPart API call that overwrites a particular multipart upload chunk with a new chunk uploaded from the client. CopyObjectPart+CompleteMultipartUpload still requires the client to issue CopyObjectPart calls for the entire object.

nostrebored 4y ago
Distributed patching becomes hell. You need transactional semantics and files are not laid out well to help you define invariants that should reject the transaction.
- supriyo-biswas 4y ago
  There is no reason why the descriptor of objects can’t be updated with a new value that has all of the old chunks and a new one, since S3 doesn’t do deduplication anyway the other chunks may be resized internally with an asynchronous process that gets rid of the excess data corresponding to the now overridden chunk.
wmf 4y ago
Eventually AWS always gives customers what they want even if it's a "bad idea".
- mlhpdx 4y ago
  Indeed.
- znpy 4y ago
  Bad ideas are very relative. Some bad ideas work extremely well if they fit your use case, you understand very well the tradeoffs and you’re building safeguards (disaster recovery). Some other companies try to convince (force?) you into a workflow or into a specific solution. Aws just gives you the tools and some guidance on how to use them best.

tolmasky 4y ago

Couldn't tell from the README, does this do any sort of cache management or LRU type thing? In other words, does it fetch the underlying S3 object in real time, and then eventually eject them from memory and/or the backing FS when they haven't been used for a while?

renewiltord 4y ago
`catfs` is a FUSE FS that can do this for you. You'll need some changes to make it work well. I'll have a friend upstream them soon, but they're easy to make yourself. Could replace `goofys` with this and then stick `catfs` in front.

sfritz 4y ago

There's a similar project under awslabs for using S3 as a FileSystem within the Java JVM: https://github.com/awslabs/aws-java-nio-spi-for-s3 https://github.com/awslabs/aws-java-nio-spi-for-s3

ary 4y ago

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new files, and must be done sequentially. Modifying existing files will not be supported. Truncation will not be supported. The sequential requirement for writes is the part that I've been mulling over whether or not it's actually required in S3. Last year I discovered that S3 can do transactional I/O via multipart upload[2] operations combined with the CopyObject[3] operation. This should, in theory, allow for out of order writes, existing partial object re-use, and file appends. [1] https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMAN... [2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuove... [3] https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje...

deleted 4y ago
[deleted]
FullyFunctional 4y ago
Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.
- Scarbutt 4y ago
  S3 is slow but at the same time low cost, if you want fast AWS has other alternatives but pricier.
  - mlhpdx 4y ago
    This is misleading. S3 is also incredibly fast. The former when you’re sequentially writing (or reading) objects and the latter when concurrently writing (or reading) vast numbers of them.

IceWreck 4y ago

They should benchmark it against rclone

BrianHenryIE 4y ago

JungleDisk was backup software I used ~2009 that allowed mounting S3. They were bought by Rackspace and the product wasn't updated. Seems to be called/part of Cyberfortress now. Later I used Panic's Transmit Disk but they removed the feature. Recently I'd been looking at s3fs-fuse to use with gocryptfs but haven't actually installed it yet! https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse https://github.com/rfjakob/gocryptfs https://github.com/rfjakob/gocryptfs

CharlesW 4y ago
> Later I used Panic's Transmit Disk but they removed the feature. BTW, Panic seemingly intends to re-build Transmit Disk. Hopefully it'll be part of Transmit 6: https://help.panic.com/transmit/transmit5/transmit-disk/#technical-details-and-the-future-of-transmit-disk https://help.panic.com/transmit/transmit5/transmit-disk/#tec... A supported macOS option appears to be Mountain Duck: https://mountainduck.io/ https://mountainduck.io/
- drcongo 4y ago
  ForkLift also lets you mount S3 as a drive. https://binarynights.com https://binarynights.com
legorobot 4y ago
We've used the s3fs-fuse library for a while at work for SFTP/FTP server alternatives (AWS wants you to pay $150+/server/month last I checked!) and it's worked like a dream. We scripted the setup of new users via a simple bash script and the S3 CloudWatch events for file uploads is a dream. Its been pretty seamless and hasn't caused many headaches. We've had to perform occasional maintenance but its operated for years with no major issues. 99% are solved with a server restart + a startup script to auto-re-mount s3fs-fuse in all the appropriate places. Give them a try, I recommend it!

mlindner 4y ago

There's some really confusing use of unsafe going on. For example I'm not sure what they're doing here: https://github.com/awslabs/mountpoint-s3/blob/main/mountpoint-s3-crt/src/io/stream.rs#L111 https://github.com/awslabs/mountpoint-s3/blob/main/mountpoin...

Dinux 4y ago

This is exactly what I need. The current python scrips are good enough but a rust utility would be preferable

hkgjjgjfjfjfjf 4y ago

[dead]

wanderingmind 4y ago

For anyone looking to mount S3 as file system, I will suggest giving rclone a shot. It can mount, copy and do all file operations not just on s3 but on a wide range of cloud providers, you can also declare a remote as encrypted so it does client side encryption

hawski 4y ago

I would suggest changing the title to "Mountpoint-S3 - ..." as that's the project name to avoid confusion with mountpoint(1): https://man7.org/linux/man-pages/man1/mountpoint.1.html https://man7.org/linux/man-pages/man1/mountpoint.1.html

arretevad 4y ago

I find all the “written in Rust” qualifier in all these post title to just be a distraction. It doesn’t feel like it’s telling me anything.

grogenaut 4y ago
Amazon is starting to invest in rust internally, strategically, since joining some of the leadership (https://aws.amazon.com/blogs/opensource/why-aws-loves-rust-and-how-wed-like-to-help/ https://aws.amazon.com/blogs/opensource/why-aws-loves-rust-a...). It's considered a good replacement for C++, and like go, is really good for releasing tools. Tools like the AWS CLI... that work great when you can plop a single exe down as your install story, as opposed to say a python install and app (aws cli). But it's also still new. Releasing a tool like this is likely a big deal in the area, and they're likely quite prod of it due to the effort of things like getting legal approval, marketing, etc, let alone the cool nerd factor of a filesystem, who doesn't want to show off by having written a filesystem or hell a fuse plugin... file system over dns anyone?

zsy056 4y ago

If you are looking for something supports atomic rename, you can checkout Blobfuse2[0] + ADLS Gen2[1] Disclaimer: work for MSFT [0] https://github.com/Azure/azure-storage-fuse https://github.com/Azure/azure-storage-fuse [1] https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace https://learn.microsoft.com/en-us/azure/storage/blobs/data-l...

baggiponte 4y ago

I don’t understand whether this is just a higher level abstraction of boto’s s3 client (à la s3fs)

biorach 4y ago
It looks to be a completely different codebase from boto/s3fs. Not having used s3fs I'm going to guess that s3fs is limited due to the limits of the underlying language - Python - namely poor performance overall and poor multi-thread story. I'd imagine s3fs is useful for stuff like backing up personal projects, quickly sharing files between developers etc. For operating at any kind of scale - in terms of concurrent requests, number or size of files etc - I'd guess that Mountpoint would be the only viable solution.

rdxm 4y ago

[dead]