17 ms·
S3 is files, but not a filesystem
- 3weeksearlier 3y agoI dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.
- fiddlerwoaroof 3y agoYeah, it’s sort of funny how “POSIXish semantics” has become our definition of these things, when it’s just one kind of thing that’s been called a filesystem historically.
- mickael-kerjean 3y agoFun experiment I made with my mum, building a storage independent dropbox like UI [1] for anything that implement this interface: type IBackend interface { Ls(path string) ([]os.FileInfo, error) Cat(path string) (io.ReadCloser, error) Mkdir(path string) error Rm(path string) error Mv(from string, to string) error Save(path string, file io.Reader) error Touch(path string) error } My mum really couldn't care less about the posix semantic as soon as she can see the pictures of my kid which happen to be on S3 [1] https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash
- wwalexander 3y agoReducing things to basically the interface you laid out is the point of 9p [1], and is what Plan 9’s UNIX-but-distributed design was built on top of. Same inventor as Go! If you haven’t dived down the Plan 9 rabbit hole yet, it’s a beautiful and haunting vision of how simple cloud computing could have been. [1] https://9fans.github.io/plan9port/man/man9/intro.html https://9fans.github.io/plan9port/man/man9/intro.html
- MrJohz 3y agoI think this interface is less interesting than the semantics behind it, particularly when it comes to concurrency: what happens when you delete a folder, and then try and create a file in that folder at the same time? What happens when you move a folder to a new location, and during that move, delete the new or old folders? Like yes, for your mum's use case, with a single user, it's probably not all that important that you cover those edge cases, but every time I've built pseudo-filesystems on top of non-filesystem storage APIs, those sorts of semantic questions have been where all the problems have hidden. It's not particularly hard to implement the interface you've described, but it's very hard to do it in such a way that, for example, you never have dangling files that exist but aren't contained in any folder, or that you never have multiple files with the same path, and so on.
- mickael-kerjean 3y agoAll those considerations are important when implementing the interface but the interface itself isn't invalidated by those concerns and can cope with those constraints fine.
- DonHopkins 3y agoCan S3 murder your wife like ReiserFS and Reiser4? https://en.wikipedia.org/w/index.php?title=Comparison_of_file_systems&oldid=209063556#Features https://en.wikipedia.org/w/index.php?title=Comparison_of_fil...
- CobrastanJorji 3y agoThey are necessary because as soon as someone decides that S3 is a filesystem, they will look at the other cloud "filesystems," notice that S3 is cheaper than most of them, and then for some reason they will decide to run giant Hadoop fs stuff on it or mount a relational database on it or all other manner of stupidity. I guarantee you S3's customer-facing engineers are fielding multiple calls per week from customers who are angry that S3 isn't as fast as some real filesystem solution that the customer migrated from because S3 was cheaper. When people decide that X is a filesystem, they try to use it like it's a local, POSIX filesystem, and that's terrible because it won't be immediately obvious why it's a stupid plan.
- albert_e 3y agoIf a customer makes an IT decision as big as running Hadoop or RDBMS with S3 as storage ... but does not consult at least a Associate level AWS Certified architect (who are doke a dozen) for at least one day worth of advice which is probably a couple of hundred dollars at most ... Can we really blame AWS? I am sure none of official AWS documentations or examples show such an architecture. ---- Amazon EMR can run Hadoop and use Amazon S3 as storage via EMR FS. "S3 mountpoints" are a feature specifically for workloads that need to see S3 as a file system. For block storage workloads there is EBS and EFS and FSx that AWS heavily advertises.
- albert_e 3y ago*dime a dozen (Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)
- karmasimida 3y agoExactly, especially when the concept of filesystem really is defined before the whole internet scale becomes a thing or reality. Maybe S3 isn't a filesystem according to this definition, but does it really matter to make it one? I doubt it. The Elastic Filesystem is also an AWS product, but you can't really work as one as you have locally, any folder over 20k files basically will timeout if you do a ls. Does it make EFS a filesystem or not?
- deleted 3y ago[deleted]
- yencabulator 3y agoThe problem is once you let go of those semantics, a lot of software stops working if run against such a "filesystem". If you dilute the meaning of "filesystem" too much, it becomes less useful as a term. https://en.wikipedia.org/wiki/Andrew_File_System https://en.wikipedia.org/wiki/Andrew_File_System was interesting, I'd actually love to see something similar re-implemented with modern ideas, but it's more of an direct-access archival system than a general-purpose filesystem[1], you can't just put files written by arbitrary software on it. It's a bit like NFS without locks&leases, but even less like a normal filesystem; only really good for files created once that "settle down" into effectively being read-only. [1]: I wrote https://github.com/bazil/plop https://github.com/bazil/plop that is (unfortunately undocumented) content-addressed immutable file storage over object storage, used in conjunction with a git repo with symlinks to it to manage the "naming layer". See https://bazil.org/doc/ https://bazil.org/doc/ for background, plop is basically a simplification of the ideas to get to working code easier. Site hasn't been updated in almost a decade, wow. It's in everyday use though!
- leetrout 3y agoMy big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to support the idea of folders. https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
- riehwvfbk 3y agoIs that really so different from how folders work on other systems? A directory inode is just an inode.
- daynthelife 3y agoThe payload still contains a list of other inodes though
- klodolph 3y agoYes. It is, in practice, incredibly different. Imagine you have a file named /some/dir/file.jpg. In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table: +-------+--------+----------+-------+ | inode | parent | name | data | +-------+--------+----------+-------+ | 1 | (null) | some | (dir) | | 2 | 1 | dir | (dir) | | 3 | 2 | file.jpg | jpeg | +-------+--------+----------+-------+ In S3 (and other object stores), the table is like this: +-------------------+------+ | key | data | +-------------------+------+ | some/dir/file.jpg | jpeg | +-------------------+------+ The kind of queries you can do is completely different. There are no inodes in S3. There is just a mapping from keys to objects. There’s an index on these keys, so you can do queries—but the / character is NOT SPECIAL and does not actually have any significance to the S3 storage system and API. The / character only has significance in the UI. You can, if you want, use a completely different character to separate “components” in S3, rather than using /, because / is not special. If you want something like “some:dir:file.jpg” or “some.dir.file.jpg” you can do that. Again, because / is not special.
- inkyoto 3y agoS3 is a tagged versioned object storage with file like semantics implemented in the AWS SDK (via AWS S3 API's). The S3 object key is the tag. Files and folders are used to make S3 buckets more approachable to those who either don't know or don't want to know what it actually is, and one day they get a surprise.
- deleted 3y ago[deleted]
- Twirrim 3y agoS3 is a key value store. Just happens to be able to store really large values.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- dmarinus 3y agoI talked to people at AWS who work in RDS Aurora and they hinted they use S3 internally as a backend for MySQL and PostgreSQL.
- readyman 3y agoBig if true. That was definitely not in the AWS cert I took lol.
- multani 3y agoSeparating compute and storage is one of the core idea behind Aurora. They talked about it in several places, for instance: * https://www.amazon.science/publications/amazon-aurora-design-considerations-for-high-throughput-cloud-native-relational-databases https://www.amazon.science/publications/amazon-aurora-design... * https://d1.awsstatic.com/events/reinvent/2019/REPEAT_Amazon_Aurora_storage_demystified_How_it_all_works_DAT309-R.pdf https://d1.awsstatic.com/events/reinvent/2019/REPEAT_Amazon_...
- WatchDog 3y agoMaybe for snapshots, but certainly not for live data.
- shell_game 3y agoEBS not S3
- YouWhy 3y agoThe article is well written, but I am annoyed at the attempt to gatekeep the definition of a filesystem. Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I suspect latency would kill the hypothetical use case of Postgres over S3 even if S3 had incorporated the necessary API semantics - could somebody more knowledgeable chime in?). A more interesting venue to pursue would be - what other additions could be made to the S3 API to make it more usable on its own right - for example, why doesn't S3 offer more than one filename per blob? (e.g., a similar to what links do in POSIX)
- bilalq 3y agoThis might be of interest to you: https://neon.tech/blog/bring-your-own-s3-to-neon https://neon.tech/blog/bring-your-own-s3-to-neon. There's also the OG Aurora whitepaper: https://www.amazon.science/publications/amazon-aurora-design-considerations-for-high-throughput-cloud-native-relational-databases https://www.amazon.science/publications/amazon-aurora-design...
- zX41ZdbW 3y agoClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, possibly as a result of a background merge operation. S3 API is almost enough, except for cases of concurrent database updates. In this case, it is not possible to rely on S3 only because it does not support an atomic "write if not exists" operation. That's why external, strongly consistent metadata storage is needed, which is handled by ClickHouse Keeper.
- afiori 3y agoIs a "write if not exists" atomic operation enouhg as a concurrency primitive for database locks?
- 3y ago
- readyman 3y ago[flagged]
- smcin 3y agoTheir writing seems okay to me; which specific parts do you find atrocious?
- calpaterson 3y agoAuthor here, I can't reply to the GP because that comment is "dead" but yes, please, be specific! Then I can fix the sentences that are bad and perhaps also improve in the future.
- readyman 3y ago"Even though the file API handles all those concerns, but it doesn't expose them to you. A narrow interface handling a large number of concerns - that makes the unix file API a "deep" module." Both sentences here are incomplete, incoherent. I did not read past this point.
- calpaterson 3y agoThanks. I take your point on the first one and have corrected it (maybe you need to shift+F5 to bust your cache in order to see it). For the second one, what's your objection? That it's fragmentary?
- danrob 3y ago[flagged]
- gjvc 3y agosure seems like it
- throwaway892238 3y ago> The "simple" in S3 is a misnomer. S3 is not actually simple. It's deep. Simple doesn't mean "not deep". It means having the fewest parts needed in order to accomplish your requirements. If you require a distributed, centralized, replicated, high-availability, high-durability, high-bandwidth, low-latency, strongly-consistent, synchronous, scalable object store with HTTP REST API, you can't get much simpler than S3. Lots of features have been added to AWS S3 over the years, but the basic operation has remained the same.
- svat 3y ago> It means having the fewest parts needed in order to accomplish your requirements. That is exactly what "deep" means, in the terminology of this post (from Ousterhout's book A Philosophy of Software Design). Simple means "not complex" (see also Rich Hickey's talk Simple Made Easy: https://www.infoq.com/presentations/Simple-Made-Easy/ https://www.infoq.com/presentations/Simple-Made-Easy/), while "deep" means providing/having a lot of internally-complex functionality via a small interface. The latter is a better description of S3 (which is what you seem to be saying too) than "simple" which would mean there isn't much to it.
- throwaway892238 3y agoHickey's definition of simple is wrong. It's not the opposite of complex at all. They are not opposites, nor mutually exclusive. - Easy is when something does not require much effort. - Simple means the least complex it can be and still work. - Complex means there are lots of components. These are all quite different concepts: - Easy is a concept that distinguishes the amount of work needed to use a solution - Simple is a concept that distinguishes whether or not there is an excess number of interacting properties in a system - Complex is a concept describing the quality of having a number of interacting properties in a system Hickey's talk is useful in terms of thinking about software, but it also contains many over-generalizations which are incorrect and lead to incorrect thinking about things that aren't software. (Even some of his declarations about software are wrong) "Deep", in the context of software complexity, probably only makes sense in terms of describing the number of layers involved in a piece of technology. You could make something have many layers, and it could still be simple, or be complex, or easy.
- type_Ben_struct 3y agoTools like LucidLink and Weka go a way to making S3 even more of a “file system”. They break files into smaller chunks (S3 objects) which helps with partial writes, reads and performance. Alongside tiering of data from S3 to disk when needed for performance.
- hnlmorg 3y agoI don’t know a whole lot about LucidLink but Weka basically uses S3 as a dataplane for their own file system.
- rwmj 3y agoSomeone contributed an nbdkit S3 plugin which basically works the way you described. It uses numbered S3 chunks using the pattern "key/%16x", allowing the virtual disk to be updated. (https://libguestfs.org/nbdkit-S3-plugin.1.html https://libguestfs.org/nbdkit-S3-plugin.1.html https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/S3 https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/S3)
- cuno 3y agoThe problem with these approaches is that the data is scrambled on the backend, so you can't access the files directly from S3 anymore. Instead you need an S3 gateway to convert from scrambled S3 to unscrambled S3. They rely on a separate database to reassemble the pieces back together again.
- hn72774 3y ago> Filesystem software, especially databases, can't be ported to Amazon S3 Hudi, Delta, iceberg bridge that gap now. Databricks built a company around it. Don't try to do relational on object storage on your own. Use one of those libraries. It seems simple but it's not. Late arriving data, deletes, updates, primary key column values changing, etc.
- albert_e 3y agoThere is specifically block storage service (EBS) and falvirs of it like EBS multi-attach and EFS that can ne used if there is a need to port software/databases to the cloud with low level filesystem support. Why would we need to do it on object storage which addresses a different type of storage need. Nevertheless there are projects like EMRFS and S3 file system mount points that try to provide files stem interfaces to workloads that need to see S3 as a filesystem.
- albert_e 3y ago*flavors *can be used *file system (Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)
- hn72774 3y agoS3 is better for large datasets. It's cheaper and handles large file sizes with ease. It has become a de-facto standard for distributed, data-intensive workloads like those common with spark. A key benefit is decoupling the data from the compute so that they can scale independently. EBS is tightly coupled to iops and you pay extra for that. (Source: a long time working in data engineering)
- albert_e 3y agoYes and I also believe: Experienced Spark / Data Engineering teams would not assume S3 is readily useable as a filesystem. This [1] seems like a good guide on how to configure spark for working with Cloud object stores, while recognizing the limitations and pitfalls. [1]: https://spark.apache.org/docs/latest/cloud-integration.html https://spark.apache.org/docs/latest/cloud-integration.html --- Amazon EMR offers a managed way to run hadoop or spark clusters and it implements an "EMR FS" [2] system to interface with S3 as storage. [2]: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-fs.html https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-fs.h... AWS Glue is another option which is "serverless" ETL. Source and Destination can be S3 data lakes read through a data catalog (hive or glue data catalog). During processing AWs Glue can optionally use S3 [3,4,5] for shuffle partition. [3]: https://aws.amazon.com/blogs/big-data/introducing-amazon-s3-shuffle-in-aws-glue/ https://aws.amazon.com/blogs/big-data/introducing-amazon-s3-... [4]: https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shuffle-manager.html https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shu... [5]: https://aws.amazon.com/blogs/big-data/introducing-the-cloud-shuffle-storage-plugin-for-apache-spark/ https://aws.amazon.com/blogs/big-data/introducing-the-cloud-...
- zmmmmm 3y agoThe limitations of S3 (and all the cloud "file systems") are quite astonishing when you consider you're paying for it as a premium service. Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you: - can't store log files because it can't append anything to an existing files - can't copy files more than 5GB - can't rename or move a file When challenged on how you are supposed to make all your applications work with limitations like that, they glibly told you "oh you're supposed to rewrite them all".
- throwaway290 3y agoIt's for building things on top. If you want to rename/move/copy data, implement a layer that maps objects to "filenames" or any metadata you like (or use some lib). If you want to write logs, implement append and rotation. But I for example don't and won't need any of that and if it helps keep the API simpler and more reliable then I benefit. being a conventional filesystem for S3 would be either a very leaky abstraction or completely different product
- School-Cotton 3y agoAmazon doesn’t market S3 as a replacement for file systems, that’s why EBS exists. Also, is S3 really “very expensive”? Relative to what?
- vbezhenar 3y agoS3 usually is the cheapest storage, not only for Amazon, but for other clouds. I don’t understand why.
- ForHackernews 3y agoThis is not true in my experience https://www.backblaze.com/cloud-storage/pricing https://www.backblaze.com/cloud-storage/pricing
- 3y ago
- hiAndrewQuinn 3y agoI feel like I understand the lasting popularity of the humble FTP fileserver a bit better now. Thank you.
- jugg1es 3y agooh but amazon offers SFTP on top of S3 so you don't have to miss out.
- hiAndrewQuinn 3y agoIf it's offered on top of S3, though, doesn't it still have all the same issues of needing to totally overwrite files?
- globular-toast 3y agoA filesystem is an abstraction built on a block device. A block device just gives you a massive array of bytes and lets you read/write from them in blocks (e.g. write these 300 bytes at position 273041). A block device itself is an abstraction built on real hardware. "Write these 300 bytes" really means something like "move needle on platter 2 to position 6... etc" S3 is just a different abstraction that is also built on raw storage somehow. It's a strictly flat key-object store. That's it. I don't know why people have a problem with this. If you need "filesystem stuff" then implement it in your app, or use a filesystem. You only need to append? Use a database to keep track of the chain of appends and store the chunks in S3. Doesn't work for you? Use something else. Need to "copy"? Make a new reference to the same object in your db. Doesn't work for you? Use something else. S3 works for a lot of people. Stop trying to make it something else. And stop trying to change the meaning of super well-established names in your field. A filesystem is described in text books everywhere. S3 is not a filesystem and never claimed to be one. Oh and please study a bit of operating system design. Just a little bit. It really helps and is great fun too.
- gjvc 3y agoJFC the people on this thread missing the difference between object storage and a blocks-and-inodes filesystem is alarming
- nickcw 3y agoGreat article - would have been useful to read before starting out on the journey of making rclone mount (mount your cloud storage via fuse)! After a lot of iterating we eventually came up with the VFS layer in rclone which adapts S3 (or any other similar storage system like Google Cloud Storage, Azure Blob, Openstack Swift, Oracle Object Storage, etc) into a POSIX-ish file system layer in rclone. The actual rclone mount code is quite a thin layer on top of this. The VFS layer has various levels of compatibility "off" where it just does directory caching. In this mode, like the article states you can't read and write to a file simultaneously and you can't write to the middle of a file and you can only write files sequentially. Surprisingly quite a lot of things work OK with these limitations. The next level up is "writes" - this supports nearly all the POSIX features that applications want like being able to read and write to the same file at the same time, write to the middle of the file, etc. The cost for that though is a local copy of the file which is uploaded asynchronously when it is closed. Here are some docs for the VFS caching modes - these mirror the limitations in the article nicely! https://rclone.org/commands/rclone_mount/#vfs-file-caching https://rclone.org/commands/rclone_mount/#vfs-file-caching By default S3 doesn't have real directories either. This means you can't have a directory with no files in, and directories don't have valid metadata (like modification time). You can create zero length files ending in / which are known as directory markers and a lot of tools (including rclone) support these. Not being able to have empty directories isn't too much of a problem normally as the VFS layer fakes them and most apps then write something into their empty directories pretty quickly. So it is really quite a lot of work trying to convert something which looks like S3 into something which looks like a POSIX file system. There is a whole lot of smoke and mirrors behind the scene when things like renaming an open file happens and other nasty corner cases like that. Rclone's lower level move/sync/copy commands don't bother though and use the S3 API pretty much as-is. If I could change one thing about S3's API I would like an option to read the metadata with the listings. Rclone stores modification times of files as metadata on the object and there isn't a bulk way of reading these, you have to HEAD the object. Or alternatively a way of setting the Last-Modified on an object when you upload it would do too.
- Hakkin 3y ago> If I could change one thing about S3's API I would like an option to read the metadata with the listings. Rclone stores modification times of files as metadata on the object and there isn't a bulk way of reading these, you have to HEAD the object. Or alternatively a way of setting the Last-Modified on an object when you upload it would do too. I wonder if you couldn't hack this in by storing the metadata in the key name itself? Obviously with the key length limit of 1024 you would be limited in how much metadata you could store, but it's still quite a lot of space, even taking into account the file path. You could use a deliminator that would be invalid in a normalized path, like '//', for example: /path/to/file.txt//mtime=1710066090 You would still be able to fetch "directories" via prefixes and direct files by using '<filename>//' as the prefix. This kind of formatting would probably make it pretty incompatible with other software though.
- wodenokoto 3y agoIs there a generic name for these distributed cloud file storages? AWS is S3, google is buckets, Azure is blob storage, the open source version is … ?
- dexwiz 3y agoObject Storage
- jeffbr13 3y agoI tend to go by Binary Large OBject (BLOB) storage to discern between this kind of object storage and “object” as in OOP. BLOB is also what databases call files stored in columns.
- OJFord 3y agoWhen would that be confusing? As in what would an AWS service offering OOP object storage be/mean?
- gilbetron 3y ago"blob storage" is the usual generic term, even though Azure uses it explicitly. It's like calling adhesive bandages, "bandaids" even though that is a specific company's term.
- surajrmal 3y agoGoogle buckets is a bit off - the product is called Google storage. Buckets are also a term used by s3 and are equivalent to azure blob storage containers. They are an intermediary layer that determines attributes for the objects stored within it such as ACLs and storage class (and therefore cost and performance). As to your question, object storage[1] seems to be the generic term for the technology. Internally they all rely on naming files based on the hash of their contents for quick lookup, deduplication, and avoiding name clashes. 1: https://en.wikipedia.org/wiki/Object_storage https://en.wikipedia.org/wiki/Object_storage
- tison 3y agoIt's ever discussed in https://github.com/apache/arrow-rs/issues/3888 https://github.com/apache/arrow-rs/issues/3888 for comparing object_store in Apache Arrow to the APIs provided by Apache OpenDAL. Briefly, Apache OpenDAL is a library providing FS-like APIs over multiple storage backends, including S3 and many other cloud storage. A few database systems, such as GreptimeDB and Databend, use OpenDAL as a better S3 SDK to access data on cloud storage. Other solutions exist to manage filesystem-like interfaces over S3, including Alluxio and JuiceFS. Unlike Apache OpenDAL, Alluxio and JuiceFS need to be deployed standalone and have a dedicated internal metadata service.
- Lucasoato 3y agoI'm not sure if Alluxio could be substituted by OpenDAL as a local cache layer for TrinoDB.
- tison 3y agoIf I get "local cache layer" correctly, it's possible. And it's even desired if you want to reduce the deployment burden. Here are some related codes on how we implement such a layer in GreptimeDB: * https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/object-store/src/layers/lru_cache.rs https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/o... * https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/mito2/src/sst/parquet/helper.rs https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/m...
- cynicalsecurity 3y agoBackblaze B2 is worth mentioning while we are speaking of S3. I'm absolutely in love with their prices (3 times lower than of S3). (I'm not their representative).
- silvertaza 3y agoWith every alternative, the prevailing issue is the fact that your data is as safe as the company your data is with. But I think this can be remedied by doubly external backups.
- didgeoridoo 3y agoB2 having an S3-compatible API available makes this particularly easy :)
- OJFord 3y agoBackblaze is like if Amazon spun AWS S3 out as its own business (and it added some backup helper tooling as a result) though, I wouldn't really worry any more about it. You could write a second copy to S3 Glacier Deep Archive (using B2 for instant access when you wanted to restore or on a new device) and still be much cheaper.
- deleted 3y ago[deleted]
- overstay8930 3y agoWe liked B2 but not enough to pay for IPv4 addresses, insane they advertise as a multi-cloud solution but basically kill any chance at adoption when NAT gateways and IPv4 charges are everywhere. We would literally save money paying B2 bandwidth fees (high read low write) but not when being pushed through a NAT64 gateway, or paying an hourly charge just to be able to access B2.
- Kwpolska 3y agoHow could they launch a cloud service like this and not have IPv6 in 2015? What other basic things did they cheap out on?
- orf 3y ago> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful. What makes it useful is listing. In an unversioned bucket (or one with no delete markers), listing any given prefix is essentially constant time: I can take any given string, in a bucket with 100 billion objects, and say “give me the next 1000 keys alphabetically that come after this random string”. What’s more, using “/“ as a delimiter is just the default - you can use any character you want and get a set of common prefixes. There are no “directories”, ”directories” are created out of thin air on demand. This is super powerful, and it’s the thing that lets you partition your data in various ways, using whatever identifiers you need, without worrying about performance. If listing was just “slow”, couldn’t list on file prefixes and got slower proportional to the number of keys (I.e a traditional unix file system), then it wouldn’t be useful at all.
- adrian_b 3y agoSince 30 years ago (starting with XFS in 1993, which was inspired by HPFS) all the good UNIX file systems implement the directories as some kind of B trees. Therefore they do not get slower proportional to the number of entries and listing based on file prefixes is extremely fast.
- orf 3y agoYes they do. What APIs does Linux offer that allows you to list a directories contents alphabetically starting at a specific filename in constant time? You have to iterate the directory contents. You can maybe use “d_off” with readdir in some way, but that’s specific to the filesystem. There’s no portable way to do this with POSIX. Regardless of if you can do it with a single directory, you can’t do it for all files recursively under a given prefix. You can’t just ignore directories, or say that “for this list request, ‘-‘ is my directory separator”. The use of b-trees in file systems is completely beside the point.
- 3y ago
- finalhacker 3y agoS3 not implementate vfs api, but you can treat it as a software defined storage filesystem. Just like Ceph. there are so many applications depends on file storage, such as Mysql. But horizontal scale for those app still difficult in many case. Replace from vfs api to s3 storage perhaps is trending in my experience.
- foxhop 3y ago[dead]
- igtztorrero 3y agoCheck out kopia.io is a backup software that uses S3 to store files by blocks or pages. You can browse, search and sort the files and directories of the different snapshot or versions of the file. I love it ! For me it's a file system in S3. Bonus: you must use a key, to encrypt the files.
- MatthiasPortzel 3y agoThis article was an epiphany for me because I realized I've been thinking of the Unix filesystem as if it has two functions: read_file and write_file. (And then getting frustrated with the filesystem APIs in programming languages.)
- markhahn 3y agoSo you came from an S3 or other put-get world, and found actual filesystems odd? I suppose that's not so different from a WMP user's epiphany when they discover processes, shells, etc.
- MatthiasPortzel 3y agoWell I’m used to an application-level view of the file system. A document editor or text editor opens files and saves files, but these are whole-document operations. I can’t open a document in Sublime Text without reading it, and I can’t save part of a file without saving all of it. So it’s not obvious that these would be different at an OS level. As the post points out, there are uses for Unix’s sub-file-level read-and-write commands, but I’ve never needed them.
- arvindamirtaa 3y agoLike Gmail is emails but not IMAP. It's fine. We have seen that these kinds of wrappers work pretty well most of the time considering the performance and simplicity they bring in building and managing these systems.
- chrisblackwell 3y agoRandom note: Has anyone noticed how fast the author's webpage is? I know it's static, but I mean it's fast even for the DNS lookup. I would love to know what they have on.
- adverbly 3y agoThe response headers include server: cloudflare You said it though - the reason is that its static without any js/frameworks/SPA round trip requests.
- overstay8930 3y agoFull stack Cloudflare is really fast
- calpaterson 3y agoThere is an nginx server running on debian stable somewhere in the dark heart of Germany. But I do have numerous tricks (too many, probably) to keep things quick. But there are still ways to be quicker. For example, the header photo is smaller than the vector diagrams on the page, by about tenfold.
- prima-facie 3y agoCould be using Cloudflare pages hosted on a R2 bucket: https://pages.cloudflare.com/ https://pages.cloudflare.com/
- alphazard 3y agoS3 is not even files, and definitely not a filesystem. The thing I would expect from a file abstraction is mutability. I should be able to edit pieces of a file, grow it, shrink it, read and write at random offsets. I shouldn't have to go back up to the root, or a higher level concept once I have the file in hand. S3 provides a mutable listing of immutable objects, if I want to do any of the mutability business, I need to make a copy and re-upload. As originally conceived, the file abstraction finds some sectors on disk, and presents them to the client as a contiguous buffer. S3 solves a different problem. Many people misinterpret the Good Idea from UNIX "everything is a file" to mean that everything should look like a contiguous virtual buffer. That's not what the real Good Idea is. Really: everything can be listed in a directory, including directories. There will be base leaves, which could be files, or any object the system wants to present to a process, and there will be recursive trees (which are directories). The directories are what make the filesystem, not the type of a particular leaf. Adding a new type of leaf, like a socket or a frame buffer, or whatever, is almost boring, and doesn't erode the integrity of the real good idea. Adding a different kind of container like a list, would make the structure of the filesystem more complex, and that would erode the conceptual integrity. S3 doesn't do any of these things, and that's fine. I just want a place to put things that won't fit in the database, and know they won't bitrot when I'm not looking. The desire to make S3 look more like a filesystem comes from client misunderstanding of what it's good at/for, and poor product management indulging that misunderstanding instead of guarding the system from it.
- akerl_ 3y agoHow do read-only filesystems align with your definition?
- yencabulator 3y agoYou can't create new things on a read-only filesystem, you can in S3; not a good analogy.
- akerl_ 3y agoI wasn’t making an analogy. I was asking how read-only filesystem works given the parent commenters description of what makes something a filesystem.
- _9hey 3y agoIt’s been a while, but I really like the way google handles its file system internally. No confusion.
- _9hey 3y agoHoly crap I hate Hacker News! Why tf did this get a downvote? Colossus is a good fs.
- remram 3y agoI am currently pondering this exact problem. I want to run a file-sharing web application (think: NextCloud) but I don't want to use expensive block storage or the dedicated server's disk space for the files, as some of them will be accessed infrequently. I am wondering if s3fs/rclone-mount is sufficient, or if I should use something like JuiceFS that adds random-access, renaming, etc on top of it. Are those really necessary APIs for my use case? Is there only one way to find out? (The app doesn't have native S3 support)
- cuno 3y agoIt depends on if you want to expose filesystem semantics or metadata to applications using it. For example random access writes are done by ffmpeg, which is a workhorse of the media industry, but most things can't handle that or are too slow. We had to build our own solution cunoFS to make it work properly at high speeds.
- jkoudys 3y agoI absolutely loved this article. Super well written with interesting insights.
- calpaterson 3y agoThat is very kind - glad to hear that you enjoyed it.
- donatj 3y ago> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem. I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me his pre-populated cache. My initial thought was “this can’t really be necessary” and started poking. We have ~100,000 root level directories for our individual assets. Each of those have five or six directories with a handful of files. Probably less than a million files total, maybe 3 levels deep at its deepest. Recursively listing these files takes literally fifteen minutes. I poked and prodded suggestions from stack overflow and ChatGPT at potential ways to speed up the process and got nothing notable. That’s absurdly slow. Why on earth is it so slow? Why is this something Amazon has not fixed? From the outside really seems like they could slap some B-trees on the individual buckets and call it a day. If it is a difficult problem, I’m sure it would be for fascinating reasons I’d love to hear about.
- returningfory2 3y agoAre you performing list calls sequentially? If you have O(100k) directories and are doing O(100k) requests sequentially, 15 minutes works out at O(10ms) per request which doesn’t seem that bad? (assuming my math is correct…)
- luhn 3y agoAt risk of being pedantic, you seem to be using big O to mean “approximately” or “in the order of”, but that’s not what it means at all. Big O is an expression of the growth rate of a function. Any constant value has a growth rate of 0, so O(100k) isn’t meaningful: It’s exactly the same as O(1).
- wetmore 3y agoYou're right technically, it's an abuse of notation that isn't uncommon. My physics profs would do it in college.
- breckognize 3y ago> I haven't heard of people having problems [with S3's Durability] but equally: I've never seen these claims tested. I am at least a bit curious about these claims. Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture. AWS' availability zone isolation is better than the other cloud providers. When I worked at S3, customers would beat us up over pricing compared to GCP blob storage, but the comparison was unfair because Google would store your data in the same building (or maybe different rooms of the same building) - not with the separation AWS did. The entire organization was unbelievably paranoid about data integrity (checksum all the things) and bigger events like natural disasters. S3 even operates at a scale where we could detect "bitrot" - random bit flips caused by gamma rays hitting a hard drive platter (roughly one per second across trillions of objects iirc). We even measured failure rates by hard drive vendor/vintage to minimize the chance of data loss if a batch of disks went bad. I wouldn't store critical data anywhere else. Source: I wrote the S3 placement system.
- tracerbulletx 3y agoMy first job was at a startup in 2012 where I was expected to build things at a scale way over what I really had the experience to do. Anyways the best choice I ever made was using RDS and S3 (and django).
- notso411 3y ago[dead]
- supriyo-biswas 3y agoChecksumming the data is not based out of paranoia but simply as a result of having to detect which blocks are unusable in order to run the Reed-Solomon algorithm. I'd also assume that a sufficient number of these corruption events are used as a signal to "heal" the system by migrating the individual data blocks onto different machines. Overall, I'd say the things that you mentioned are pretty typical of a storage system, and are not at all specific to S3 :)
- svat 3y agoIt's nice to see Ousterhout's idea of module depth (the main idea from his A Philosophy of Software Design) getting more mainstream — mentioned in this article with attribution only in "Other notes", which suggests the author found it natural enough not to require elaboration. Being obvious-in-hindsight like this is a sign of a good idea. :-) > The concept of deep vs shallow modules comes from John Ousterhout's excellent book. The book is [effectively] a list of ideas on software design. Some are real hits with me, others not, but well worth reading overall. Praise for making it succinct.
- calpaterson 3y agoI was tempted to define and cite the term more carefully before I used it but that leadened the article a lot right in the middle and so I cut it and just hoped. It is a great concept and also a great book. I really enjoyed it but I've never found a convincing way to persuade people to read it. I read it on personal recommendation but that only works if it comes from someone you respect (as in my case).
- ahepp 3y agoAre filesystems the correct abstraction to build databases on? Isn’t a filesystem a database in a way? Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction? To say you can’t build an efficient database on top of S3 makes sense to me. S3 is already a certain kind of data-storing abstraction optimized for certain usages. If you try and build another data-storing abstraction optimized for incompatible usages on top of that, you are going to have a difficult time.
- d0gsg0w00f 3y agoIn my $dayjob as cloud architect I sometimes suggest S3 as an alternative to pulling massive JSON blobs from RDS Postgres/Redis etc. As long as their latency minimums are high enough there's no reason you can't.
- jandrewrogers 3y agoThe traditional POSIX filesystem is the wrong abstraction for a database, but not filesystems per se. All databases that care about performance and scalability implement their own filesystems, either directly against raw block devices or as an overlay on top of a POSIX filesystem that bypasses some of its limitations. The performance and scalability gains by doing so are not small. The issue with POSIX filesystems is that they are required to make a set of tradeoffs to support features a database engine doesn't need, to the significant detriment of scalability and performance in areas that databases care about a lot. For example, one such database filesystem I've used occasionally over the years, while a bit dated at this point, is designed such that you can have tens of millions of files in a single directory where you are creating and destroying tens of thousands of files every second, on upwards of a petabyte of storage. Very far from being POSIX compatible but you don't get anything like that type of scalability on POSIX. Object storage is far from ideal as database storage. The biggest issue, though, is the terrible storage bandwidth available in the cloud. It is a small fraction of what is available in a normal server and modern database engines are capable of fully exploiting a large JBOD of NVMe.
- calpaterson 3y ago> Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction? Oracle DB for a long time supported running on raw partitions which I think suggests that the answer is "not really". Snowflake (and I hear Clickhouse) can run on S3 which I think is more evidence against running on a filesystem. Not to mention the torrid time Postgres has had with fsync on linux.
- d-z-m 3y ago> S3 is a cloud filesystem, not an object-whatever. [...]I think the idea that S3 is really "Amazon Cloud Filesystem" is a bit of a load bearing fiction. Does anyone actually think this? I have never encountered anyone who has described S3 in these terms.
- teaearlgraycold 3y agoNot sure if the author is aware of EFS
- chubot 3y ago> Filesystem software, especially databases, can't be ported to Amazon S3 This seems mistaken. Porting databases that run on local disk to S3 seems like a good way to get a lashing from https://aphyr.com/ https://aphyr.com/ Can any databases do it correctly? If so, I doubt they work with the model of partial overwrites. They probably have to do something very custom, and either sacrifice a lot of tail latency, or their uptime is capped by the uptime of a single AWS availability zone. Doesn't seem like a great design. (copy of lobste.rs comment)
- est31 3y agoMy employer (Neon) offers Postgres databases that run on top of a couple of caching layers at the end of which there is S3: https://neon.tech/docs/introduction/architecture-overview https://neon.tech/docs/introduction/architecture-overview Directly exposing every write to S3 gives you the partial overwrite issues as described. But one can collect a bunch of traffic and push state to S3 once it reaches a threshold. Instead, a few writes in the postgres WAL are held outside of S3 in a replicated on-disk cache.
- chubot 3y agoThanks for the link. But I searched the docs for "durability" and got zero results. Before I use anything like this, I'd like to see what durability settings are used: https://www.postgresql.org/docs/current/non-durability.html https://www.postgresql.org/docs/current/non-durability.html Litestream documents the their data loss window, it seems like Neon should too: https://litestream.io/tips/ https://litestream.io/tips/ By default, Litestream will replicate new changes to an S3 replica every second. During this time where data has not yet been replicated, a catastrophic crash on your server will result in the loss of data in that time window. I also searched for "data loss" and got zero results -- this is important because Neon is almost certainly sacrificing durability for performance.
- yencabulator 3y agoNeon handles that by staging the WAL segments on 3x replicated Safekeeper nodes. Durability relies on not having all of those blow up at the same time. I'd expect it to be much safer than traditional Postgres replication mechanisms (with the trade-off having a comparatively large minimum node count; Neon really is built for multitenancy where that cost can be amortized across lots of databases).
- BirAdam 3y agoUnderneath the software, there’s still a filesystem with files. If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy SSDs. There’s just a bunch of stuff on top of that. It’s cool, but to say that there’s no filesystem is simply what the customer or middle person sees, not what is actually happening.
- seabrookmx 3y agoS3 actually uses a completely custom system[1] for writing bytes to disk. I haven't seen much in the way of details on the on-disk format but I certainly wouldn't assume it resembles a normal filesystem. [1]: https://aws.amazon.com/blogs/storage/how-automated-reasoning-helps-us-innovate-at-s3-scale/ https://aws.amazon.com/blogs/storage/how-automated-reasoning...
- aseipp 3y agoNo there isn't. AWS does not use the traditional filesystem layer to store data; that would be a massive mistake from a performance and reliability POV; the POSIX filesystem specification is notoriously vague about things like fsync consistency under particular scenarios, i.e. "do I need to fsync the parent directory before or after fsyncing the contents" for instance and has many bizarre performance cliffs if you aren't careful. At the scale AWS is at even a 10% performance cliff or performance delta would be worth clawing back if it meant removing the POSIX filesystem. Filesystems are not free; they incur "complexity" (that favorite bugbear everyone on HN loves to complain about) just as much as any other component in the stack does. > If you stand up an S3 instance with Ceph, Okay, but AWS does not run on Ceph. Even then, Ceph is an example that recommends the opposite. Nowadays they recommend solutions like the Bluestore OSD backend to store actual data directly on raw block devices, completely bypassing the filesystem layer -- for the exact same reasons I outlined above and many, many others (the actual metadata does use "BlueFS" which is a small FS shim, but this is mostly so that RocksDB can write directly to the block device too, next to the data segments, and BlueFS is in no way a real POSIX filesystem, it's just a shim for existing software). See "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution" written by the Ceph authors[1] about why they finally gave in and wrote Bluestore. The spoiler alert is they got rid of the filesystem precisely because "a filesystem with files" underneath, as you describe, was problematic and worked poorly in comparison (see the conclusion in Section 9.) Many places do use POSIX filesystems for various reasons, even at large scale, of course. [1] https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf
- somedudetbh 3y ago> Amazon S3 is the original cloud technology: it came out in 2006. "Objects" were popular at the time and S3 was labelled an "object store", but everyone really knows that S3 is for files. S3 Alternative theory: everyone who worked on this knew that it was not a filesystem and "object store" is a description intended to describe everything else pointed out in this post. "Objects were really popular" is about objects as software component that combines executable code with local state. None of the original S3 examples were about "hey you can serialize live objects to this store and then deserialize them into another live process!" It was all like "hey you know how you have all those static assets for your website..." "Objects" was used in this sense in databases at the time in the phrase "binary large object" or "blob". S3 was like "hey, stuff that doesn't fit in your database, you know...objects...this is a store for them." This is meant to describe precisely things like "listing is slow" because when S3 was designed, the launch usecases assumed an index of contents existed _somewhere else_, because, yeah, it's not a filesystem. it's an object store.
- senderista 3y agoYes, the author doesn't seem to realize that "object storage" is a term of art in storage systems that has nothing to do with OOP. https://en.wikipedia.org/wiki/Object_storage https://en.wikipedia.org/wiki/Object_storage
- paulddraper 3y agoYeah, I'm really worried the author is confusing OOP with an object store. To quote GCP: > Object storage is a data storage architecture for storing unstructured data, which sections data into units—objects—and stores them in a structurally flat data environment > https://cloud.google.com/learn/what-is-object-storage https://cloud.google.com/learn/what-is-object-storage That is (1) unstructured (2) flat organization (3) whole-item operations (read, write)
- tutfbhuf 3y agoS3 is obviously not a filesystem in the sense of a POSIX filesystem. And I would argue it is not a filesystem, even if we were to relax POSIX filesystem semantics (do not implement the full spec). But what is certainly possible is to span a filesystem on top of S3. It is basically possible to span a filesystem on anything that can store data. You can even go crazy for demonstration purposes and put a filesystem on top of YouTube (there are some tech demos for that on GitHub). I think a better question is whether there are any good filesystem implementations on top of S3. There are many attempts like s3fs-fuse[^1] or seaweedfs[^2], but I have not heard many stories about their use at scale from big companies. Just recently there was a post here about cunoFS[^3]. It is a startup that implements a POSIX-compliant (supports symlinks, hard links (emulated), UIDs & GIDs, permissions, random writes, etc.) filesystem on top of S3/AZ/GCP storage and claims to have really good performance. I think only time will tell if it works out in practice for companies to use S3 as a filesystem through fs implementations on top of S3. [^1]: https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse [^2]: https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs [^3]: https://news.ycombinator.com/item?id=39640307 https://news.ycombinator.com/item?id=39640307
- ein0p 3y agoA bit off topic but also related: I use Minio as a local "S3" to store datasets and model checkpoints for my garage compute. Minio, however, has a bunch of features that I simply don't need. I just want to be able copy to/from, list prefixes, and delete every now and then. I could use nfs I suppose, but that'd be a bit inconvenient since I also use Minio to store build deps (which Bazel then downloads), and I'd like to be able to comfortably build stuff on my laptop. In particular, one feature I do not need is the constant disk access than Minio does to "protect against bit rot" and whatever. That protection is already provided by periodic scrubs on my raidz6. So what's the current best (preferably statically linked) self-hosted, single-node option for minimal S3 like "thing" that just lets me CRUD the files and list them?
- paulddraper 3y agoFYI, Minio used to have a "File System" mode that did exactly this. But they deprecated it. (You can still use it, but it's not getting updates.)
- mr_toad 3y agoA web server?
- OnlyMortal 3y agoIt can be a file system. I’ve written my own FUSE that uses Rabin Chunking and stores the data (and meta) in S3. The C++/AWS SDK FUSE is connected to a Go SMB server that runs locally on my Mac and works with (local) TimeMachine. I use Wasabi for cost and speed reasons.
- jcims 3y agoIt seems like they’re moving away from this with S3 directory buckets and express zone.
- siliconc0w 3y agoI feel like a lot of applications can S3 but due to latency needs typically build a layer that sits in front that basically writes logs out to SSDs and then tiers to S3. If S3 offered a fast, reasonably priced Append() API that probably go a long way in capturing those use-cases.
- yellowapple 3y ago> Filesystem software, especially databases, can't be ported to Amazon S3 Except they can be. You don't need to overwrite the whole DB file on every INSERT/UPDATE/DELETE; those can be (and often are) stored in memory and periodically checkpointed. You might lose some writes if the process goes down between checkpoints, but for a lot of applications that's entirely acceptable. Indeed, for SQLite in particular there are tools like Litestream that support replication to and restoration from S3. Alternately, you could split the DB across multiple files, and then an INSERT/UPDATE/DELETE would only need to overwrite the files actually affected. This is already how server-based RDBMSs usually work.
- randombits0 3y agoIt can be misused as a filesystem. S3 wants objects? Here are some 512 or 4096 byte objects called clusters…
- SirOibaf 3y agoAt Hopsworks we built HopsFS-S3 to improve things like listing (becomes a partition pruned scan of a in-memory DB), atomic renames and added a block/object caching layer using NVMe drives. You can read the research paper here if you are curious: https://www.hopsworks.ai/research-papers/hopsfs-s3-extending-object-stores-with-posix-like-semantics https://www.hopsworks.ai/research-papers/hopsfs-s3-extending...
- bartwe 3y agoThis screwdriver makes for a horrible hammer.
- snotrockets 3y agoVery mild take: S3 is a very reliable, relatively cheap, high latency, key-value store. The reason I don't think about flies on a UNIX-derived systems as a key-value store (filename->file content), is that in such systems, we have many things that aren't really files that expose a file system interface regardless.