8 ms·
An intro to DeepSeek's distributed file system
- vFunct 2y agoCan we replicate this with ZFS drives distributed across multiple machines?
- eatonphil 2y agoAs far as I'm aware ZFS does not scale out. https://unix.stackexchange.com/a/99218 https://unix.stackexchange.com/a/99218
- db48x 2y agoYea, it wasn’t designed to scale that way. In principle you could use fiberchannel to connect a really large number (2²⁴ iirc) of disks to a single server and then create a single ZFS pool using all of them. This lets you scale the _storage_ as high as you want. But that still limits you to however many requests per second that your single server can handle. You can scale that pretty high too, but probably not by a factor of 2²⁴.
- jack_pp 2y agoI don't have direct experience with distributed file systems but it so happens I did a tiny bit of research in the past month and.. there are quite a few open source ones available. Would've been nice for the authors to explain why the already existing solutions didn't work for them.
- dboreham 2y agoThey have an HFT background so probably it was developed long ago for that workload (which tends to be outside the design envelope for off the shelf solutions).
- londons_explore 2y agoThis seems like a pretty complex setup with lots of features which aren't obviously important for a deep learning workload. Presumably the key necessary features are PB's worth of storage, read/write parallelism (can be achieved by splitting a 1PB file into say 10,000 100GB shards, and then having each client only read the necessary shards), and redundancy Consistency is hard to achieve and seems to have no use here - your programmers can manage to make sure different processes are writing to different filenames.
- sungam 2y agoI wonder whether it may have been originally developed for the quantitive hedge fund
- huntaub 2y agoYes, I think this is probably true. I've worked with a lot of different hedge funds who have a similar problem -- lots of shared data that they need in a file system so that they can do backtesting of strategies with things like kdb+. Generally, these folks are using NFS which is kind of a pain -- especially for scaleability -- so building your own for that specific use case (which happens to have a similar usage pattern for AI training) makes a lot of sense.
- ammo1662 2y agoYes, as I mentioned in other comments. The 3FS was designed in 2019. You can check [0] (Chinese) [0] https://www.high-flyer.cn/blog/3fs/ https://www.high-flyer.cn/blog/3fs/
- threeseed 2y ago> Consistency is hard to achieve and seems to have no use here Famous last words. It is very common when operating data platforms like this at this scale to lose a lot of nodes over time especially in the cloud. So having a robust consistency/replication mechanism is vital to making sure your training job doesn't need to be restarted just because the block it needs isn't on the particular node.
- jamesblonde 2y agoArchitecturally, it is a scale-out metadata filesystem [ref]. Other related distributed file systems are Collosus, Tectonic (Meta), ADLSv2 (Microsoft), HopsFS (Hopsworks), and I think PolarFS (Alibaba). They all use different distributed row-oriented DBs for storing metadata. S3FS uses FoundationDB, Collosus uses BigTable, Tectonic some KV store, ADLSv2 (not sure), HopsFS uses RonDB. What's important here with S3FS is that it supports (1) a fuse client - it just makes life so much easiter - and (2) NVMe storage - so that training pipelines aren't Disk I/O bound (you can't always split files small enough and parallel reading/writing enough to a S3 object store). Disclaimer: i worked on HopsFS. HopsFS adds tiered storage - NVMe for recent data and S3 for archival. [ref]: https://www.hopsworks.ai/post/scalable-metadata-the-new-breed-of-file-systems https://www.hopsworks.ai/post/scalable-metadata-the-new-bree...
- nickfixit 2y agoI've been using JuiceFS since the start for my AI stacks. Similar and used postgresql for the meta.
- jamesblonde 2y agoJuiceFS is very good. I didn't have it as a scaleout metadata FS - it supports lots of DBs (single host and distributed DBs).
- threeseed 2y agoTiered storage and FUSE has existed with Alluxio for years. And NVMe optimisations e.g. NVMeoF in OpenEBS (Mayastor). None of it is particularly ground breaking just a lot of pieces brought together.
- jamesblonde 2y agoThe difference is scale-out metadata in the filesystem. Alluxio uses Raft, i believe, for metadata - that has to fit on a single server.
- 2y ago
- randomtoast 2y agoWhy not use CephFS instead? It has been thoroughly tested in real-world scenarios and has demonstrated reliability even at petabyte scale. As an open-source solution, it can run on the fastest NVMe storage, achieving very high IOPS with 10 Gigabit or faster interconnect. I think their "Other distributed filesystem" section does not answer this question.
- tempest_ 2y agoWe have a couple ceph clusters. If my systems guys are telling me the truth is it a real time sink to run and can require an awful lot of babysitting at times.
- huntaub 2y agoIMO this is the problem with all storage clusters that you run yourself, not just Ceph. Ultimately, keeping data alive through instance failures is just a lot of maintenance that needs to happen (even with automation).
- _joel 2y agoI admin'd a cluster about 10 years back and it was 'ok' then, around bluestore. One issue was definitely my mistake but it wasn't all that bad.
- elashri 2y agoCERN use CephFS with ~50PB for different applications and they are happy with it.
- dfc 2y agoI thought they used ceph too. But I started looking around and it seems like they have switched to CernVM-FS and in house solution. I'm not sure what changed.
- elashri 2y agoThey didn't switch, they use both for different needs. EOS (CVMFS) is used mainly for physics data storage and user data. Ceph is used for many other things like infrastructure, selfhosted apps..etc.
- huntaub 2y agoI think that the author is spot on, there are a couple of dimensions in which you should evaluate these systems: theoretical limits, efficiency, and practical limits. From a theoretical point of view, like others have pointed out, parallel distributed file systems have existed for years -- most notably Lustre. These file systems should be capable of scaling out their storage and throughput to, effectively, infinity -- if you add enough nodes. Then you start to ask, well how much storage and throughput can I get with a node that has X TiB of disk -- starting to evaluate efficiency. I ran some calculations (against FSx for Lustre, since I'm an AWS guy) -- and it appears that you can run 3FS in AWS for about 12-30% cheaper depending on the replication factors that you choose against FSxL (which is good, but not great considering that you're now managing the cluster yourself). Then, the third thing you start to ask is anecdotally, are people able to actually configure these file systems into the size of deployment that I want (which is where you hear things like "oh it's hard to get Ceph to 1 TiB/s") -- and that remains to be seen from something like 3FS. Ultimately, I obviously believe that storage and data are really important keys to how these AI companies operate -- so it makes sense that DeepSeek would build something like this in-house to get the properties that they're looking for. My hope is that we, at Archil, can find a better set of defaults that work for most people without needing to manage a giant cluster or even worry about how things are replicated.
- jamesblonde 2y agoMaybe AWS could start by making fast NVMes available - without requiring multi TB disks just to get 1 GB/s. S3FS experiments were run on 14 GB/s NVMe disks - an order of magnitude higher throughput than anything available in AWS today. SSDs Have Become Ridiculously Fast, Except in the Cloud: https://news.ycombinator.com/item?id=39443679 https://news.ycombinator.com/item?id=39443679
- kridsdale1 2y agoOn my home LAN connected with 10gbps fiber between MacBook Pro and server, 10 feet away, I get about 1.5gbps vs the non-network speed of the disks of ~50 gbps. (Bits, not bytes) I worked this out to the macOS SMB implementation really sucking. I set up a NFS driver and it got about twice as fast but it’s annoying to mount and use, and still far from the disk’s capabilities. I’ve mostly resorted to abandoning the network (after large expense) and using Thunderbolt and physical transport of the drives.
- stapedium 2y agoI’m just a small business & homelab guy, so I’ll probably never use one of these big distributed file systems. But when people start talking petabytes, I always wonder if these things are actually backed up and what you use for backup and recovery?
- huntaub 2y agoWell, for active data, the idea is that the replication within the system is enough to keep the data alive from instance failure (assuming that you're doing the proper maintenance and repairing hosts pretty quickly after failure). Backup and recovery, in that case, is used more for saving yourself against fat-fingering an "rm -rf /" type command. Since it's just a file system, you should be able to use any backup and recovery solution that works with regular files.
- shermantanktop 2y agoBackup and recovery is a process with a non-zero failure rate. The more you test it, the lower the rate, but there is always a failure mode. With these systems, the runtime guarantees of data integrity are very high and the failure rate is very low. And best of all, failure is constantly happening as a normal activity in the system. So once you have data integrity guarantees that are better in you runtime system than your backup process, why backup? There are still reasons, but they become more specific to the data being stored and less important as a general datastore feature.
- Eikon 2y ago> why backup? Because of mistakes and malicious actors...
- overfeed 2y ago...and the "Disaster" in "Disaster recovery" may have been localized and extensive (fire, flooding, major earthquake, brownouts due to a faulty transformer, building collapse, a solvent tanker driving through the wall into the server room, a massive sinkhole, etc)
- mertleee 2y agoWhat are the odds 3fs is backdoored?
- huntaub 2y agoI think that's a pretty odd concern to have. What would you imagine that looks like? If you're running these kinds of things securely, you should be locking down the network access to the hosts (they don't need outbound internet access, and they shouldn't need inbound access from anything except your application).
- xpe 2y ago> I think that's a pretty odd concern to have. Thinking about security risks is an odd concern to have?
- huntaub 2y agoI think that worrying that a self-hosted file system has a backdoor to exfiltrate data is an odd concern. Security concerns are (obviously) normal, but you should not be exposing these kinds of services to the public internet (or giving them access to the public internet), eliminating the concern that it's giving your data away.
- antonvs 2y agoBackdoors in self-hosted systems are one of the major state-level attack vectors, and there are many known examples of this. The idea that "you should not be exposing these kinds of services to the public internet (or giving them access to the public internet)" is naive. Aside from the fact that this requires every user to show that level of diligence - a completely unrealistic expectation - if a system is backdoored, it can also have means of exfiltrating data beyond just sending it out on the public internet. And then there's the obvious point that if you suspect a device is compromised, it's completely irresponsible to use it anyway and assume that you're going to be able to prevent unauthorized access. As some examples, there are documented concerns about Huawei/ZTE routers, which have been banned in Australia, New Zealand, Japan, Taiwan and the US for that reason. Unauthorized third-party code was found in Juniper Networks firewalls. Fortinet had hardcoded admin credentials in its firewalls and VPNs - probably a self-inflicted mistake, but still useful to attackers. Similarly, Western Digital NAS devices had a hardcoded backdoor account. D-Link routers had such a backdoor in their device web interface. There are many more examples like this. Snowden revealed some of the US government activities in this area. The US, Russia, China, North Korea and other countries have all been involved in attacks involving BIOS/UEFI firmware, router firmware, NAS, and manufacturing supply chains. Covert exfiltration has been involved in many of these cases, using techniques other than transmitting over the internet. And of course there was the recent (reported late 2024 Salt Typhoon attack by China on US and other Western telecom networks, which relied on these kinds of techniques, and gained access to large amounts of data, including audio and text of targeted people.
- robinhoodexe 2y agoI’m interested in how it is compared to seaweedfs[1], which we use for storing weather data (about 3 PB) for ML training. [1] https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs
- huntaub 2y agoMy guess is going to be that performance is pretty comparable, but it looks like Seaweed contains a lot more management features (such as tiered storage) which you may or may not be using.
- rfoo 2y agoIMO they serve similar at a glance, but actually very different use cases. SeaweedFS is more about amazing small object read performance because you effectively have no metadata to query to read an object. You just distribute volume id, file id (+cookie) to clients. 3FS is less extreme in this, supports actual POSIX interface, and isn't particularly good at how fast you can open() files. On the other hand, it shards files into smaller (e.g. 512KiB) chunks, demands RDMA NICs and makes reading randomly from large files scary fast [0]. If your dataset is immutable you can emulate what SeaweedFS does, but if it isn't then SeaweedFS is better. [0] By scary fast I mean being able to completely saturate 12 PCIe Gen 4 NVMe SSD at 4K random reads on a single storage server and you can horizontally scale that.
- jszymborski 2y agoI wonder how close to something like 3FS you can get by mounting SeaweedFS with S3FS, which mounts using FUSE. https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse
- rfoo 2y agoI'd estimate that there would be two orders of magnitude of difference in 4K random IOPS. If not three.
- 2y ago
- urlshortuk1234 2y ago[dead]
- seethishat 2y agoHow easy is it to disable DeepSeek's distributed FS? Say for example a US college has been authorized to use DeepSeek for research, but must ensure no data leaves the local research cluster filesystem? Edit: I am a DeepSeek newbie BTW, so if this question makes no sense at all, that's why ;)
- ikeashark 2y agoI might need more clarification, but if one is paranoid or is dealing with this sensitive of information the DeepSeek model and 3FS are able to be deployed locally offline and not connected to the internet.
- seethishat 2y agoThank you, that answers my question.
- ajcp 2y agoDeepSeek is a company. This article is about a distributed file system they have developed. It is a separate, unrelated piece software from their open-weight models(DeepSeek-R1, DeepSeek-V3, etc). In your example it is likely the US college has been authorized to use a DeepSeek model for research, not the DeepSeek 3FS distributed file system.
- snthpy 2y agoSimilar to the SeaweedFS question in sibling comment, how does this compare to JuiceFS? In particular for my homelab setup I'm planning to run JuiceFS on top of S3 Garage. I know garage is only replication without any erasure coding or sharding so it's not really comparable but I don't need all that and it looked at lot simpler to set up to me.
- huntaub 2y agoIt's a very different architecture. 3FS is storing everything on SSDs, which makes it extremely expensive but also low latency (think ~100-300us for access). JuiceFS stores data in S3, which is extremely cheap but very high latency (~20-60ms for access). The performance scalability should be pretty similar, if you're able to tolerate the latency numbers. Of course, they both use databases for the metadata layer, so assuming you pick the same one -- the metadata performance should also be similar.
- snthpy 2y agoThanks, that's good to know. For the most part I can live with the latency. For frequently or repeatedly accessed files I believe JuiceFS also has a caching layer to bring down the latency on those. My use cases are quite different from what DeepSeek requires. I was just curious whether I needed to investigate 3FS since it's the new kid on the block. What appeals to me about JuiceFS is that I can easily mount it into my containers for a distributed FS on a small cluster for some resilience. It looked simpler than Ceph or GlusterFS for that and it's backed by S3 so that I could swap out my S3 Garage setup for a cloud based S3 backend if required.
- dkkergoog 2y ago[flagged]
- nodesocket 2y agoDoes this get us closer to fully and performant distributed LLMs? Instead of spending ludicrous amounts on Nvidia GPUs, can just deploy commodity clusters of AMD EPYC servers with tons of memory, NVMe disks, and 40G or 100G networking which is vastly less expensive. Goodbye Nvidia moat!
- dang 2y agoRelated. Others? Understanding Smallpond and 3FS - https://news.ycombinator.com/item?id=43232410 https://news.ycombinator.com/item?id=43232410 - March 2025 (47 comments) Smallpond – A lightweight data processing framework built on DuckDB and 3FS - https://news.ycombinator.com/item?id=43200793 https://news.ycombinator.com/item?id=43200793 - Feb 2025 (73 comments) Fire-Flyer File System (3FS) - https://news.ycombinator.com/item?id=43200572 https://news.ycombinator.com/item?id=43200572 - Feb 2025 (101 comments)
- JSR_FDED 2y agoWhat happens when you want to add capacity to a distributed file system? Do you have to add the extra drives spread out across the nodes? Do you have to use certain size increments?
- dheera 2y agoWhat if mgmtd has a hardware failure? Is there a distributed file system where you can take out any 2 arbitrary nodes and it keeps running intact, waiting for the 2 failed nodes to be replaced?