3 ms·

For those who are interested, the design was originally published here: (Chinese) https://www.high-flyer.cn/blog/3fs/ https://www.high-flyer.cn/blog/3fs/ This

For those who are interested, the design was originally published here: (Chinese) https://www.high-flyer.cn/blog/3fs/ https://www.high-flyer.cn/blog/3fs/ This file system has been developed and utilized by them for several years . Compared to the traditional file systems, it is more focused on model training that contains a lot of random reads. Read cache and prefetching are useless in this case. Therefore, they designed the file system without those features to improve the performance. I google translated some key parts here: 3FS is a special file system because it is almost only used in the scenario of batch reading sample data in the computing node during AI training, and accelerates model training through high-speed computing and storage interaction. This is a large-scale random reading task, and the read data will not be used again in a short time, so we cannot use the most important tool "read cache" to optimize file reading, and even advance reading is useless. Therefore, the implementation of 3FS is also quite different from other file systems. Specifically, as shown in the figure above, 3FS uses the Linux-based AIO and io_uring interfaces to complete sample reading, because in the 3FS scenario, File Cache has no effect at all, but will consume system memory in a way that is difficult for users to control, affecting the operation of subsequent tasks, so we turned off File Cache and only used Direct I/O mode to read data. But it should be noted that when reading in this way, the buffer pointer, offset and length all need to be aligned. If the user is allowed to do this alignment, additional memory copies will be generated, so we have done the alignment inside the file system, which not only optimizes performance but also facilitates users.

rvba 2y ago
Why is that a random read? Also is it truely random, or from seed? But if prng then they could cache right?
- kilburn 2y ago
  Random is prng. They still cannot cache though because they do many reading "passes" through the same data. If build a cache that gets hits on the first pass, then it won't work for the second and later passes.
vlovich123 2y ago
I hope they chose a multiple of 4096 for the alignment to minimize flash read amplification. QLC drives even use 16kib pages.
dekhn 2y ago
How critical is random reading of training data when assembling batches? Put another way: in my experience, supporting fast random reads is a challenging problem, while supporting high sequential reads is fairly straightforward. When is random access to a training set absolutely necessary for training a model?
- c4wrd 2y ago
  Imagine you're studying for a test where you are given an image and need to answer the correct class. To prepare, you're given a deck of flashcards with an image on the front and the class on the back. (Random) You shuffle the deck every time you go through it. You're forced to learn the images and their classifications without relying on any specific sequence, as the data has no signal from sequence order. (Fixed order) Every time you go through the deck, the images appear in the exact same order. Over time you may start to unconsciously memorize the sequence of flashcards, rather than the actual classification of each image. When it comes to actually training a model, if the batches are sampled sequentially from a dataset, it risks learning from correlations caused by the sequencing of the data, resulting in poor generalization. In contrast, when you sample the batches randomly, the model is biased and encouraged to learn features from the data itself rather than from any signals that arise from artifacts of the ordering.
  - dekhn 2y ago
    Why, then are so many successful models trained on multiple passes through sequential data? Note, I'm not naive in this field, but as an infra person, random reads make my life very difficult, and if they're not necessary, I'd rather not deal with them by having to switch to an approach that can't use readahead and other strategies for high throughput io.
    - Terretta 2y ago
      Why did they take time to invent an entire FS?
      - 2y ago

rvba 2y ago

Why is that a random read? Also is it truely random, or from seed? But if prng then they could cache right?

kilburn 2y ago
Random is prng. They still cannot cache though because they do many reading "passes" through the same data. If build a cache that gets hits on the first pass, then it won't work for the second and later passes.

vlovich123 2y ago

I hope they chose a multiple of 4096 for the alignment to minimize flash read amplification. QLC drives even use 16kib pages.

dekhn 2y ago

How critical is random reading of training data when assembling batches? Put another way: in my experience, supporting fast random reads is a challenging problem, while supporting high sequential reads is fairly straightforward. When is random access to a training set absolutely necessary for training a model?

c4wrd 2y ago
Imagine you're studying for a test where you are given an image and need to answer the correct class. To prepare, you're given a deck of flashcards with an image on the front and the class on the back. (Random) You shuffle the deck every time you go through it. You're forced to learn the images and their classifications without relying on any specific sequence, as the data has no signal from sequence order. (Fixed order) Every time you go through the deck, the images appear in the exact same order. Over time you may start to unconsciously memorize the sequence of flashcards, rather than the actual classification of each image. When it comes to actually training a model, if the batches are sampled sequentially from a dataset, it risks learning from correlations caused by the sequencing of the data, resulting in poor generalization. In contrast, when you sample the batches randomly, the model is biased and encouraged to learn features from the data itself rather than from any signals that arise from artifacts of the ordering.
- dekhn 2y ago
  Why, then are so many successful models trained on multiple passes through sequential data? Note, I'm not naive in this field, but as an infra person, random reads make my life very difficult, and if they're not necessary, I'd rather not deal with them by having to switch to an approach that can't use readahead and other strategies for high throughput io.
  - Terretta 2y ago
    Why did they take time to invent an entire FS?
    - 2y ago