3 ms·
I totally agree that we're continuing to see a trend of applications which are designed to work directly on S3. However, like the S3 protocol, I think that the
by huntaub 2y ago
I totally agree that we're continuing to see a trend of applications which are designed to work directly on S3.
However, like the S3 protocol, I think that the file protocol is cemented in time as something that we will be using 100 years from now. For example, most AI applications do still download data sets to local file system devices to actually load and use, this is why you see a lot of HPC workloads use things like Lustre. Postgres, SQLite, etc all use file system semantics to operate the database.
I totally respect folks who rewrite their applications to work directly with S3, but as you point out, it comes with a different set of challenges (around caching and chunking).