3 ms·
Tools like LucidLink and Weka go a way to making S3 even more of a “file system”. They break files into smaller chunks (S3 objects) which helps with partial wri
by type_Ben_struct 3y ago
Tools like LucidLink and Weka go a way to making S3 even more of a “file system”. They break files into smaller chunks (S3 objects) which helps with partial writes, reads and performance. Alongside tiering of data from S3 to disk when needed for performance.
- hnlmorg 3y agoI don’t know a whole lot about LucidLink but Weka basically uses S3 as a dataplane for their own file system.
- rwmj 3y agoSomeone contributed an nbdkit S3 plugin which basically works the way you described. It uses numbered S3 chunks using the pattern "key/%16x", allowing the virtual disk to be updated. (https://libguestfs.org/nbdkit-S3-plugin.1.html https://libguestfs.org/nbdkit-S3-plugin.1.html https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/S3 https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/S3)
- cuno 3y agoThe problem with these approaches is that the data is scrambled on the backend, so you can't access the files directly from S3 anymore. Instead you need an S3 gateway to convert from scrambled S3 to unscrambled S3. They rely on a separate database to reassemble the pieces back together again.