ylow

9 ms·

31.
by ylow 3y ago
Another possibility I am interested in is SMB3. (SMB4 is rather nasty). Similarly, supported by most operating systems, and does not look too horrifying to implement (The protocol documentation is really really hard to read though). And has
32.
by ylow 3y ago
Very nice. The mount protocol is pretty minimal, though different OS clients seem to use it differently. The NFSv3 chattiness is definitely a thing, but since its all localhost its not "too bad". v4 will be nice and is something I
33.
by ylow 3y ago
Interesting. I do not know much about webdav. Great to learn something new! Will take a look.
34.
by ylow 3y ago
Doing some quick googling, I see https://lwn.net/Articles/353831/ which sounds like an xattr protocol extension to nfsv3. I do not know the extent to which it is supported though...
35.
by ylow 3y ago
Exactly. You build the server-side (as a FUSE alternative), and just use the OS's NFS client to mount it.
36.
by ylow 3y ago
This NFS Server on one side, and the OS's builtin NFS client on the other.
37.
by ylow 3y ago
I have not looked at WebDAV closely. I was looking for something that is already supported in all the major operating systems without additional libraries. I don't believe WebDAV is commonly supported out of the box?
38.
by ylow 3y ago
I tested by uh... building a "mirrorfs" and compiling this project in it :-) (cargo does stress it pretty hard). I have not looked for POSIX compliance testers. Is there one you can point me to? NFSv4 did throw out a lot of the le
39.
by ylow 3y ago
NFSv4 is quite a bit more complicated and is also stateful which makes it a bit more challenging to implement. NFSv3 is a good starting point, and is still supported on all the major platforms. Certainly, extending with a v4 implementation
40.
by ylow 3y ago
Hi! Author here. Happy to answer any questions!
41.
NFS > FUSE: Why We Built Our Own NFS Server in Rust(about.xethub.com)
141 points by ylow 3y ago|91 comments
42.
by ylow 3y ago
A PhD is one of the hardest, yet most rewarding time of my life. Its perhaps the only period where I have the opportunity to do nothing else, but "think"; where I can spend many months just reading papers, learning and attacking w
43.
Git xet mount: Rust FS mount implementation for large datasets (using NFSv3)(about.xethub.com)
14 points by ylow 3y ago|2 comments
44.
by ylow 3y ago
Downloading large datasets can be a pain, especially when you only need a fraction of the data. So we built git xet mount, a file system mount solution using Rust and NFSv3. Using NFSv3 means that there is no FUSE driver needed, and works o
45.
by ylow 4y ago
Even better :-) We have a paper. Its on the way.
46.
by ylow 4y ago
CEO/Cofounder here. We are file format agnostic and will happily take everything. Not too familiar with the needs around pbix, but please do try it out and let us know what you think!
47.
by ylow 4y ago
We are significantly faster? :-) Also, block-level dedupe, scalability, perf, visualization, mounting, etc.
48.
by ylow 4y ago
Exactly for the giant repo use case, we have a mount feature that will let you get a filesystem mount of any repo at any commit very very quickly.
49.
by ylow 4y ago
We are file format agnostic and you should be able to put anything in the repo. We have special support for CSV files for visualizations. Sorry for the UI perf... there are a lot of optimizations we need to work on.
50.
by ylow 4y ago
Entirely in house. In Rust!
51.
by ylow 4y ago
The simplest really is to chunk row-wise so adding columns will unfortunately rewrite all the chunks. If you have a parquet file, adding columns will be cheap.
52.
by ylow 4y ago
Nope. No kernel driver needed :-) We wrote an localhost NFS server.
53.
by ylow 4y ago
CEO/Cofounder here. Thanks! Agreed, we think data versioning is an important problem and we are at related, but opposite parts of the space. (BTW we really wanted gitfordata.com. Or perhaps we can split the domain? OLTP goes here, Unst
54.
by ylow 4y ago
By the way, our mount mechanism has one very interesting novelty. It does not depend on a FUSE driver on Mac :-)
55.
by ylow 4y ago
We have found pointer files to be surprisingly efficient as long as you don't have to actually materialize those files. (Git's internals actually very well done). Our mount mechanism does avoid materializing pointer files which
56.
by ylow 4y ago
Yeah... The compression does defeat the chunking (your mileage may vary. We do a small amount of dedupe in some experiments but never quite investigated it in detail.). That said, we have experimental preprocessors / chunkers that are
57.
by ylow 4y ago
Today we just have a variation of FastCDC in production, but we have alternate experimental chunkers for some file formats (ex: a heuristic chunker for CSV files that will enable almost free subsampling). Hope to have them enter production
58.
by ylow 4y ago
CEO/Cofounder here! Content defined chunking. Specifically a variation of FastCDC. We have a paper coming out soon with a lot more technical details.
59.
by ylow 4y ago
Indeed, there is a lot of pain if you actually try to store large binary data in git. But we managed to make that work! So a question worth asking is how might things change IF you can store large binary data in git??
60.
by ylow 4y ago
Cofounder/CEO here! I think it less about "versioning", but the ability to modify with confidence knowing that you can go back in time anytime. (Minor clarification: we are not quite storing diffs; holding snapshots just like

More ›