11 ms·
Show HN: ZFS Implementation in Python
- lunixbochs 7y agoNote: this was implemented without referencing any ZFS source code and should not be subject to the CDDL.
- josteink 7y agoSo we port this python to something not slow, and all the kernel-people can shut up about ZFS being terrible ;)
- j88439h84 7y agoDont forget, pypy is fast.
- spullara 7y agoFaster that CPython doesn't mean it is fast.
- twa927 7y agoI was trying to speed up a log processing service running on PyPy by rewriting it in Java. I was surprised that the result was about twice slower (I know Java quite well and I didn't see obvious optimizations; most of the time was spent in GC). So it can be quite fast even in more absolute terms (VM languages), at least for some types of code.
- spullara 7y agoIf more than 1% of your time is in GC you are doing something very wrong.
- tigershark 7y agoI know that it’s asking a lot, but any chance that you can post a minimum reproducible sample? From what I know it is quite smelly...
- twa927 7y agoI don't have access to this codebase now but I'll try to write some benchmark.
- tigershark 7y agoIf you manage to do it would be awesome, otherwise thanks a lot anyway for the effort :) I’m just curious to understand why it happens because it’s exactly the opposite of what I would expect. The only explanation that comes to my mind is excessive gc as someone else already mentioned, but it would be interesting to see the original code.
- twa927 7y agoI started doing it but yes, it's too much effort to get two full benchmarks, I'm sorry :). But I think it went down to the inefficiency of String.split(): https://stackoverflow.com/questions/37007189/string-split-temporary-objects-and-garbage-collect https://stackoverflow.com/questions/37007189/string-split-te... and generally the Java's String built-in methods not being GC-friendly: https://stackoverflow.com/questions/20336459/garbage-friendly-alternative-to-substring https://stackoverflow.com/questions/20336459/garbage-friendl.... I'm guessing that when such parts can be coded in a non-VM environment (CPython/PyPy runtime) they can be made much faster, and Java to these days has the standard library coded in pure Java?
- auscompgeek 7y agoCPython and PyPy are both VMs, similarly to the JVM.
- twa927 7y ago
- deleted 7y ago[deleted]
- michaelmrose 7y agoThe fact that a singular implementation was better than java says less about the languages and more about the particular software.
- d0mine 7y agoI would have expected it: https://stackoverflow.com/questions/9371238/why-is-reading-lines-from-stdin-much-slower-in-c-than-python https://stackoverflow.com/questions/9371238/why-is-reading-l...
- int_19h 7y agoI had a binary parser written in Python that took around 30 seconds on typical input on CPython. PyPy took that down to about 10 seconds. Rewriting it in C# took it down to 200 ms.
- twa927 7y agoIf this was using a loop processing a single byte in an iteration I would expect a greater speedup on PyPy. I've seen 100x speedup in such cases.
- int_19h 7y agoNot single byte, but individual fields (float32/int32/string etc). Yes, I expected a much more significant speed-up as well. It's probably because a lot of that code was driven by reflection-type techniques. Curiously, IronPython did better than anything (but still slow). Haven't tried Jython. Compiling the whole thing with Cython was less effective than PyPy.
- twa927 7y agoYes, reflection voids many JIT paths in PyPy, AFAIK. Maybe it was worth rewriting the Python code to get rid of the reflection?
- cure 7y agoPyPy is faster than Python, yes. But Go, C and many other (compiled) languages are way faster than PyPy. Plus, if you use a language like Go or Rust then you avoid Python's GIL and you'll have much more reasonable memory usage. Best of all, deploying is a matter of copying a binary, rather than having to deal with the absolute disaster that is Python packaging.
- klyrs 7y agoFilesystem with a GIL, what could possibly go wrong? /s
- emmelaich 7y agoA lot less will go wrong than a filesystem without a GIL. GIL is for safety and correctness, not speed.
- y4mi 7y agoUh, no? Python's global interpreter lock was added for single threaded speed and c library integrations, which often can't be used multithreaded There was some talk about removing it recentlish to improve pythons multithreaded performance and Guido said something along the lines of > "I'll remove it as long as single threaded performance doesn't suffer" Which nobody succeeded in
- nine_k 7y agoGo? A GC'd language in kernel? (Well, yes, this has been done, from Lua to Haskell, but only experimentally.)
- tanilama 7y agoOnly when comparing to CPython
- fragmede 7y agopypy is fast, but even were this written in C, there's still the kernel-userland boundary to contend with.
- moonbug 7y agoFuse
- yjftsjthsd-h 7y agoIf you're going to run the file system in user space, there's no reason not to just use normal ZFS. The problem with ZFS licensing is only in combining CDDL+GPL in one unit. If you're working across the kernel/userspace boundary, there's already no problem. ZoL even already ships a fuse version that works fine.
- Conan_Kudo 7y ago> ZoL even already ships a fuse version that works fine. This is not true. A FUSE implementation is wanted though: https://github.com/zfsonlinux/zfs/issues/8 https://github.com/zfsonlinux/zfs/issues/8
- yjftsjthsd-h 7y agoOops; I didn't realize the existing fuse version wasn't built from ZoL. Thanks for pointing that out.
- GuB-42 7y agoMaybe plenty fast for most applications but a filesystem is not one of these IMHO, especially for something as naturally resource hungry as ZFS. A good filesystem implementation requires tight memory management and good control of what happens at the OS level. I am not saying it can't be be done in python, but it clearly isn't the right tool for the job. I meant that for a production implementation. Python is perfectly fine for a proof of concept, in fact, it may be better than jumping straight down to C. But keeping it for production is foolish IMHO.
- mehrdadn 7y agoI routinely fail to get speedup on PyPy. In fact I frequently get slowdowns. I imagine it's only fast if your code is slower than it needs to be to begin with.
- twa927 7y agoIt works well for tight loops processing much data, or heavy object-orientation (multiple levels of class hierarchies). It probably won't work well for regular Django webapps or scripts. Also, real-world Python numerical/AI code uses numpy/ML libs so there's not much to optimize in Python...
- aidenn0 7y agoI can Port from Python -> Common Lisp at a rate of ~100 LoC per hour, and that's a pretty friendly port (the yield expression is the only thing that can't be just done line-by-line; truthyness is the only real "gotcha" as there are a lot of "false" values in Python, but only 1.5 in Lisp[1]). 1: I say 1.5 because there is only one false value, but it has 2 idiomatic meanings: nil (equivalent to python's None) and the empty list.
- newnewpdro 7y agoWould the CDDL matter for a python implementation that will never become part of the kernel?
- dr0verride 7y agoIt's clear to me that the solution is to integrate python into the kernel.
- yjftsjthsd-h 7y agoI assume you mean this as a joke, but I would point out that at least one of the BSD family has gone and baked lua into their kernel. Granted, lua is rather meant for that kind of thing and python isn't, but it is entertaining to point out an interpreted language that has been stuck into a unix kernel:)
- riffraff 7y agothere was lua in linux too, with lunatik https://github.com/lunatik-ng/lunatik-ng https://github.com/lunatik-ng/lunatik-ng
- CaliforniaKarl 7y agoYou'd want a reverse-engineering lawyer, so be certain. But my (IANAL) guess is: If this is a proper reverse-engineered implementation, you could then convert _this_ implementation to C, and contribute _that_ into the kernel. Except, it seems this is BSD-licensed, so I'm not sure how that would work in the kernel (which is GPLv2).
- lunixbochs 7y agoBSD is a subset of GPL's restrictions, so you can include BSD-licensed code in a GPL work.
- loeg 7y ago
- lelf 7y agoBad title. It’s not a ZFS implementation, so hold your horses.
- AHTERIX5000 7y agoWhat do you mean? Can't you mount & read ZFS filesystems with this one?
- Annatar 7y agoCDDL isn't a fascist license like GPL, so there is no penalty for looking at the source code licensed under CDDL. Let's leave the GPL ideology out of this.
- AlexanderDhoore 7y agoWhat?! How?! Why?! This is the greatest thing ever. I wish I could just write code for the fun of it. Every time I wonder whether people will use it and give up before I even get started.
- mirceal 7y agowhy not? writing code can fall into one of a few bucket. one of them is play.
- anon4242 7y agoExactly! Write code that you find interesting and/or need for something and then share it. If someone uses your stuff, then great, if not, at least you've become a slightly better programmer! It's a win-win!
- ehsankia 7y agoAbsolutely, some of the most fun I've had coding was reverse engineering/implementing known protocols. Although something this big may be a little overboard :)
- max0563 7y agoI had this problem too. I’ve been able to get over it by from coming up with a scenario, even if it’s completely fabricated, where what I am doing can be useful. I also make sure that I incorporate something new that I want to learn in the project. Whether it’s a language, library, whatever. Then I give myself a date I can quit. Normally it’s about two months. This makes me really consider whether I want to take something on because if I do I force myself to dedicate two months of time to it. If I enjoy it still at the end of two months then I continue otherwise I move on to another idea. At least in that time I because a little better at whatever I was trying to do. That’s the real goal anyway.
- js2 7y agoI wrote this to scratch my own itch but mostly just for the fun of it and some people ended up using it. https://github.com/jaysoffian/eap_proxy https://github.com/jaysoffian/eap_proxy This was a really simple project but tickled all my fancies: Python, low-level, networking, reverse engineering, system administration. Just do it! Who cares if people use it? Alternatively, contribute something to some open source project you use. I’ve done that too. Just small stuff here and there but that’ll guarantee someone uses your code if that’s what’s important to you. It only takes 39 commits to get on this page: https://github.com/git/git/graphs/contributors https://github.com/git/git/graphs/contributors :-)
- ivanbakel 7y agoPierre Menard, author of The Filesystem. But I'm surprised this is possible without a specification - how can you test a filesystem through hexdumps? The effects of some operations are going to pretty far-reaching, surely?
- randrus 7y agoMight be interesting/useful to aim for zfs send/receive compatibility :) And thanks for the Borges callout.
- aerovistae 7y agoOne of my favorite short stories, and nobody else has ever read it. So glad to see someone else reference it.
- cryptonector 7y agoThere are lots of blog posts, lots of docs. There's ZFS code in GRUB that is GPL, etc.
- lelf 7y agoNo ARC/L2ARC? Edit: of course not. This is actually just it’s just a ZFS user-facing ”front-end”, not a ZFS implementation.
- lunixbochs 7y agoIt's capable of doing IO against a real ZFS array without any other code. ARC is an implementation detail and not necessary for correctness. If you removed ARC from ZoL it would still work, just slower. ARC is far from the most interesting milestone for a reimplementation effort because an ARC implementation doesn't need to be anything like the Sun version internally, as long as it offers similar performance. This project is cool not because you're going to run the Python in your kernel today, but because someone can use it as a documented reference implementation of all of the data structures and transactions that is not covered by the CDDL, so another implementation based on this can live in the Linux kernel without problem.
- chasil 7y agoIt would certainly be wonderful if this led to a Linux kernel module that was free of Oracle.
- hnlmorg 7y agoIf the GPLv2 GRUB ZFS code[1] wasn't enough to get someone started then I doubt this will make any different in porting ZFS to GPL given there would be more work involved in turning this into a usable kernel driver. Not taking anything away from the work that the author has done though. It's a nice project. I just think a little pragmatism is needed before we get carried away with the ZFS GPL comments. [1] https://blogs.oracle.com/solaris/zfs-under-gplv2-already-exists-no-kidding-v2 https://blogs.oracle.com/solaris/zfs-under-gplv2-already-exi...
- lelf 7y agoYou don’t need CDDL to use ZFS. Which is all the library does. It does not implement ZFS.
- foxhop 7y agoHey alcari! I know you from ICV - we used to hang out online on the forums and IRC. Nice work on this project, I'm looking forward to diving into the codebase!
- rashkov 7y agoJust curious, what is / was ICV?
- foxhop 7y agoIt was / is an community of people who formed around a book called 1337 h4x0r h4ndb00k by tapeworm. ICV was the name of the forum for the book, now defunct (icodeviruses.com) https://www.amazon.com/1337-h4x0r-h4ndb00k-tapeworm/dp/0672327279 https://www.amazon.com/1337-h4x0r-h4ndb00k-tapeworm/dp/06723...
- 4oo4 7y agoThis is awesome! Do you plan on blogging anything about how you went about reverse-engineering?
- likehsvilius 7y agohttps://coincircle.com/l/MGkjIpNb~j https://coincircle.com/l/MGkjIpNb~j
- likehsvilius 7y agohttps://coincircle.com/l/MGkjIpNb~j https://coincircle.com/l/MGkjIpNb~j best earnings
- PaulHoule 7y agoi love userspace implementations of filesystems. note that the issues are entirely different from those with a kernel implementation since you aren't having to think about page cache et all.
- burmecia 7y agoThere is an userspace implementations of filesystems ZboxFS: https://github.com/zboxfs/zbox https://github.com/zboxfs/zbox.
- adamnemecek 7y agoThere's also TFS a ZFS inspired FS in Rust https://github.com/redox-os/tfs https://github.com/redox-os/tfs
- gaze 7y agoHell yeah, dude. This is awesome.
- garmaine 7y agoSimultaneously fucking awesome (that you pulled it off at all), and fucking useless (performance...). Thanks for sharing though. Maybe could be useful in making a suite of zfs inspection tools? Is the OP here? How difficult would it be creat a zfs reshaping tool, allowing for the offline expansion of a vdev?
- mfsch 7y agoDoes someone know whether it would be legal for someone to go through the ZFS code and write a specification of the features this author hasn’t figured out yet? I.e. could someone write a detailed description of the missing functionality that doesn’t include any details about the implementation so other people can implement it in non-CDDL code?
- ummonk 7y agoEdit: yeah that is how you avoid copyright infringement https://en.wikipedia.org/wiki/Clean_room_design https://en.wikipedia.org/wiki/Clean_room_design Original comment: I could swear this was actually the standard practice for writing an implementation of an unknown file format or interface without infringing on copyright. But I don't remember the term for it.
- atomicwrites 7y agoThat's called a clean room implementation and was the standard way to make x-compatible products (like for example, the bios on an IBM PC clone). Not sure what the current legal standing of that method is. EDIT: Ninjad because I left the reply in a tab without posting.
- zymhan 7y agoReverse engineering is legal in the US, but you had better have detailed records proving no one who knew the insides of the original product ever influenced the clone. And be prepared to explain that in court.
- Someone 7y agoThe on-disk format is available (http://www.giis.co.in/Zfs_ondiskformat.pdf http://www.giis.co.in/Zfs_ondiskformat.pdf) That PDF says ”Unless otherwise licensed, use of this software is authorized pursuant to the terms of the license found at: http://developers.sun.com/berkeley_license.html”* http://developers.sun.com/berkeley_license.html”*. That link is broken, but it seems that’s Berkeley license (whatever that means for a specification, and for which variant?) According to http://open-zfs.org/wiki/Developer_resources http://open-zfs.org/wiki/Developer_resources, its outdated, but still useful. I think I would use that, rather than spend months diffing disk images.
- RantyDave 7y agoThis is all kinds of funny. I'm awash with awe and admiration.
- sara7262 7y agoReality about terrorism Must watch this https://youtu.be/l_C_T1NCvDw https://youtu.be/l_C_T1NCvDw See the friendship between cat and squirrel That's so cute see how much they are taking care of each others http://bit.ly/2VdHs4E http://bit.ly/2VdHs4E Friendship between cat and dog see in video how much they are teasing each others also they are loving http://bit.ly/2WoRU66 http://bit.ly/2WoRU66
- Emily1636 7y agoReality about terrorism Must watch this https://youtu.be/l_C_T1NCvDw https://youtu.be/l_C_T1NCvDw See the friendship between cat and squirrel That's so cute see how much they are taking care of each others http://bit.ly/2VdHs4E http://bit.ly/2VdHs4E Friendship between cat and dog see in video how much they are teasing each others also they are loving http://bit.ly/2WoRU66 http://bit.ly/2WoRU66
- jchwnrts 7y agowhy don't you tell the truth about were u put the python asshole
- Annatar 7y agoSomeone has way too much free time on their hands. Newsflash: ZFS already comes with such a program: it's called zdb and it's used in production environments and during development for ZFS debugging.