4 ms·

The Parrot Virtual File System

79 points by tempay 9y ago

joshumax 9y ago
This is an interesting method of providing a user space VFS later without the need of a kernel driver, like FUSE. However, that said, I wonder if it would still work in cases when a file is memory-mapped (via mmap() and friends) and modified directly in memory rather than direct FI/O calls. Would parrot still work in this case?
asimpletune 9y ago
Cool
brazzledazzle 9y ago
This is really neat. I was looking at FUSE for something last year and it never occurred to me to come from the other direction, so to speak.
eridius 9y ago
I can't resolve the domain. I've seen websites fall down due to load but I've never heard of a domain disappearing due to load. Edit: And now it's working.
- matte_black 9y ago
  Works on my machine.
- quickthrower2 9y ago
  on windows: ipconfig /flushdns
  - TeMPOraL 9y ago
    Sadly, that doesn't help on the bullshit Chrome is apparently pulling with name resolution internally.
Firerouge 9y ago
Would this be able to make a device acting as a USB mass storage device work as a network storage backed USB Drive? I don't see any one command from this guide that the utility could be run before to enable this. https://www.raspberrypi.org/magpi/pi-zero-w-smart-usb-flash-drive/ https://www.raspberrypi.org/magpi/pi-zero-w-smart-usb-flash-...
- oh_sigh 9y ago
  no, you would still need something like a file server serving read/write requests to the USB drive. Think of this program like automatically giving any program that has a "File -> Open" menu the ability to load files from FTP, or HTTP, or any other file serving protocol. You still need something serving up the data - it's just possible for you to treat it like any old file on your file system now.
rb2k_ 9y ago
I wonder how feasible this would be to trick Plex into transparently 'mounting' a remote server's media files. (e.g. a VPS that can burst CPU to re-encode data once/twice a day and another one with barely any CPU but large storage) I guess things like sshfs or rclone already do similar things, but it would be really neat to see the performance characteristics of Parrot vs other solutions.
metaobject 9y ago
Reminded me of the Parrot language prank: http://archive.oreilly.com/pub/a/oreilly/news/parrotstory_0401.html http://archive.oreilly.com/pub/a/oreilly/news/parrotstory_04... https://slashdot.org/story/17381 https://slashdot.org/story/17381
fnord77 9y ago
wouldn't using Dtrace be a lot more efficient than using ptrace?
- viraptor 9y ago
  What they're doing is trapping the execution on syscall and providing own implementation. That's not the kind of thing dtrace is good for. Also, this works on Linux, so there's no full dtrace.
sedachv 9y ago
The idea of extending file-related system calls to remote pathnames was present in a few systems in the 1980s: ITS, Lisp Machines, VMS, probably a bunch of others I am not aware of. You can see the vestiges of this in Common Lisp pathnames. Kent Pitman provides a good summary in a comp.lang.lisp post from 2002: https://groups.google.com/d/msg/comp.lang.lisp/UCeOl3sUlFQ/VkdFCoAXW4gJ https://groups.google.com/d/msg/comp.lang.lisp/UCeOl3sUlFQ/V... This approach is much more powerful than the Unix style of mounting. FUSE provides some degree of extensibility, but it is not composable. Pitman's post mentions the "chained hosts" of VMS. GNU Emacs TRAMP has a completely general and composable mechanism called "multi-hops": https://www.gnu.org/software/tramp/#Ad_002dhoc-multi_002dhops https://www.gnu.org/software/tramp/#Ad_002dhoc-multi_002dhop... Using explicit remote pathnames as opposed to mounting is a very powerful idea, and I am glad it is becoming practical today. I think it is possible to write something like a portable Parrot as a library and set of shadow include files you could use to recompile applications.
- sedachv 9y ago
  > This approach is much more powerful than the Unix style of mounting. It occurred to me I should provide a qualifier here. Single-tree Unix pathnames have one very powerful property that qualified schemes from the 1980s lack: self-similarity at every level of the directory hierarchy. This is why filesystem virtualization schemes like loop devices (CD and disk images, and things like EncFS), chroot, and overlays are possible and work so well. The 1980s schemes did not support these use cases and were more tedious to type out, and they rightly died out. But it is not either-or: TRAMP shows that you can have Unix pathnames for your local machine and qualified pathnames for when you need to specify where/how to access files.
- happyguy43 9y ago
  Just curious, in what way is FUSE not composable where TRAMP is?
  - sedachv 9y ago
    For example, if you want to use SSHFS to access files on a machine that is only reachable via a gateway, you would have to change your SSH configuration with the details of the gateway, or set up port forwarding: https://undeadly.org/cgi?action=article&sid=20070925181947 https://undeadly.org/cgi?action=article&sid=20070925181947 TRAMP provides a pathname syntax for this kind of connection: /ssh:bird@bastion|ssh:you@remotehost:/path So the steps to get to the remote files are part of the pathname, and you can compose several steps with |. With FUSE, or any kind of mounting, this information is not part of the pathname; next time you need to access somewhere convoluted you have to set up the proxying/forwarding again. You can also share the pathname. It is easy to copy-paste an SHH to sudo to Docker container multi-hop; setting up and tearing down the mounts to do the same thing would be fairly involved.
viraptor 9y ago
If you'd like to see code for a minimal version of something similar, I wrote one that's loaded using LD_PRELOAD rather than ptrace. Toy quality: https://github.com/viraptor/libremotec https://github.com/viraptor/libremotec
ec109685 9y ago
Is there any supper for running Fuse file systems using Parrot so that fuse can run completely from userland without any exotic privs needed (e.g. if you wanted to mount a squashfs file system for a process from within a docker container running inside Kubernetes).
torrance 9y ago
I was interested to read about CVMFS - is this something people have experience with outside of CERN?

79 points by tempay 9y ago

This is an interesting method of providing a user space VFS later without the need of a kernel driver, like FUSE. However, that said, I wonder if it would still work in cases when a file is memory-mapped (via mmap() and friends) and modified directly in memory rather than direct FI/O calls. Would parrot still work in this case?

asimpletune 9y ago

Cool

brazzledazzle 9y ago

This is really neat. I was looking at FUSE for something last year and it never occurred to me to come from the other direction, so to speak.

eridius 9y ago

I can't resolve the domain. I've seen websites fall down due to load but I've never heard of a domain disappearing due to load. Edit: And now it's working.

matte_black 9y ago
Works on my machine.
quickthrower2 9y ago
on windows: ipconfig /flushdns
- TeMPOraL 9y ago
  Sadly, that doesn't help on the bullshit Chrome is apparently pulling with name resolution internally.

Firerouge 9y ago

Would this be able to make a device acting as a USB mass storage device work as a network storage backed USB Drive? I don't see any one command from this guide that the utility could be run before to enable this. https://www.raspberrypi.org/magpi/pi-zero-w-smart-usb-flash-drive/ https://www.raspberrypi.org/magpi/pi-zero-w-smart-usb-flash-...

oh_sigh 9y ago
no, you would still need something like a file server serving read/write requests to the USB drive. Think of this program like automatically giving any program that has a "File -> Open" menu the ability to load files from FTP, or HTTP, or any other file serving protocol. You still need something serving up the data - it's just possible for you to treat it like any old file on your file system now.

rb2k_ 9y ago

I wonder how feasible this would be to trick Plex into transparently 'mounting' a remote server's media files. (e.g. a VPS that can burst CPU to re-encode data once/twice a day and another one with barely any CPU but large storage) I guess things like sshfs or rclone already do similar things, but it would be really neat to see the performance characteristics of Parrot vs other solutions.

metaobject 9y ago

Reminded me of the Parrot language prank: http://archive.oreilly.com/pub/a/oreilly/news/parrotstory_0401.html http://archive.oreilly.com/pub/a/oreilly/news/parrotstory_04... https://slashdot.org/story/17381 https://slashdot.org/story/17381

fnord77 9y ago

wouldn't using Dtrace be a lot more efficient than using ptrace?

viraptor 9y ago
What they're doing is trapping the execution on syscall and providing own implementation. That's not the kind of thing dtrace is good for. Also, this works on Linux, so there's no full dtrace.

sedachv 9y ago

The idea of extending file-related system calls to remote pathnames was present in a few systems in the 1980s: ITS, Lisp Machines, VMS, probably a bunch of others I am not aware of. You can see the vestiges of this in Common Lisp pathnames. Kent Pitman provides a good summary in a comp.lang.lisp post from 2002: https://groups.google.com/d/msg/comp.lang.lisp/UCeOl3sUlFQ/VkdFCoAXW4gJ https://groups.google.com/d/msg/comp.lang.lisp/UCeOl3sUlFQ/V... This approach is much more powerful than the Unix style of mounting. FUSE provides some degree of extensibility, but it is not composable. Pitman's post mentions the "chained hosts" of VMS. GNU Emacs TRAMP has a completely general and composable mechanism called "multi-hops": https://www.gnu.org/software/tramp/#Ad_002dhoc-multi_002dhops https://www.gnu.org/software/tramp/#Ad_002dhoc-multi_002dhop... Using explicit remote pathnames as opposed to mounting is a very powerful idea, and I am glad it is becoming practical today. I think it is possible to write something like a portable Parrot as a library and set of shadow include files you could use to recompile applications.

sedachv 9y ago
> This approach is much more powerful than the Unix style of mounting. It occurred to me I should provide a qualifier here. Single-tree Unix pathnames have one very powerful property that qualified schemes from the 1980s lack: self-similarity at every level of the directory hierarchy. This is why filesystem virtualization schemes like loop devices (CD and disk images, and things like EncFS), chroot, and overlays are possible and work so well. The 1980s schemes did not support these use cases and were more tedious to type out, and they rightly died out. But it is not either-or: TRAMP shows that you can have Unix pathnames for your local machine and qualified pathnames for when you need to specify where/how to access files.
happyguy43 9y ago
Just curious, in what way is FUSE not composable where TRAMP is?
- sedachv 9y ago
  For example, if you want to use SSHFS to access files on a machine that is only reachable via a gateway, you would have to change your SSH configuration with the details of the gateway, or set up port forwarding: https://undeadly.org/cgi?action=article&sid=20070925181947 https://undeadly.org/cgi?action=article&sid=20070925181947 TRAMP provides a pathname syntax for this kind of connection: /ssh:bird@bastion|ssh:you@remotehost:/path So the steps to get to the remote files are part of the pathname, and you can compose several steps with |. With FUSE, or any kind of mounting, this information is not part of the pathname; next time you need to access somewhere convoluted you have to set up the proxying/forwarding again. You can also share the pathname. It is easy to copy-paste an SHH to sudo to Docker container multi-hop; setting up and tearing down the mounts to do the same thing would be fairly involved.

viraptor 9y ago

If you'd like to see code for a minimal version of something similar, I wrote one that's loaded using LD_PRELOAD rather than ptrace. Toy quality: https://github.com/viraptor/libremotec https://github.com/viraptor/libremotec

ec109685 9y ago

Is there any supper for running Fuse file systems using Parrot so that fuse can run completely from userland without any exotic privs needed (e.g. if you wanted to mount a squashfs file system for a process from within a docker container running inside Kubernetes).

torrance 9y ago

I was interested to read about CVMFS - is this something people have experience with outside of CERN?