5 ms·

I am in the process of compiling this with emscripten to javascript+wasm. The integration between your block storage implementation and FatFs is straightforwar

I am in the process of compiling this with emscripten to javascript+wasm. The integration between your block storage implementation and FatFs is straightforward other than ioctl, which is mysterious to me and will take me a long time to figure out what all my responibilities are. This is my first time playing with emscripten and sharing buffers between wasm and javascript is awkward.

loeg 4y ago
http://elm-chan.org/fsw/ff/doc/dioctl.html http://elm-chan.org/fsw/ff/doc/dioctl.html describes the expected ioctls to implement and their behavior. The mandatory ioctls seem pretty straightforward. CTRL_SYNC will be a nop on a browser-based storage, GET_BLOCK_SIZE should return 1 (unknown), and CTRL_TRIM looks like it could be nop (or you could zero the range to more closely match real devices).
- EdSharkey 4y ago
  I didn't study in my operating systems class! If only I could shake dumbass young version of me and have them work harder at school! I know what you are saying is true and I should have no fear. I just need to write lots of tests and log like a mofo until I get all the ioctl commands it needs built out on the javascript side.
tremon 4y ago
Sorry, but I can't help wonder: why? What's the value of having direct device access in Javascript? Are we expected to give the browser raw block device access now? Maybe we could implement a FAT layer on top of S3, but I really don't see the value in reimplementing block-level storage on top of object storage.
- kaoD 4y ago
  Might be useful for non-browser VMs? E.g. a microcontroller running WASM.
  - marcosdumay 4y ago
    Even tough yes, that's a good answer to the GP... Are people expecting WASM machines to be a thing? And with a different fate from the Java machines? (I don't see how they are different enough to change that fate.)
    - topspin 4y ago
      > Are people expecting WASM machines to be a thing? People are expecting to embed WASM[1]. A generic FAT can provide a lightweight storage abstraction for such work. The WAMR AOT compiled runtime is 50K; 2.5-10% of the flash memory on a STM32F4, for example. Obviously not appropriate for the smallest extant MCUs. Otherwise it's a viable choice. [1] https://bytecodealliance.github.io/wamr.dev/ https://bytecodealliance.github.io/wamr.dev/
- kzsdvkajsf 4y ago
  so you can snoop on users pendrives if they make the mistake of allowing USB access from their google-chomeTM browser to your domain. ;)
  - beatrobot 4y ago
    I don't think Chrome allows access to storage devices through webusb.
- EdSharkey 4y ago
  I am working on a virtual computer for kids and retro loving adults. I wanted to simulate block storage using Local Storage as the backing store because I felt that blocks were very understandable. I've had block storage working for many years, but when I went to build a filesystem atop it, my design was uninspired and my implementation was buggy. When I found FatFs, I realized I had a perfect fit: a realish filesystem with an API that I could map right onto system calls, and FAT32 - a klunky old jalopy that will need defragmenters and other disk tools to maintain it. It'll make for an altogether more interesting ecosystem and a gentle challenge for the user to maintain. Also, there are many areas of my javascript that are primed for conversion to C -> WASM, like sprite collision detection. JavaScript optimizers have done ok with my code, but I'd like my compy to run fast on mobile too. If I am successful with FatFs, I will have a convenient spot in my codebase for converting my busy number-intensive routines into WASM.
- efrecon 4y ago
  There is a project doing some version of ext FS on top of S3. Just can't remember what it's called right now. It's active and on GH.
snvzz 4y ago
ioctl really is a ugly unixism that's totally out of place there. It could have been replaced by 3 or 4 calls with a clear single purpose, and I'd hope they'll realize this and fix the API in a future version.
- raggi 4y ago
  ++ don't ioctl / sockopt unless you absolutely have to for backward compatibility. It's an old hazardous design. We have oodles of space for symbols even on modern embedded systems.
  - sweetjuly 4y ago
    What is the recommended way under Linux then? It seemed like everyone was going full steam ahead with adding devices under /dev and /proc with control via read/write/ioctl. ioctl in my experience has been the best way to add a clean, modular interface to kernel modules since you don't need to hack in syscalls or do terrible, custom shared memory communication.
    - snvzz 4y ago
      The context isn't Linux (an UNIX clone, ioctl is endemic there, no hope). It is fatfs, a tiny fat library for embedded use, which most of the time will be used w/o an OS.
    - raggi 4y ago
      Netlink is becoming common, but it's also messy because well, it started as a single purpose interface and is now among other things a misnomer as it's not limited to network link controls. kdbus could probably provide richer interfaces eventually, but proximity to freedesktop will probably perpetuate a certain amount of pushback. The main point is that the ioctl interface is limited in its expression. It is hard to filter for security, and have bad evolutionary properties. Many of the subprotocol designs over the top of ioctl have memory safety hazards even beyond the immediate issue of needing to often make and trim buffers in less than ideal ways. This is to say the interface is hard to design well with, and a more formally typed system would provide a better solution. Regardless, OP isn't on Linux, and the point is to not copy the bad interface!
gavinray 4y ago
> This is my first time playing with emscripten and sharing buffers between wasm and javascript is awkward. You are using "SharedArrayBuffer", I assume?
- EdSharkey 4y ago
  I am preferring to use Uint8Array, Uint32Array or just let emscripten do the allocations for me. I haven't dabbled much with web workers, so I have avoided the whole transferrables thing.
oso2k 4y ago
There's an old an OS-free/BIOS-based implementation of FAT12 called MDCFS.C (Minimal Dos Compatible File System) in Dave Dunfield's EMBEDPC.ZIP (only in the Internet Archive now). You'll also need peek at DK86IO.C from that zip as well as come up with an implementation for those BIOS calls. Dave's been active on vogons lately. https://web.archive.org/web/20210505023207/http://dunfield.classiccmp.org/dos/index.htm https://web.archive.org/web/20210505023207/http://dunfield.c...
- EdSharkey 4y ago
  Thanks for the tip!

loeg 4y ago

http://elm-chan.org/fsw/ff/doc/dioctl.html http://elm-chan.org/fsw/ff/doc/dioctl.html describes the expected ioctls to implement and their behavior. The mandatory ioctls seem pretty straightforward. CTRL_SYNC will be a nop on a browser-based storage, GET_BLOCK_SIZE should return 1 (unknown), and CTRL_TRIM looks like it could be nop (or you could zero the range to more closely match real devices).

EdSharkey 4y ago
I didn't study in my operating systems class! If only I could shake dumbass young version of me and have them work harder at school! I know what you are saying is true and I should have no fear. I just need to write lots of tests and log like a mofo until I get all the ioctl commands it needs built out on the javascript side.

tremon 4y ago

Sorry, but I can't help wonder: why? What's the value of having direct device access in Javascript? Are we expected to give the browser raw block device access now? Maybe we could implement a FAT layer on top of S3, but I really don't see the value in reimplementing block-level storage on top of object storage.

kaoD 4y ago
Might be useful for non-browser VMs? E.g. a microcontroller running WASM.
- marcosdumay 4y ago
  Even tough yes, that's a good answer to the GP... Are people expecting WASM machines to be a thing? And with a different fate from the Java machines? (I don't see how they are different enough to change that fate.)
  - topspin 4y ago
    > Are people expecting WASM machines to be a thing? People are expecting to embed WASM[1]. A generic FAT can provide a lightweight storage abstraction for such work. The WAMR AOT compiled runtime is 50K; 2.5-10% of the flash memory on a STM32F4, for example. Obviously not appropriate for the smallest extant MCUs. Otherwise it's a viable choice. [1] https://bytecodealliance.github.io/wamr.dev/ https://bytecodealliance.github.io/wamr.dev/
kzsdvkajsf 4y ago
so you can snoop on users pendrives if they make the mistake of allowing USB access from their google-chomeTM browser to your domain. ;)
- beatrobot 4y ago
  I don't think Chrome allows access to storage devices through webusb.
EdSharkey 4y ago
I am working on a virtual computer for kids and retro loving adults. I wanted to simulate block storage using Local Storage as the backing store because I felt that blocks were very understandable. I've had block storage working for many years, but when I went to build a filesystem atop it, my design was uninspired and my implementation was buggy. When I found FatFs, I realized I had a perfect fit: a realish filesystem with an API that I could map right onto system calls, and FAT32 - a klunky old jalopy that will need defragmenters and other disk tools to maintain it. It'll make for an altogether more interesting ecosystem and a gentle challenge for the user to maintain. Also, there are many areas of my javascript that are primed for conversion to C -> WASM, like sprite collision detection. JavaScript optimizers have done ok with my code, but I'd like my compy to run fast on mobile too. If I am successful with FatFs, I will have a convenient spot in my codebase for converting my busy number-intensive routines into WASM.
efrecon 4y ago
There is a project doing some version of ext FS on top of S3. Just can't remember what it's called right now. It's active and on GH.

snvzz 4y ago

ioctl really is a ugly unixism that's totally out of place there. It could have been replaced by 3 or 4 calls with a clear single purpose, and I'd hope they'll realize this and fix the API in a future version.

raggi 4y ago
++ don't ioctl / sockopt unless you absolutely have to for backward compatibility. It's an old hazardous design. We have oodles of space for symbols even on modern embedded systems.
- sweetjuly 4y ago
  What is the recommended way under Linux then? It seemed like everyone was going full steam ahead with adding devices under /dev and /proc with control via read/write/ioctl. ioctl in my experience has been the best way to add a clean, modular interface to kernel modules since you don't need to hack in syscalls or do terrible, custom shared memory communication.
  - snvzz 4y ago
    The context isn't Linux (an UNIX clone, ioctl is endemic there, no hope). It is fatfs, a tiny fat library for embedded use, which most of the time will be used w/o an OS.
  - raggi 4y ago
    Netlink is becoming common, but it's also messy because well, it started as a single purpose interface and is now among other things a misnomer as it's not limited to network link controls. kdbus could probably provide richer interfaces eventually, but proximity to freedesktop will probably perpetuate a certain amount of pushback. The main point is that the ioctl interface is limited in its expression. It is hard to filter for security, and have bad evolutionary properties. Many of the subprotocol designs over the top of ioctl have memory safety hazards even beyond the immediate issue of needing to often make and trim buffers in less than ideal ways. This is to say the interface is hard to design well with, and a more formally typed system would provide a better solution. Regardless, OP isn't on Linux, and the point is to not copy the bad interface!

gavinray 4y ago

> This is my first time playing with emscripten and sharing buffers between wasm and javascript is awkward. You are using "SharedArrayBuffer", I assume?

EdSharkey 4y ago
I am preferring to use Uint8Array, Uint32Array or just let emscripten do the allocations for me. I haven't dabbled much with web workers, so I have avoided the whole transferrables thing.

oso2k 4y ago

There's an old an OS-free/BIOS-based implementation of FAT12 called MDCFS.C (Minimal Dos Compatible File System) in Dave Dunfield's EMBEDPC.ZIP (only in the Internet Archive now). You'll also need peek at DK86IO.C from that zip as well as come up with an implementation for those BIOS calls. Dave's been active on vogons lately. https://web.archive.org/web/20210505023207/http://dunfield.classiccmp.org/dos/index.htm https://web.archive.org/web/20210505023207/http://dunfield.c...

EdSharkey 4y ago
Thanks for the tip!