6 ms·

Hi HN! I wrote an article about this initial support series and what it took to make things work properly on this machine: https://asahilinux.org/2021/03/progr

Hi HN! I wrote an article about this initial support series and what it took to make things work properly on this machine: https://asahilinux.org/2021/03/progress-report-january-february-2021/ https://asahilinux.org/2021/03/progress-report-january-febru... It still applies to what got merged; the past month was spent on clean-ups, fixes, and review feedback, but there are no major changes to the approach. This initial merge was quite complex as it has to touch a number of subsystems and core kernel code in order to support the M1's quirks, and it was CCed to ~20 people. Now that it's done we can more efficiently work on individual drivers and subsystems, so I expect the pace to pick up. If you would like to support my work on this project, I do have a Patreon: https://www.patreon.com/marcan https://www.patreon.com/marcan I also streamed initial development on YouTube and will be resuming streams next week (patch feedback and git rebasing and munging doesn't make for very interesting streams, but now I'm back to coding). Coming up I have some driver support other people have been working on, and then I'll be writing a minimal hypervisor that can run macOS as a guest, to help reverse engineer the hardware - this will be important for reverse engineering more complex drivers cleanly, especially the GPU kernel side. Until now we've used a serial cable to debug/load kernels (needs a DIY arduino thing, an proper design I'm working on which is still vapourware, or another M1 box), but Sven added support for the USB device controller and I'll merge it into our bootloader soon, so from this point on anyone will be able to do quick kernel iteration, debugging, and hardware exploration with just a standard USB cable and any other host machine.

sillysaurusx 6y ago
(Side note: I'm happy to see that shilling your patreon is becoming a socially accepted norm. Best of luck with your work!) One thing I was surprised by: LuaJIT doesn't compile on an M1. LuaJIT 2.1.0 technically compiles, but it core dumps whenever you have a syntax error, which makes it quite a bit less useful. A reason for this might be due to the lack of __threadlocal support in M1's (apparently, or so says clang). Is there some way this could be addressed in the Linux work? I'm not sure I have a coherent question here, but my goal is mainly (a) to express surprise that LuaJIT, of all things, doesn't run on an M1, and (b) to maybe suggest that it would be cool as heck to have it running in the Linux M1 work as a benchmark, if possible. I'd hate to see such a wonderful tool get left behind by the new arch.
- ithkuil 6y ago
  Have you tried to use the zld linker to build luajit?
- otabdeveloper4 6y ago
  LuaJIT is extremely impressive engineering, but also a giant pile of hacks and shortcuts. I'm not surprised.
- tyingq 6y ago
  >lack of __threadlocal support in M1 An informative post about this from the Zig folks: http://www.jakubkonka.com/2021/01/21/llvm-tls-apple-silicon.html http://www.jakubkonka.com/2021/01/21/llvm-tls-apple-silicon....
  - comex 6y ago
    That's not a lack of support, it's just LLD, the LLVM linker, being broken. I hear LLD for Mach-O is being worked on again, after a long period of stagnation where it was sort of but not fully working. But the default linker on Apple platforms is still ld64.
    - tyingq 6y ago
      I'd give them a little more grace. It sounds like the M1 requires a different approach than other ARMV7 cpus, and that fact wasn't documented. Mimalloc had to trudge through the same thing: https://github.com/microsoft/mimalloc/issues/343 https://github.com/microsoft/mimalloc/issues/343
      - comex 6y ago
        Well, in theory, Zig doesn't need to care about the specific approach. In the LLVM IR that Zig produces, thread-local variables are still handled as an abstract "thread_local" flag on global variables. The platform-specific parts are handled by a combination of the IR-to-native-object-file translation done by LLVM, and the linking step. If both of those worked correctly, thread locals would "just work" on M1. The problem is that (a) Zig uses LLD for linking and (b) LLD is buggy. I'm not surprised that Zig is using LLD. Zig wants to be fully self-contained, able to cross-compile for any supported platform without requiring the user to install a toolchain for that platform. And they already use LLVM for code generation, so LLD for linking is the natural choice. It's a beautiful vision - really. Toolchain installation sucks. And the stuff they're doing with processing and bundling OS C headers [1] is quite innovative. It's just that using LLVM for everything is not a reality for Apple platforms yet. Hopefully it will be soon, once Apple whips LLD into shape. Edit: mimalloc is different; it reimplements thread-local storage by hand, using `asm` statements, rather than using native `thread_local` variables, presumably because the latter can be slow. [1] https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace...
wdb 6y ago
Impressive work everyone!
zamadatix 6y ago
Hey Marcan, really impressed with the progress it's been 100% worth the patreon sponsorship. I first ran into you back in the Wii hacking days and it's awesome to see you're still doing cool reverse engineering stuff after all this time :). How are things looking on the "primary job goal" accounting for GitHub sponsors as well? Think you'll hit it this summer?
ndesaulniers 6y ago
Thank you so much for the work put in to documentation; written in kernel or blogged, and streamed. I've watched hours of your streams. I've been working on a bootloader and porting Linux to the Nintendo 3DS; I don't really know what I'm doing, but your streams clarified many questions I had about porting Linux to a new device.
- in9 6y ago
  This is so awesome dude! Please let us know where we can find more about your project :D
  - ndesaulniers 6y ago
    https://github.com/linux-3ds https://github.com/linux-3ds https://github.com/linux-3ds/linux/wiki/Table-of-Contents https://github.com/linux-3ds/linux/wiki/Table-of-Contents It's not my project in the sense that I didn't start it, or do most of the work; I just enjoy contributing some fixes here and there and its my first experience working on a bootloader. There's a ton of great documentation (from the community, and ARM) at https://www.3dbrew.org/wiki/Main_Page https://www.3dbrew.org/wiki/Main_Page as well as existing code to reference from others in the community. So I'm still in awe of how marcan goes about finding out how the hardware works without documentation from Apple (which IIUC he has done before with the Wii and BootMii). I'm hoping to put out a release soon, and maybe stream some development on https://www.twitch.tv/ndesaulniers https://www.twitch.tv/ndesaulniers, then pursue upstreaming what we have.

sillysaurusx 6y ago

(Side note: I'm happy to see that shilling your patreon is becoming a socially accepted norm. Best of luck with your work!) One thing I was surprised by: LuaJIT doesn't compile on an M1. LuaJIT 2.1.0 technically compiles, but it core dumps whenever you have a syntax error, which makes it quite a bit less useful. A reason for this might be due to the lack of __threadlocal support in M1's (apparently, or so says clang). Is there some way this could be addressed in the Linux work? I'm not sure I have a coherent question here, but my goal is mainly (a) to express surprise that LuaJIT, of all things, doesn't run on an M1, and (b) to maybe suggest that it would be cool as heck to have it running in the Linux M1 work as a benchmark, if possible. I'd hate to see such a wonderful tool get left behind by the new arch.

ithkuil 6y ago
Have you tried to use the zld linker to build luajit?
otabdeveloper4 6y ago
LuaJIT is extremely impressive engineering, but also a giant pile of hacks and shortcuts. I'm not surprised.
tyingq 6y ago
>lack of __threadlocal support in M1 An informative post about this from the Zig folks: http://www.jakubkonka.com/2021/01/21/llvm-tls-apple-silicon.html http://www.jakubkonka.com/2021/01/21/llvm-tls-apple-silicon....
- comex 6y ago
  That's not a lack of support, it's just LLD, the LLVM linker, being broken. I hear LLD for Mach-O is being worked on again, after a long period of stagnation where it was sort of but not fully working. But the default linker on Apple platforms is still ld64.
  - tyingq 6y ago
    I'd give them a little more grace. It sounds like the M1 requires a different approach than other ARMV7 cpus, and that fact wasn't documented. Mimalloc had to trudge through the same thing: https://github.com/microsoft/mimalloc/issues/343 https://github.com/microsoft/mimalloc/issues/343
    - comex 6y ago
      Well, in theory, Zig doesn't need to care about the specific approach. In the LLVM IR that Zig produces, thread-local variables are still handled as an abstract "thread_local" flag on global variables. The platform-specific parts are handled by a combination of the IR-to-native-object-file translation done by LLVM, and the linking step. If both of those worked correctly, thread locals would "just work" on M1. The problem is that (a) Zig uses LLD for linking and (b) LLD is buggy. I'm not surprised that Zig is using LLD. Zig wants to be fully self-contained, able to cross-compile for any supported platform without requiring the user to install a toolchain for that platform. And they already use LLVM for code generation, so LLD for linking is the natural choice. It's a beautiful vision - really. Toolchain installation sucks. And the stuff they're doing with processing and bundling OS C headers [1] is quite innovative. It's just that using LLVM for everything is not a reality for Apple platforms yet. Hopefully it will be soon, once Apple whips LLD into shape. Edit: mimalloc is different; it reimplements thread-local storage by hand, using `asm` statements, rather than using native `thread_local` variables, presumably because the latter can be slow. [1] https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace...

wdb 6y ago

Impressive work everyone!

zamadatix 6y ago

Hey Marcan, really impressed with the progress it's been 100% worth the patreon sponsorship. I first ran into you back in the Wii hacking days and it's awesome to see you're still doing cool reverse engineering stuff after all this time :). How are things looking on the "primary job goal" accounting for GitHub sponsors as well? Think you'll hit it this summer?

ndesaulniers 6y ago

Thank you so much for the work put in to documentation; written in kernel or blogged, and streamed. I've watched hours of your streams. I've been working on a bootloader and porting Linux to the Nintendo 3DS; I don't really know what I'm doing, but your streams clarified many questions I had about porting Linux to a new device.

in9 6y ago
This is so awesome dude! Please let us know where we can find more about your project :D
- ndesaulniers 6y ago
  https://github.com/linux-3ds https://github.com/linux-3ds https://github.com/linux-3ds/linux/wiki/Table-of-Contents https://github.com/linux-3ds/linux/wiki/Table-of-Contents It's not my project in the sense that I didn't start it, or do most of the work; I just enjoy contributing some fixes here and there and its my first experience working on a bootloader. There's a ton of great documentation (from the community, and ARM) at https://www.3dbrew.org/wiki/Main_Page https://www.3dbrew.org/wiki/Main_Page as well as existing code to reference from others in the community. So I'm still in awe of how marcan goes about finding out how the hardware works without documentation from Apple (which IIUC he has done before with the Wii and BootMii). I'm hoping to put out a release soon, and maybe stream some development on https://www.twitch.tv/ndesaulniers https://www.twitch.tv/ndesaulniers, then pursue upstreaming what we have.