5 ms·

FuzzOS – an operating system which is designed specifically for fuzzing

122 points by URfejk 6y ago

AgloeDreams 6y ago
Can someone tell me what the living heck is `Fuzzing`? I read this twice and I really don't have a single clue other than it having something to do with or requiring fast memory?
- mehrdadn 6y ago
  Wikipedia explains it: https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing
- SAI_Peregrinus 6y ago
  Testing code via semi-random inputs[1]. The most common fuzzers, AFL-Fuzz[2] and libFuzzer[3] are coverage-guided: they compile the program with special instrumentation to determine code coverage, then call the program repeatedly, changing the inputs via genetic algorithm to try to maximize the code paths executed. When unexpected behavior is observed (typically the test harness crashing) the fuzzer saves the test's input for future use. Basically automatic generation of test case inputs. It's non-deterministic, so it won't always find problems, but it can save a lot of manual effort. [1] https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing [2] https://lcamtuf.coredump.cx/afl/ https://lcamtuf.coredump.cx/afl/ [3] https://www.llvm.org/docs/LibFuzzer.html https://www.llvm.org/docs/LibFuzzer.html
  - davidw 6y ago
    For an interesting, similar idea, see also: https://en.wikipedia.org/wiki/QuickCheck https://en.wikipedia.org/wiki/QuickCheck
- lambda_obrien 6y ago
  Fuzzing: give a program structured random garbage as input and see what happens, then fix the resulting bugs.
- Forge36 6y ago
  Originally: for each terminal program, pass every file as input. If crash results: document it. Effectively: random inputs to achieve unexpected results. It's now come to mean "random data testing of an API"
- amelius 6y ago
  Here is a tutorial I found: https://fuzzing-project.org/tutorial1.html https://fuzzing-project.org/tutorial1.html
yjftsjthsd-h 6y ago
Huh. So my initial response was, "why on earth would you need a whole OS for that", but memory snapshotting and improved virtual memory performance might actually be a good justification. Linux does have CRIU which might be made to work for such a purpose, but I could see a reasonable person preferring to do it from a clean slate. On the other hand, if you need qemu to run applications (which I'm really unclear about; I can't tell if the plan is to run stuff natively on this OS or just to provide enough system to run qemu and then run apps on linux on qemu) then I'm surprised that it's not easier to just make qemu do what you want (again, I'm pretty sure qemu already has its own memory snapshotting features to build on). Of course, writing an OS can be its own reward, too:)
- gamozolabs 6y ago
  Oooh, wasn't really expecting this to make it to HN cause it was meant to be more of an announcement than a description. But yes, I've done about 7 or 8 operating systems for fuzzing in the past and it's a massive performance (and cleanliness) cleanup. This one is going to be like an operating system I wrote 2-3 years ago for my vectorized emulation work. To answer your QEMU questions, the goal is to effectively build QEMU with MUSL (just to make it static so I don't need a dynamic loader), and modify MUSL to turn all syscalls to `call` instructions. This means a "syscall" is just a call to another area, which will by my Rust Linux emulator. I'll implement the bare minimum syscalls (and enum variants to those syscalls) to get QEMU to work, nothing more. The goal is not to run Linux applications, but run a QEMU+MUSL combination which may be modified lightly if it means a lower emulation burden (eg. getting rid of threading in QEMU [if possible] so we can avoid fork()) The main point of this isn't performance, it's determinism, but that is a side effect. A normal syscall instruction involves a context switch to the kernel, potentially cr3 swaps depending on CPU mitigation configuration, and the same to return back. This can easily be hundreds of cycles. A `call` instruction to something that handles the syscall is on the order of 1-4 cycles. While for syscalls this isn't a huge deal, it's even more emphasized when it comes to KVM hypercalls. Transitions to a hypervisor are very expensive, and in this case, the kernel, the hypervisor, and QEMU (eg. device emulation) will all be running at the same privilege level and there won't be a weird QEMU -> OS -> KVM -> other guest OS device -> KVM -> OS -> QEMU transition every device interaction. But then again, it's mainly for determinism. By emulating Linux deterministically (eg. not providing entropy through times or other syscall returns), we can ensure that QEMU has no source of external entropy, and thus, will always do the same thing. Even if it uses a random-seeded hash table, the seed would be derived from syscalls, and thus, will be the same every time. This determinism means the guest always will do the same thing, to the instruction. Interrupts happen on the same instructions, context switches do, etc. This means any bug, regardless of how complex, will reproduce every time. All of this syscall emulation + determinism I have also done before, in a tool called tkofuzz that I wrote for Microsoft. That used Linux emulation + Bochs, and it was written in userspace. This has proven incredibly successful and it's what most researchers are using at Microsoft now. That being said, Bochs is about 100x slower than native execution, and now that people have gotten a good hold of snapshot fuzzing (there's a steep learning curve), it's time to get a more performant implementation. With QEMU with get this with a JIT, which at least gets us a 2-5x improvement over Bochs while still "emulating", but even more value could be found if we get the KVM emulation working and can use a hypervisior. That being said, I do plan to support a "mode" where guests which do not touch devices (or more specifically, snapshots which are taken after device I/O has occurred) will be able to run without QEMU at all. We're really only using QEMU for device emulation + interrupt control, thus, if you take a snapshot to a function that just parses everything in one thread, without process IPC or device access (it's rare, when you "read" from a disk, you're likely just hitting OS RAM caches, and thus not devices), we can cut out all the "bloat" of QEMU and run in a very very thin hypervisor instead. In fuzzing it's critical to have ways to quickly map and unmap memory as most fuzz cases last for hundreds of microseconds. This means after a few hundred microseconds, I want to restore all memory back to the state "before I handled user input" and continue again. This is extremely slow in every conventional operating system, and there's really no way around it. It's of course possible to make a driver or use CRIU, but these are still not exactly the solution that is needed here. I'd rather just make an OS that trivially runs in KVM/Hyper-V/Xen, and thus can run in a VM to get the cross-platform support, rather than writing a driver for every OS I plan to use this on. Stay cute, ~gamozo
codetrotter 6y ago
I’ve been watching this guy stream on Twitch and I can tell you that he is legit. Also his streams are often insanely long, going 7 to 13 hours. So I only ever watch his streams live for a while and then I catch the remainder on VOD. He also has a YouTube page with archive of past streams beyond the retention of Twitch. https://www.youtube.com/channel/UC17ewSS9f2EnkCyMztCdoKA https://www.youtube.com/channel/UC17ewSS9f2EnkCyMztCdoKA Honestly the knowledge he shares is so interesting that I selfishly did not want other people to even know about it. But realistically speaking I am not going to have time to make real use of the knowledge myself anytime soon. He puts out quality content and he deserves all the attention he can get. And also, even though competition to find security bugs and earn bounties might become too hard that I myself or you ever get to find one and claim some money, the products that we all use will be more secure the more people in the world that work on finding these bugs and reporting them.
- albntomat0 6y ago
  Do you have any thoughts on how to approach the videos? I have a good OS and fuzzing background, but an 8 hour video seems like an ordeal and harder to extract value from than something written
  - codetrotter 6y ago
    I think my best advice would be that you tune in on one of the streams on Twitch when it is live and ask him about it. Then maybe the two of you can figure out what is most relevant to you of his content compared to what you already know?
    - muricula 6y ago
      codetrotter is right just tune into the stream and ask questions/do some light heckling as appropriate ;)
      - albntomat0 6y ago
        Thanks, I'll give it a shot! Definitely going to read some code as well, which I suspect will also provide more insight!

122 points by URfejk 6y ago

Can someone tell me what the living heck is `Fuzzing`? I read this twice and I really don't have a single clue other than it having something to do with or requiring fast memory?

mehrdadn 6y ago
Wikipedia explains it: https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing
SAI_Peregrinus 6y ago
Testing code via semi-random inputs[1]. The most common fuzzers, AFL-Fuzz[2] and libFuzzer[3] are coverage-guided: they compile the program with special instrumentation to determine code coverage, then call the program repeatedly, changing the inputs via genetic algorithm to try to maximize the code paths executed. When unexpected behavior is observed (typically the test harness crashing) the fuzzer saves the test's input for future use. Basically automatic generation of test case inputs. It's non-deterministic, so it won't always find problems, but it can save a lot of manual effort. [1] https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing [2] https://lcamtuf.coredump.cx/afl/ https://lcamtuf.coredump.cx/afl/ [3] https://www.llvm.org/docs/LibFuzzer.html https://www.llvm.org/docs/LibFuzzer.html
- davidw 6y ago
  For an interesting, similar idea, see also: https://en.wikipedia.org/wiki/QuickCheck https://en.wikipedia.org/wiki/QuickCheck
lambda_obrien 6y ago
Fuzzing: give a program structured random garbage as input and see what happens, then fix the resulting bugs.
Forge36 6y ago
Originally: for each terminal program, pass every file as input. If crash results: document it. Effectively: random inputs to achieve unexpected results. It's now come to mean "random data testing of an API"
amelius 6y ago
Here is a tutorial I found: https://fuzzing-project.org/tutorial1.html https://fuzzing-project.org/tutorial1.html

yjftsjthsd-h 6y ago

Huh. So my initial response was, "why on earth would you need a whole OS for that", but memory snapshotting and improved virtual memory performance might actually be a good justification. Linux does have CRIU which might be made to work for such a purpose, but I could see a reasonable person preferring to do it from a clean slate. On the other hand, if you need qemu to run applications (which I'm really unclear about; I can't tell if the plan is to run stuff natively on this OS or just to provide enough system to run qemu and then run apps on linux on qemu) then I'm surprised that it's not easier to just make qemu do what you want (again, I'm pretty sure qemu already has its own memory snapshotting features to build on). Of course, writing an OS can be its own reward, too:)

gamozolabs 6y ago
Oooh, wasn't really expecting this to make it to HN cause it was meant to be more of an announcement than a description. But yes, I've done about 7 or 8 operating systems for fuzzing in the past and it's a massive performance (and cleanliness) cleanup. This one is going to be like an operating system I wrote 2-3 years ago for my vectorized emulation work. To answer your QEMU questions, the goal is to effectively build QEMU with MUSL (just to make it static so I don't need a dynamic loader), and modify MUSL to turn all syscalls to `call` instructions. This means a "syscall" is just a call to another area, which will by my Rust Linux emulator. I'll implement the bare minimum syscalls (and enum variants to those syscalls) to get QEMU to work, nothing more. The goal is not to run Linux applications, but run a QEMU+MUSL combination which may be modified lightly if it means a lower emulation burden (eg. getting rid of threading in QEMU [if possible] so we can avoid fork()) The main point of this isn't performance, it's determinism, but that is a side effect. A normal syscall instruction involves a context switch to the kernel, potentially cr3 swaps depending on CPU mitigation configuration, and the same to return back. This can easily be hundreds of cycles. A `call` instruction to something that handles the syscall is on the order of 1-4 cycles. While for syscalls this isn't a huge deal, it's even more emphasized when it comes to KVM hypercalls. Transitions to a hypervisor are very expensive, and in this case, the kernel, the hypervisor, and QEMU (eg. device emulation) will all be running at the same privilege level and there won't be a weird QEMU -> OS -> KVM -> other guest OS device -> KVM -> OS -> QEMU transition every device interaction. But then again, it's mainly for determinism. By emulating Linux deterministically (eg. not providing entropy through times or other syscall returns), we can ensure that QEMU has no source of external entropy, and thus, will always do the same thing. Even if it uses a random-seeded hash table, the seed would be derived from syscalls, and thus, will be the same every time. This determinism means the guest always will do the same thing, to the instruction. Interrupts happen on the same instructions, context switches do, etc. This means any bug, regardless of how complex, will reproduce every time. All of this syscall emulation + determinism I have also done before, in a tool called tkofuzz that I wrote for Microsoft. That used Linux emulation + Bochs, and it was written in userspace. This has proven incredibly successful and it's what most researchers are using at Microsoft now. That being said, Bochs is about 100x slower than native execution, and now that people have gotten a good hold of snapshot fuzzing (there's a steep learning curve), it's time to get a more performant implementation. With QEMU with get this with a JIT, which at least gets us a 2-5x improvement over Bochs while still "emulating", but even more value could be found if we get the KVM emulation working and can use a hypervisior. That being said, I do plan to support a "mode" where guests which do not touch devices (or more specifically, snapshots which are taken after device I/O has occurred) will be able to run without QEMU at all. We're really only using QEMU for device emulation + interrupt control, thus, if you take a snapshot to a function that just parses everything in one thread, without process IPC or device access (it's rare, when you "read" from a disk, you're likely just hitting OS RAM caches, and thus not devices), we can cut out all the "bloat" of QEMU and run in a very very thin hypervisor instead. In fuzzing it's critical to have ways to quickly map and unmap memory as most fuzz cases last for hundreds of microseconds. This means after a few hundred microseconds, I want to restore all memory back to the state "before I handled user input" and continue again. This is extremely slow in every conventional operating system, and there's really no way around it. It's of course possible to make a driver or use CRIU, but these are still not exactly the solution that is needed here. I'd rather just make an OS that trivially runs in KVM/Hyper-V/Xen, and thus can run in a VM to get the cross-platform support, rather than writing a driver for every OS I plan to use this on. Stay cute, ~gamozo

codetrotter 6y ago

I’ve been watching this guy stream on Twitch and I can tell you that he is legit. Also his streams are often insanely long, going 7 to 13 hours. So I only ever watch his streams live for a while and then I catch the remainder on VOD. He also has a YouTube page with archive of past streams beyond the retention of Twitch. https://www.youtube.com/channel/UC17ewSS9f2EnkCyMztCdoKA https://www.youtube.com/channel/UC17ewSS9f2EnkCyMztCdoKA Honestly the knowledge he shares is so interesting that I selfishly did not want other people to even know about it. But realistically speaking I am not going to have time to make real use of the knowledge myself anytime soon. He puts out quality content and he deserves all the attention he can get. And also, even though competition to find security bugs and earn bounties might become too hard that I myself or you ever get to find one and claim some money, the products that we all use will be more secure the more people in the world that work on finding these bugs and reporting them.

albntomat0 6y ago
Do you have any thoughts on how to approach the videos? I have a good OS and fuzzing background, but an 8 hour video seems like an ordeal and harder to extract value from than something written
- codetrotter 6y ago
  I think my best advice would be that you tune in on one of the streams on Twitch when it is live and ask him about it. Then maybe the two of you can figure out what is most relevant to you of his content compared to what you already know?
  - muricula 6y ago
    codetrotter is right just tune into the stream and ask questions/do some light heckling as appropriate ;)
    - albntomat0 6y ago
      Thanks, I'll give it a shot! Definitely going to read some code as well, which I suspect will also provide more insight!