9 ms·

Show HN: Fuzz Map – a GUI fuzzer, interactive demo

205 points by jonathanyc 4y ago

huhtenberg 4y ago
> Here's a 5-second video of real-time fuzzing on my laptop. The space where the video is supposed to be is blank. Clicking on play instantly skips to 0:05 mark and nothing else happens. Firefox on Windows.
- wongarsu 4y ago
  It's a bit pointless anyways imho (on Desktop). Just close the popup and click Fuzz.
  - anarcat 4y ago
    i kind of like having the video. actually running the fuzzer uses a lot of CPU: it makes my fan startup. it works fine here on Firefox + Debian GNU/Linux 11 stable/bullseye.
- solardev 4y ago
  Plays fine in my Firefox for Windows, FWIW. Also works in Chrome desktop and mobile.
wongarsu 4y ago
I imagine this would be a very useful tool to have if you want to communicate about the design or user flow of an existing app, in addition to the highlighted use case of finding unexpected behavior and crashes.
- fullstackchris 4y ago
  And, if you can save 'snapshots' of a given fuzzmap to detect potential regressions and / or changes in general to the flow.
blondin 4y ago
so is this a fuzzer for generating GUI elements to see what works or not? or one to generate input states that crash a GUI?
- michaelmior 4y ago
  I'm not sure what you mean by "see what works." My understanding is that it's designed to automatically interact with a GUI and identify the possible different states that the UI could be in and the inputs required to enter those states. (Presumably this includes the state of being crashed.)
- solardev 4y ago
  It runs GUI code (like React) and fuzzes the inputs not just to generate possible crashes but to discover different code paths and states. The example they provide shows a simple food ordering system and the fuzzer eventually discovers how to place an order, but not before first encountering validation errors and fuzzing past them.
solardev 4y ago
Super cool, thanks for sharing!
atemerev 4y ago
Wow, that's really impressive. I'll probably start using this right away.
thomasfromcdnjs 4y ago
Wow.
makach 4y ago
wow! this is very cool. I wonder how this would work with slightly more complicated UIs. Does it create too much noise? Does anyone need this kind of verbosity?
- jonathanyc 4y ago
  Thank you! Good question about complicated GUIs. Keeping the map simple enough to be usable ended up being the most interesting part of the project, engineering-wise--the end of the write up touches on map simplification! There's a lot more I'd like to do.
kretaceous 4y ago
> In the short term, I'd like to make a plug-and-play version of Fuzz Map that supports end-to-end React fuzzing. You won't have to change your build process or use a special browser—just run ./fuzzmap localhost:8080 -p 9090 and start fuzzing or live programming! A reverse proxy will add instrumentation on the fly. Eagerly waiting for this. I need this. I don't write tests for a certain React codebase [1], and this could be a life saver. Amazing work! 1: Before you hate, I'm not against tests and I advise on writing them. There are certain situations in this case.
joeycodes 4y ago
I am so excited to see this live! I always wanted something like this for testing games and complex SPA's. UI's are particularly problematic for writing tests -- there's usually much more churn in that area of an application versus the business logic, and no matter how many tests we did try to write, users (and ambitious internal testers that clicked everywhere like crazy) would find a creative way to break things. I think having UI fuzzing in the CI pipeline will go a long way to deploying UI updates with more confidence. Congrats on launching!
stephendause 4y ago
Very nice! Does this tool have a state model that you can check the application state against? In other words, does it have property-based testing? Here[1] is a demo of what that looks like. I [1] https://medium.com/criteo-engineering/detecting-the-unexpected-in-web-ui-fuzzing-1f3822c8a3a5 https://medium.com/criteo-engineering/detecting-the-unexpect...
- jonathanyc 4y ago
  Thank you! I'd like to support property-based testing in the future. For this demo I focused on visualization and more exploratory testing. I figured that'd make the demo more approachable to people who weren't already into fuzzing or formal methods. But being able to automatically discover non-crashing bugs using property-based testing could be pretty powerful.
terpimost 4y ago
Wow, that is cool. How do you do code coverage? Console.log() every function call?
- jonathanyc 4y ago
  We add instrumentation at compile-time, just like conventional fuzzers! if (e) a else b becomes if (e) { hit(1); a } else { hit(2); b } There's more detail in the writeup if you curious, just scroll down!
tpoacher 4y ago
Why is this process called "Fuzzing"? Isn't this a really uninformative/ misleading name?
- WhitneyLand 4y ago
  Because it’s a fuzzy representation of input, in that it’s partially valid enough so to be able to use the system, but also invalid in a way that’s designed to find bugs. https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing
  - tpoacher 4y ago
    Thanks. Actually, if anything, the original source mentioned in that article strengthens my opinion. The term has nothing to do with 'fuzziness' as understood today. It was a fun, arbitrary term that was chosen for a student project on a whim rather than careful consideration. The 'fuzz' was referring to the 'fuzz generator', where 'fuzz' here seems to have been used as a synonym for 'crud', 'dirt', 'fluff', 'noise' etc, i.e as opposed to clean/useful/careful inputs. Nothing to do with the term 'fuzzy' as understood today (i.e. in terms of fuzzy logic or fuzzy set/measure theory). Good to know though. I kept hearing this term and didn't know what it referred to. Thanks for the link!
- deleted 4y ago
  [deleted]
nyanpasu64 4y ago
It's a HTML/JS GUI fuzzer, but the author saw this as the "default" to the point they didn't even specify that. At this point I've given up hope that native libraries and frameworks will return to the mainstream of desktop apps, but I'm still going to continue working with native libraries.
- pabs3 4y ago
  Even native programs are moving towards web tech, for eg GNOME shell is written in JavaScript and the PolicyKit component of the GNU/Linux desktop world uses JavaScript as a scripting language to define policies in. Of course there are Electron/Tauri apps for wrapping web tech in a native porting layer.
  - Abishek_Muthian 4y ago
    > eg GNOME shell is written in JavaScript Is that some way related to why almost every GNOME shell extension is a memory hog (due to memory leaks?), I was fed up of replacing shell extensions in the hope of finding better one and finally disabled shell extensions altogether. I've hardly had any GNOME crash after disabling shell extensions. P.S. Congratulations on the launch, OP. Seems like a very useful too, Would try to play with it later.
    - pabs3 4y ago
      I've noticed the memory leaks too. Unfortunately I can't live without GNOME shell extensions and I need to reboot/logout for security updates every now and then, so I just got enough RAM to let me continue working while leaks are going on.
      - Abishek_Muthian 4y ago
        I was doing the same, But unexpected crashes due to OOM killing became too much to handle with 32GB memory. I replaced the extensions I use with standalone applications and I've never been happier with GNOME. But its sad that the extensions ruin the experience of GNOME and possibly one of the main reasons for the hate it gets as a DE.
Klaster_1 4y ago
Do you plan to extend the support to other frameworks too, like Angular? A UI fuzzer would be handy to have in the toolbelt. Really enjoyed the write up, it answered some of the questions I had when thinking about fuzzers, for example the branch count blow up and optimizations, would read more if you decide to follow up with more development notes.

205 points by jonathanyc 4y ago

> Here's a 5-second video of real-time fuzzing on my laptop. The space where the video is supposed to be is blank. Clicking on play instantly skips to 0:05 mark and nothing else happens. Firefox on Windows.

wongarsu 4y ago
It's a bit pointless anyways imho (on Desktop). Just close the popup and click Fuzz.
- anarcat 4y ago
  i kind of like having the video. actually running the fuzzer uses a lot of CPU: it makes my fan startup. it works fine here on Firefox + Debian GNU/Linux 11 stable/bullseye.
solardev 4y ago
Plays fine in my Firefox for Windows, FWIW. Also works in Chrome desktop and mobile.

wongarsu 4y ago

I imagine this would be a very useful tool to have if you want to communicate about the design or user flow of an existing app, in addition to the highlighted use case of finding unexpected behavior and crashes.

fullstackchris 4y ago
And, if you can save 'snapshots' of a given fuzzmap to detect potential regressions and / or changes in general to the flow.

blondin 4y ago

so is this a fuzzer for generating GUI elements to see what works or not? or one to generate input states that crash a GUI?

michaelmior 4y ago
I'm not sure what you mean by "see what works." My understanding is that it's designed to automatically interact with a GUI and identify the possible different states that the UI could be in and the inputs required to enter those states. (Presumably this includes the state of being crashed.)
solardev 4y ago
It runs GUI code (like React) and fuzzes the inputs not just to generate possible crashes but to discover different code paths and states. The example they provide shows a simple food ordering system and the fuzzer eventually discovers how to place an order, but not before first encountering validation errors and fuzzing past them.

solardev 4y ago

Super cool, thanks for sharing!

atemerev 4y ago

Wow, that's really impressive. I'll probably start using this right away.

thomasfromcdnjs 4y ago

Wow.

makach 4y ago

wow! this is very cool. I wonder how this would work with slightly more complicated UIs. Does it create too much noise? Does anyone need this kind of verbosity?

jonathanyc 4y ago
Thank you! Good question about complicated GUIs. Keeping the map simple enough to be usable ended up being the most interesting part of the project, engineering-wise--the end of the write up touches on map simplification! There's a lot more I'd like to do.

kretaceous 4y ago

> In the short term, I'd like to make a plug-and-play version of Fuzz Map that supports end-to-end React fuzzing. You won't have to change your build process or use a special browser—just run ./fuzzmap localhost:8080 -p 9090 and start fuzzing or live programming! A reverse proxy will add instrumentation on the fly. Eagerly waiting for this. I need this. I don't write tests for a certain React codebase [1], and this could be a life saver. Amazing work! 1: Before you hate, I'm not against tests and I advise on writing them. There are certain situations in this case.

joeycodes 4y ago

I am so excited to see this live! I always wanted something like this for testing games and complex SPA's. UI's are particularly problematic for writing tests -- there's usually much more churn in that area of an application versus the business logic, and no matter how many tests we did try to write, users (and ambitious internal testers that clicked everywhere like crazy) would find a creative way to break things. I think having UI fuzzing in the CI pipeline will go a long way to deploying UI updates with more confidence. Congrats on launching!

stephendause 4y ago

Very nice! Does this tool have a state model that you can check the application state against? In other words, does it have property-based testing? Here[1] is a demo of what that looks like. I [1] https://medium.com/criteo-engineering/detecting-the-unexpected-in-web-ui-fuzzing-1f3822c8a3a5 https://medium.com/criteo-engineering/detecting-the-unexpect...

jonathanyc 4y ago
Thank you! I'd like to support property-based testing in the future. For this demo I focused on visualization and more exploratory testing. I figured that'd make the demo more approachable to people who weren't already into fuzzing or formal methods. But being able to automatically discover non-crashing bugs using property-based testing could be pretty powerful.

terpimost 4y ago

Wow, that is cool. How do you do code coverage? Console.log() every function call?

jonathanyc 4y ago
We add instrumentation at compile-time, just like conventional fuzzers! if (e) a else b becomes if (e) { hit(1); a } else { hit(2); b } There's more detail in the writeup if you curious, just scroll down!

tpoacher 4y ago

Why is this process called "Fuzzing"? Isn't this a really uninformative/ misleading name?

WhitneyLand 4y ago
Because it’s a fuzzy representation of input, in that it’s partially valid enough so to be able to use the system, but also invalid in a way that’s designed to find bugs. https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing
- tpoacher 4y ago
  Thanks. Actually, if anything, the original source mentioned in that article strengthens my opinion. The term has nothing to do with 'fuzziness' as understood today. It was a fun, arbitrary term that was chosen for a student project on a whim rather than careful consideration. The 'fuzz' was referring to the 'fuzz generator', where 'fuzz' here seems to have been used as a synonym for 'crud', 'dirt', 'fluff', 'noise' etc, i.e as opposed to clean/useful/careful inputs. Nothing to do with the term 'fuzzy' as understood today (i.e. in terms of fuzzy logic or fuzzy set/measure theory). Good to know though. I kept hearing this term and didn't know what it referred to. Thanks for the link!
deleted 4y ago
[deleted]

nyanpasu64 4y ago

It's a HTML/JS GUI fuzzer, but the author saw this as the "default" to the point they didn't even specify that. At this point I've given up hope that native libraries and frameworks will return to the mainstream of desktop apps, but I'm still going to continue working with native libraries.

pabs3 4y ago
Even native programs are moving towards web tech, for eg GNOME shell is written in JavaScript and the PolicyKit component of the GNU/Linux desktop world uses JavaScript as a scripting language to define policies in. Of course there are Electron/Tauri apps for wrapping web tech in a native porting layer.
- Abishek_Muthian 4y ago
  > eg GNOME shell is written in JavaScript Is that some way related to why almost every GNOME shell extension is a memory hog (due to memory leaks?), I was fed up of replacing shell extensions in the hope of finding better one and finally disabled shell extensions altogether. I've hardly had any GNOME crash after disabling shell extensions. P.S. Congratulations on the launch, OP. Seems like a very useful too, Would try to play with it later.
  - pabs3 4y ago
    I've noticed the memory leaks too. Unfortunately I can't live without GNOME shell extensions and I need to reboot/logout for security updates every now and then, so I just got enough RAM to let me continue working while leaks are going on.
    - Abishek_Muthian 4y ago
      I was doing the same, But unexpected crashes due to OOM killing became too much to handle with 32GB memory. I replaced the extensions I use with standalone applications and I've never been happier with GNOME. But its sad that the extensions ruin the experience of GNOME and possibly one of the main reasons for the hate it gets as a DE.

Klaster_1 4y ago

Do you plan to extend the support to other frameworks too, like Angular? A UI fuzzer would be handy to have in the toolbelt. Really enjoyed the write up, it answered some of the questions I had when thinking about fuzzers, for example the branch count blow up and optimizations, would read more if you decide to follow up with more development notes.