5 ms·

For me, using Fuzzilli for testing a Zig code is not fuzzing, it's integration testing. If you're running code externally (e.g. wrapping binary) you cannot guar

by xlii 3mo ago

For me, using Fuzzilli for testing a Zig code is not fuzzing, it's integration testing. If you're running code externally (e.g. wrapping binary) you cannot guarantee that side effect isn't caused by IO. I consider fuzzing a low level activity with many external variables removed. Depending on where you are and how you communicate semantics matter more or less. It's very similar to compiler/transpiler. E.g. TypeScript "Compiler" is called compiler but in fact it's transpiler (it emits other high-level language as a result).

mtlynch 3mo ago
> For me, using Fuzzilli for testing a Zig code is not fuzzing, it's integration testing. If you're running code externally (e.g. wrapping binary) you cannot guarantee that side effect isn't caused by IO. I consider fuzzing a low level activity with many external variables removed. I've never heard anyone restrict the definition of "fuzzing" in this way. If I repeatedly generate inputs to a program and then run the program with those inputs, that's fuzzing. It doesn't matter if there's IO or not. > Depending on where you are and how you communicate semantics matter more or less. It's very similar to compiler/transpiler. E.g. TypeScript "Compiler" is called compiler but in fact it's transpiler (it emits other high-level language as a result). It's still a compiler. It translates code from one language to another. You can argue whether we need the term "transpiler," but a source-to-source compiler is a compiler.
- ffsm8 3mo ago
  > You can argue whether we need the term "transpiler," but a source-to-source compiler is a compiler. That's true today, but compiling was historically was defined as getting source code (human readable) to bytecode (machine runnable without an interpreter). Some people didn't like that definition, and consequently the waters have been murkied. Just like with eg crypto. Or real time.
  - milch 3mo ago
    How historical? Compilers that have translated from a source language into C and have left the C-to-bytecode translation to another compiler have been around for a long time, as have compilers that translated from a source language to Assembly
    - ffsm8 3mo ago
      I'm not sure what your point is, we all already acknowledged that people have been using the term differently, consequently changing the definition of the word over the years
jsnell 3mo ago
My point is that Kelley did not argue that what Bun does isn't really fuzzing. He wrote that the post's claim is a fabrication. But that claim is really specific, and to evaluate whether it is true it doesn't matter what Kelley's unstated definition of fuzzing is. So an argument about definitions doesn't seem super valuable here.
- FreakLegion 3mo ago
  The Fuzzilli PR was merged on Nov 20. The acquisition was announced on Dec 3. A big holiday was in the middle. The teams no longer interacted after the acquisition, and in prior interactions the Bun team would've been correct in saying they weren't fuzzing. So Jarred isn't wrong, and Andrew also isn't wrong.
  - mtlynch 3mo ago
    Andrew was wrong. He stealth edited his post to hide it. https://news.ycombinator.com/item?id=48854921 https://news.ycombinator.com/item?id=48854921
    - FreakLegion 3mo ago
      I'm aware he edited it, but the original version isn't wrong, either. Just like Jarred's > We fuzz Bun's runtime APIs 24/7 using Fuzzilli, the JavaScript engine fuzzer used by V8 & JavaScriptCore isn't wrong, even though that was only being done for the last 5-6 months of Zig Bun, and not the previous 5 years when they were accruing all of their tech debt.
      - mtlynch 3mo ago
        I'm having trouble understanding what you mean. If I say, "I run 5 miles every day" and my old neighbor says, "I lived next door to him until 9 months ago, and he definitely doesn't run 5 miles a day," and then I show my GPS logs proving I've been running 5 miles a day for the last 9 months, I am correct and my ex-neighbor is incorrect. If Sumner had said, "We've been fuzzing our code for years," then Kelley could justifiably say that's incorrect. But Sumner is saying that currently Bun fuzzes their code, which is true, so Kelley appears to be incorrect to claim it is a "fabrication."

by xlii 3mo ago

mtlynch 3mo ago

> For me, using Fuzzilli for testing a Zig code is not fuzzing, it's integration testing. If you're running code externally (e.g. wrapping binary) you cannot guarantee that side effect isn't caused by IO. I consider fuzzing a low level activity with many external variables removed. I've never heard anyone restrict the definition of "fuzzing" in this way. If I repeatedly generate inputs to a program and then run the program with those inputs, that's fuzzing. It doesn't matter if there's IO or not. > Depending on where you are and how you communicate semantics matter more or less. It's very similar to compiler/transpiler. E.g. TypeScript "Compiler" is called compiler but in fact it's transpiler (it emits other high-level language as a result). It's still a compiler. It translates code from one language to another. You can argue whether we need the term "transpiler," but a source-to-source compiler is a compiler.

ffsm8 3mo ago
> You can argue whether we need the term "transpiler," but a source-to-source compiler is a compiler. That's true today, but compiling was historically was defined as getting source code (human readable) to bytecode (machine runnable without an interpreter). Some people didn't like that definition, and consequently the waters have been murkied. Just like with eg crypto. Or real time.
- milch 3mo ago
  How historical? Compilers that have translated from a source language into C and have left the C-to-bytecode translation to another compiler have been around for a long time, as have compilers that translated from a source language to Assembly
  - ffsm8 3mo ago
    I'm not sure what your point is, we all already acknowledged that people have been using the term differently, consequently changing the definition of the word over the years

jsnell 3mo ago

My point is that Kelley did not argue that what Bun does isn't really fuzzing. He wrote that the post's claim is a fabrication. But that claim is really specific, and to evaluate whether it is true it doesn't matter what Kelley's unstated definition of fuzzing is. So an argument about definitions doesn't seem super valuable here.

FreakLegion 3mo ago
The Fuzzilli PR was merged on Nov 20. The acquisition was announced on Dec 3. A big holiday was in the middle. The teams no longer interacted after the acquisition, and in prior interactions the Bun team would've been correct in saying they weren't fuzzing. So Jarred isn't wrong, and Andrew also isn't wrong.
- mtlynch 3mo ago
  Andrew was wrong. He stealth edited his post to hide it. https://news.ycombinator.com/item?id=48854921 https://news.ycombinator.com/item?id=48854921
  - FreakLegion 3mo ago
    I'm aware he edited it, but the original version isn't wrong, either. Just like Jarred's > We fuzz Bun's runtime APIs 24/7 using Fuzzilli, the JavaScript engine fuzzer used by V8 & JavaScriptCore isn't wrong, even though that was only being done for the last 5-6 months of Zig Bun, and not the previous 5 years when they were accruing all of their tech debt.
    - mtlynch 3mo ago
      I'm having trouble understanding what you mean. If I say, "I run 5 miles every day" and my old neighbor says, "I lived next door to him until 9 months ago, and he definitely doesn't run 5 miles a day," and then I show my GPS logs proving I've been running 5 miles a day for the last 9 months, I am correct and my ex-neighbor is incorrect. If Sumner had said, "We've been fuzzing our code for years," then Kelley could justifiably say that's incorrect. But Sumner is saying that currently Bun fuzzes their code, which is true, so Kelley appears to be incorrect to claim it is a "fabrication."