Thanks for your feedback! > My main question when reading is how the results compared to manually-written code. Ah, this is exactly the question. But as you s

Thanks for your feedback! > My main question when reading is how the results compared to manually-written code. Ah, this is exactly the question. But as you say, much harder to answer. Even if you run a competition, unless you can encourage a wide range of developers to enter, you won't be getting the real value. Instead you might be getting incidence rates of code written by students/interns. Perhaps if you could get a few FAANGs on board to either share internal data (unlikely) or send a random sample of employees (also very unlikely) to make teams and then evaluate their code... It seems like a difficult question to answer. We think a more doable way would be to take snapshots of large open source codebases (e.g. off GitHub) and measure the incidence rate of CWEs, but this also presents its own challenges with analyzing the data. Also, what's the relationship between open source code and all code? Lots of avenues to consider.