4 ms·

>$1M for the AI that can close 90% of new GitHub issues If your AI can do this, it's worth several orders of magnitude more. Just FYI.

thrance 2y ago
It's 90% of a selection of new GitHub issues, we don't know about the complexity of these. I don't think they'd ask the AI for a giant refactoring of the codebase, for example.
toast0 2y ago
Can't the stale bot already do this?
segmondy 2y ago
Exactly, I'll personally buy it for $2Million for anyone that can get it and assign me the full code/weight and rights.
- noch 2y ago
  > I'll personally buy it for $2Million for anyone that can get it and assign me the full code/weight and rights. If you are serious, you should put the funds in an escrow contract and announce a bounty. There are many brilliant people who would work on this for you.
  - deleted 2y ago
    [deleted]
- andyk 2y ago
  I hope the competition will inspire people to make breakthroughs in the open, so I won't take any rights to the IP, instead the winning solutions must use open source code and open weight models.
senko 2y ago
And Linux kernel, curl, SQLite and many other open source software are worth infinitely more than the purchase price. Also, you cut off the "from the benchmark" part; this doesn't expect it to solve any random Github issue, just the ones from the (presumably manually vetted and cleaned up) bench dataset.
- minimaxir 2y ago
  Linux kernel, curl, and SQLite don't require significant compute cost to develop that put it out of reach of hobbyists, and only in the reach of organizations expecting a positive ROI.
  - senko 2y ago
    The cost of Linux kernel development alone has been estimated at a few $B (https://dwheeler.com/essays/linux-kernel-cost.html https://dwheeler.com/essays/linux-kernel-cost.html), current figure is probably over tens of billions. Also, the prize doesn't require you to train a new foundational model, just that whatever you use is open weights or open source. Theoretically, might be get away with a Llama3.3 (or any other model which you think makes sense) with a cleverly designed agentic system and a fresh codebase-understanding approach, with minimal compute cost. (ok, probably not that easy, but just saying there's much more to AI coding that the underlying model)
    - bruce511 2y ago
      >> The cost of Linux kernel development alone has been estimated at a few $B (https://dwheeler.com/essays/linux-kernel-cost.html https://dwheeler.com/essays/linux-kernel-cost.html), current figure is probably over tens of billions. I followed your link, but it doesn't seem to bear out upur assertion. The two numbers mentioned in the article are 176 mil and 612 mil. Mind you those weren't an estimate of cost, but rather an estimate to replace. Article is dated 2004, with an update in 2011. Using the lines-of-code estimation it crossed a billion in 2010 - again to replace. That has no relation to what it did actually cost. Getting from there to "tens of billions" seems a stretch. Assuming a bottom value in your estimate of 20 billion, and assuming a developer costs a million a year, that's 20 000 man-years of effort. Which implies something like 2000 people (very well paid people) working continuously for the last decade. Which seems, well, unlikely.
andyk 2y ago
(reposting from locallama and lower down here) yep that's true. one of my goals is to inspire and honor those that work on open source AI. Those people tend to be motivated by things like impact and the excitement of being part of something big. i know that's how i always feel when i'm around Berkeley and get to meet or work with OG BSD hackers or the people who helped invent core internet protocols. those people are doing this kind of OSS work and sharing it with the world anyway, without any cash prize. i think of this as a sort of thank you gift for them. and also a way to maybe convince a few people to explore that path who might not have otherwise.
frgtpsswrdlame 2y ago
If you're the only one that can come close. Kaggle competition prizes are about focusing smart people on the same problem. But it's very rare for one team to blow all the others out of the water. So if you wanted to make a business out of the problem kaggle will (probably) show the best you could do and still have no moat.

thrance 2y ago

It's 90% of a selection of new GitHub issues, we don't know about the complexity of these. I don't think they'd ask the AI for a giant refactoring of the codebase, for example.

toast0 2y ago

Can't the stale bot already do this?

segmondy 2y ago

Exactly, I'll personally buy it for $2Million for anyone that can get it and assign me the full code/weight and rights.

noch 2y ago
> I'll personally buy it for $2Million for anyone that can get it and assign me the full code/weight and rights. If you are serious, you should put the funds in an escrow contract and announce a bounty. There are many brilliant people who would work on this for you.
- deleted 2y ago
  [deleted]
andyk 2y ago
I hope the competition will inspire people to make breakthroughs in the open, so I won't take any rights to the IP, instead the winning solutions must use open source code and open weight models.

senko 2y ago

And Linux kernel, curl, SQLite and many other open source software are worth infinitely more than the purchase price. Also, you cut off the "from the benchmark" part; this doesn't expect it to solve any random Github issue, just the ones from the (presumably manually vetted and cleaned up) bench dataset.

minimaxir 2y ago
Linux kernel, curl, and SQLite don't require significant compute cost to develop that put it out of reach of hobbyists, and only in the reach of organizations expecting a positive ROI.
- senko 2y ago
  The cost of Linux kernel development alone has been estimated at a few $B (https://dwheeler.com/essays/linux-kernel-cost.html https://dwheeler.com/essays/linux-kernel-cost.html), current figure is probably over tens of billions. Also, the prize doesn't require you to train a new foundational model, just that whatever you use is open weights or open source. Theoretically, might be get away with a Llama3.3 (or any other model which you think makes sense) with a cleverly designed agentic system and a fresh codebase-understanding approach, with minimal compute cost. (ok, probably not that easy, but just saying there's much more to AI coding that the underlying model)
  - bruce511 2y ago
    >> The cost of Linux kernel development alone has been estimated at a few $B (https://dwheeler.com/essays/linux-kernel-cost.html https://dwheeler.com/essays/linux-kernel-cost.html), current figure is probably over tens of billions. I followed your link, but it doesn't seem to bear out upur assertion. The two numbers mentioned in the article are 176 mil and 612 mil. Mind you those weren't an estimate of cost, but rather an estimate to replace. Article is dated 2004, with an update in 2011. Using the lines-of-code estimation it crossed a billion in 2010 - again to replace. That has no relation to what it did actually cost. Getting from there to "tens of billions" seems a stretch. Assuming a bottom value in your estimate of 20 billion, and assuming a developer costs a million a year, that's 20 000 man-years of effort. Which implies something like 2000 people (very well paid people) working continuously for the last decade. Which seems, well, unlikely.

andyk 2y ago

(reposting from locallama and lower down here) yep that's true. one of my goals is to inspire and honor those that work on open source AI. Those people tend to be motivated by things like impact and the excitement of being part of something big. i know that's how i always feel when i'm around Berkeley and get to meet or work with OG BSD hackers or the people who helped invent core internet protocols. those people are doing this kind of OSS work and sharing it with the world anyway, without any cash prize. i think of this as a sort of thank you gift for them. and also a way to maybe convince a few people to explore that path who might not have otherwise.

frgtpsswrdlame 2y ago

If you're the only one that can come close. Kaggle competition prizes are about focusing smart people on the same problem. But it's very rare for one team to blow all the others out of the water. So if you wanted to make a business out of the problem kaggle will (probably) show the best you could do and still have no moat.