5 ms·

Come on everyone, we all know that "most of the work that goes into software engineering isn't writing code", that "it's impossible to create a working system i

Come on everyone, we all know that "most of the work that goes into software engineering isn't writing code", that "it's impossible to create a working system in one shot", etc etc etc, and therefore this tool will not give a realistic idea of what it would cost to build a system with nothing but GPT-4. No one here (including the author) is under the impression that you can just ask ChatGPT to "write a Unix-like kernel" and get Linux spit out the other side. That said, I am really curious what it would cost, as a theoretical lower bound, to pay OpenAI to write the Linux Kernel. 10 grand? 100 grand? I have no clue, but I'm glad someone wrote a tool to make it easier to find out!

londons_explore 3y ago
> No one here (including the author) is under the impression that you can just ask Chat GPT to "write a Unix-like kernel" and get Linux spit out the other side. I really don't see such a thing being far off. GPT-4 is already pretty good at writing small modules ("write a disk IO queue for my custom kernel in C"). With a little more work in allowing GPT-4 to test out code it has written and iteratively make changes, allowing it to use debuggers, benchmarks, and sanitizers, allowing it to write its own tools, and then put modules together, I think we could very soon be asking it to "write me a Unix-like kernel".
- gwoolhurme 3y ago
  It’s that good in your experience? Enough that it is close to writing something as nice as the Linux Kernel we have now? How close is close? Reading comments like these makes me feel like I’m swallowing crazy pills. It legitimately makes me feel like I’m using it wrong. I do Android and iOS native development and some IoT and it gives me really wrong code very often. To the point that I don’t see much difference between chatgpt and gpt4. But you say it’s close to just “write me a unix-like kernel”
  - starbugs 3y ago
    It's not close and I also see no big difference between GPT 3.5 and 4. Don't get hyped. I am sure it will eventually happen, but calling it close is very optimistic.
  - londons_explore 3y ago
    Not close to as in "it can nearly write it correctly", but close to as in "I believe within a small number of years, gpt-4-like tools would be able to write you a unix-like kernel from scratch and have it actually work, with no human input". I think we already have most of the pieces in place: * big language models that sometimes get the right answer. * language models with the ability to write instructions for other language models (ie. writing a project plan, and then completing each item of the plan, and then putting the results together). * language models with the ability to use tools (ie. 'run valgrind, tell the model what it says, and then the model will modify the code to fix the valgrind error') * language models with the ability to summarize large things to small. * language models with the ability to review existing work and see what needs changing to meet a goal, including chucking out work that isn't right/fit for purpose. With all these pieces, it really seems that with enough compute/budget, we are awfully close...
    - gwoolhurme 3y ago
      That seems to also miss the intentionality that goes into some things in the kernel as well… I understand now you mean when a feedback of LLMs are improved on. I guess fair enough there, no idea if that will work till we see it. However I think the problem of a Unix-like kernel is a lot less trivial due to the human intentionality that goes into some choices as well as bit-banging optimization.
      - londons_explore 3y ago
        > human intentionality that goes into some choices Many choices are made at design time to make the right tradeoffs between complexity, speed, etc. But with AI-designed things, complexity is no longer an issue as long as the AI understands it, and you no longer need to think too much about speed - just implement 100 different designs and pick the one which does best on a set of benchmarks also designed by the AI.
        kaba0 3y ago
        Current AIs can’t reason about trivial, 2 years old human cognitive things, let alone multi-million lines of code bases.
  - jakear 3y ago
    Personally I couldn't even get it to draw me a regular pentagon in CSS. It was happy to draw hexagons and call them pentagons. It was even happy to go back and fix its mistakes when I informed it that it was making hexagons, not pentagons. It, of course, readily accepted that I was correct and it had indeed drawn a hexagon, but this time it'd be different. This time, it'd draw a pentagon. And... repeat.
    - im3w1l 3y ago
      It really struggles with arithmetic, so that's kind of a worst case problem for it though.
      - jakear 3y ago
        Sure, but understanding you need to have 5 vertices to make a pentagon isn't exactly high-brow. Certainly not compared to making a Unix kernel from scratch.
        im3w1l 3y ago
        I don't think high-brow / low-brow is a useful framework for understanding these models. They are good at certain things and bad at others and those don't correspond neatly to what a human finds easy and hard.
        deleted 3y ago
        [deleted]
  - svaha1728 3y ago
    I agree. You can also go to to github.com/torvalds/linux click on code -> Download zip and have the Linux Kernel. I’m not sure what having ChatGPT write it for you would give you. Maybe 10,000 hours of frustration after you realize how far from the mark you really are?
- kaba0 3y ago
  > allowing it to use debuggers, benchmarks, and sanitizers, allowing it to write its own tools, and then put modules together That’s a huuuge if whether it can meaningfully reason about such. GPT-4 has improved a lot over 3, but I really wouldn’t call its capabilities reasoning at all. We often under-appreciate human intelligence
- lamp987 3y ago
  as a hobby osdever, this: >I really don't see such a thing being far off. GPT-4 is already pretty good at writing small modules ("write a disk IO queue for my custom kernel in C"). is completely delusional.
- chaxor 3y ago
  I tend to agree, though with perhaps a much larger tolerance in the number of years "not far off" is. What may be interesting is seeing the emergence of hardware specific bootloader's, kernels and OS, rather than using boot/Linux/etc with it's support for all different SoCs, etc. Rust embedded code for specific hardware seems like a regression, but it would make things remarkably smaller codebase for specific app deployment.
gwoolhurme 3y ago
Is this that hard of a tool? Why is it a package? Seems like something someone with a few lines of code could do. OpenAI is transparent and fairly open about pricing per token… it seems like it’s just adding bloat.
pixl97 3y ago
>I am really curious what it would cost, as a theoretical lower bound, to pay OpenAI to write the Linux Kernel Billions of dollars. Eh, what? Why in the living hell would it cost that much! Because the actual Linux kernel cost that much or more. Of course most of these costs were volunteer labor or born by someone testing out a failure on their own companies time. To greenfield an OS kernel as comprehensive as Linux and with as much hardware support as linux would be one of the more expensive human endeavors ever. But wait, why doesn't Windows cost this much? Oh, but it did. Of course Microsoft has thrown a huge amount of the costs of on to users and hardware development companies too.

londons_explore 3y ago

> No one here (including the author) is under the impression that you can just ask Chat GPT to "write a Unix-like kernel" and get Linux spit out the other side. I really don't see such a thing being far off. GPT-4 is already pretty good at writing small modules ("write a disk IO queue for my custom kernel in C"). With a little more work in allowing GPT-4 to test out code it has written and iteratively make changes, allowing it to use debuggers, benchmarks, and sanitizers, allowing it to write its own tools, and then put modules together, I think we could very soon be asking it to "write me a Unix-like kernel".

gwoolhurme 3y ago
It’s that good in your experience? Enough that it is close to writing something as nice as the Linux Kernel we have now? How close is close? Reading comments like these makes me feel like I’m swallowing crazy pills. It legitimately makes me feel like I’m using it wrong. I do Android and iOS native development and some IoT and it gives me really wrong code very often. To the point that I don’t see much difference between chatgpt and gpt4. But you say it’s close to just “write me a unix-like kernel”
- starbugs 3y ago
  It's not close and I also see no big difference between GPT 3.5 and 4. Don't get hyped. I am sure it will eventually happen, but calling it close is very optimistic.
- londons_explore 3y ago
  Not close to as in "it can nearly write it correctly", but close to as in "I believe within a small number of years, gpt-4-like tools would be able to write you a unix-like kernel from scratch and have it actually work, with no human input". I think we already have most of the pieces in place: * big language models that sometimes get the right answer. * language models with the ability to write instructions for other language models (ie. writing a project plan, and then completing each item of the plan, and then putting the results together). * language models with the ability to use tools (ie. 'run valgrind, tell the model what it says, and then the model will modify the code to fix the valgrind error') * language models with the ability to summarize large things to small. * language models with the ability to review existing work and see what needs changing to meet a goal, including chucking out work that isn't right/fit for purpose. With all these pieces, it really seems that with enough compute/budget, we are awfully close...
  - gwoolhurme 3y ago
    That seems to also miss the intentionality that goes into some things in the kernel as well… I understand now you mean when a feedback of LLMs are improved on. I guess fair enough there, no idea if that will work till we see it. However I think the problem of a Unix-like kernel is a lot less trivial due to the human intentionality that goes into some choices as well as bit-banging optimization.
    - londons_explore 3y ago
      > human intentionality that goes into some choices Many choices are made at design time to make the right tradeoffs between complexity, speed, etc. But with AI-designed things, complexity is no longer an issue as long as the AI understands it, and you no longer need to think too much about speed - just implement 100 different designs and pick the one which does best on a set of benchmarks also designed by the AI.
      - kaba0 3y ago
        Current AIs can’t reason about trivial, 2 years old human cognitive things, let alone multi-million lines of code bases.
- jakear 3y ago
  Personally I couldn't even get it to draw me a regular pentagon in CSS. It was happy to draw hexagons and call them pentagons. It was even happy to go back and fix its mistakes when I informed it that it was making hexagons, not pentagons. It, of course, readily accepted that I was correct and it had indeed drawn a hexagon, but this time it'd be different. This time, it'd draw a pentagon. And... repeat.
  - im3w1l 3y ago
    It really struggles with arithmetic, so that's kind of a worst case problem for it though.
    - jakear 3y ago
      Sure, but understanding you need to have 5 vertices to make a pentagon isn't exactly high-brow. Certainly not compared to making a Unix kernel from scratch.
      - im3w1l 3y ago
        I don't think high-brow / low-brow is a useful framework for understanding these models. They are good at certain things and bad at others and those don't correspond neatly to what a human finds easy and hard.
        deleted 3y ago
        [deleted]
- svaha1728 3y ago
  I agree. You can also go to to github.com/torvalds/linux click on code -> Download zip and have the Linux Kernel. I’m not sure what having ChatGPT write it for you would give you. Maybe 10,000 hours of frustration after you realize how far from the mark you really are?
kaba0 3y ago
> allowing it to use debuggers, benchmarks, and sanitizers, allowing it to write its own tools, and then put modules together That’s a huuuge if whether it can meaningfully reason about such. GPT-4 has improved a lot over 3, but I really wouldn’t call its capabilities reasoning at all. We often under-appreciate human intelligence
lamp987 3y ago
as a hobby osdever, this: >I really don't see such a thing being far off. GPT-4 is already pretty good at writing small modules ("write a disk IO queue for my custom kernel in C"). is completely delusional.
chaxor 3y ago
I tend to agree, though with perhaps a much larger tolerance in the number of years "not far off" is. What may be interesting is seeing the emergence of hardware specific bootloader's, kernels and OS, rather than using boot/Linux/etc with it's support for all different SoCs, etc. Rust embedded code for specific hardware seems like a regression, but it would make things remarkably smaller codebase for specific app deployment.

gwoolhurme 3y ago

Is this that hard of a tool? Why is it a package? Seems like something someone with a few lines of code could do. OpenAI is transparent and fairly open about pricing per token… it seems like it’s just adding bloat.

pixl97 3y ago

>I am really curious what it would cost, as a theoretical lower bound, to pay OpenAI to write the Linux Kernel Billions of dollars. Eh, what? Why in the living hell would it cost that much! Because the actual Linux kernel cost that much or more. Of course most of these costs were volunteer labor or born by someone testing out a failure on their own companies time. To greenfield an OS kernel as comprehensive as Linux and with as much hardware support as linux would be one of the more expensive human endeavors ever. But wait, why doesn't Windows cost this much? Oh, but it did. Of course Microsoft has thrown a huge amount of the costs of on to users and hardware development companies too.