6 ms·

This caught my attention as I found it implausible: > One DeepMind engineer even reported being able to convince ChatGPT that it was a Linux terminal and getti

This caught my attention as I found it implausible: > One DeepMind engineer even reported being able to convince ChatGPT that it was a Linux terminal and getting it to run some simple mathematical code to compute the first 10 prime numbers. Remarkably, it could finish the task faster than the same code running on a real Linux machine. Following the link, there's a screenshot to a screenshot [0] of a code-golf solution to finding primes which is quite inefficient, and the author notes > I want to note here that this codegolf python implementation to find prime numbers is very inefficient. It takes 30 seconds to evaluate the command on my machine, but it only takes about 10 seconds to run the same command on ChatGPT. So, for some applications, this virtual machine is already faster than my laptop. So it's not quite calculating primes; more likely it recognizes the code as being code to do so, and recites the numbers from memory. That's interesting in its own right, but we won't be running Python on an LLM for a performance boost any time soon. In my experience this interpreting is apparent as a limitation of the model when it keeps insisting on broken code being correct, or having its mistake pointed out, then apologizing, saying it's got some new code that fixes the issue, and proceeding to output the exact same code. [0] https://www.engraved.blog/content/images/2022/12/image-13.png https://www.engraved.blog/content/images/2022/12/image-13.pn...

namaria 4y ago
So on the one hand, these newly publicized models can render convincing representations of realities we used to get from deterministic processes. On the other hand, it's probabilistic and fails to conform to logic quite often, and in a confident way. We're building systems capable of programing computers non-deterministically. I think this is huge. But not because ChatGPT23 will be a CEO or a politician. But because this is a paradigm shift in compute similar to moving from integrator machines to general computers. I don't think LLMs will make programmers obsolete. I think large enough models will make programming something completely different from what it is now. The days of sequencing tokens for compilers/interpreters seem to be drawing to an end as the dominant way of specifying software products.
- tosser0001 4y ago
  I'm sort of glad to be retiring soon. I have a feeling everything I enjoy about programming is going to be going away.
  - namaria 4y ago
    I get the feeling. But I've always enjoyed the abstract puzzles more then anything. Computers attracted me as a form of very complex abstract puzzles. But when it comes down to it everything in life is just nth dimensional tensor puzzles. What I really cherish computers for giving me is fast and clear feedback.
  - anon7725 4y ago
    I'm about 10-15 years from retiring, and lately, I've been thinking a lot about how to handle the rest of my career with all these new AI advancements.
    - 13years 4y ago
      This disruption does appear to be different than others prior. As it is not a narrow disruption with limited boundaries for which we can plan and organize our lives for some stabilization period to follow. Instead, it is constant and accelerating destabilization. Hundreds of AI projects attempting to ride this accelerating wave were essentially just made obsolete yesterday - https://www.youtube.com/watch?v=DH-2BHDYNfk https://www.youtube.com/watch?v=DH-2BHDYNfk I feel the excitement is going to very soon turn into frustration of attempting to remain relevant ahead of the accelerating technological curve. Humans need periods of stabilization to plan and reason about their lives.
- sho_hn 4y ago
  The LLM can act as a global cache for common solutions to common problems, with the ability to perform the integration work necessary to apply them. That prime number example is a little bit like when you put a functools.lru_cache decorator on a function in Python. It's faster than computing the function call because it's able to recall the return value for the parameters from the cache "memory". Of course, many skilled programmers are also mainly used a cache for common solutions to common problems organizations have in the programming domain. As humans we can derive satisfaction from being able to tackle the same task others can, as a confirmation of our own progress. We like "doing it ourselves". But globally that's not super interesting if lots of people are constantly re-developing the same solutions to the same problems for their own benefit. I guess that's the push-and-pull (and the anxiety generator).
truculent 4y ago
So it sounds like the mechanism is something like stochastic memoisation?
- pedrovhb 4y ago
  I don't think "memoisation" is an accurate word for this; it implies doing the computation once, and storing the result in cache to return later. It's more like replacing your Python code with a SQL query of the LLM's understanding of what it's supposed to do, which may or may not be correct, and executing it on its "database" of knowledge, which may or may not have the correct data.
- mistrial9 4y ago
  this is new to me but.. a quick read of Wikipedia [1] later, it appears that this decades-old method takes a goal, and then uses branching and recording to find paths that differ the least from the goal. The article mentions that the curse of dimensionality is so bad here that approximations are often used in practice. Does that capture it? [1] https://en.wikipedia.org/wiki/Stochastic_dynamic_programming https://en.wikipedia.org/wiki/Stochastic_dynamic_programming
HervalFreire 4y ago
[flagged]
- christiangenco 4y ago
  Rude.
  - HervalFreire 4y ago
    [flagged]
- mostlysimilar 4y ago
  Equally interesting is the psychology of people who take the time to write long posts borderline gloating about tools making software engineers obsolete.
  - HervalFreire 4y ago
    Except I didn't write that and I will quote myself: >"I can tell you this. I do not know future iterations of LLMs can take over our jobs" I wrote that I don't know which is the best answer we all have at this point. Given the evidence, completely denying it, as many have is simply not realistic.
    - mostlysimilar 4y ago
      Okay, but you have another post in this thread with: > When these LLMs get normalized probably 5 years from now I'm going go back to these old threads and contact these people who are in self denial and throw it in their face. I'll just link this comment and be like I TOLD YOU, I TOLD YOU, YOU WERE WRONG.
      - HervalFreire 4y ago
        My bad. It's the caps. I'll remove the caps. Just picture me saying it a nicer way.
jmvoodoo 4y ago
What's most interesting to me, is that this is how I would expect a human to approach the problem if presented with the code and asked for the output. The LLM didn't run the code, it tried to predict the output based on its knowledge of python and primes.
- shrikant 4y ago
  When I was a bored and under-challenged student in the early days of university, one of my tests during a programming exam was to write a program that spit out the factorial of an input integer. For shits and giggles, I just just wrote a series of `if...then` statements for 0 to 5, and only actually calculated the factorial for inputs >=6. I passed after the examiner just input 3 and 5 and was satisfied that the output was correct. Oops.
  - WalterBright 4y ago
    Doing a table lookup for common cases and computing for the less common ones is perfectly valid!
    - Jeff_Brown 4y ago
      And for some problems it's even efficient. Not factorial, though.
      - Jeff_Brown 4y ago
        Oh you're right, hive mind, it's efficient. Sorry.
    - dxbydt 4y ago
      Not 100% sure but I believe this is how we landed the lunar module on the moon the first time...tan/arctan/both were too hard to compute on the processors of those days so they discretized into half angles & stored the tangents in a lookup table.
      - 4y ago
chias 4y ago
So here are a few screenshots that I personally took after telling it that it was a linux terminal: Correctly produces a convincing output for having decoded the base64-encoding of "ping google.com" and then piping it to bash: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/446/423/169/460/099/original/be1acd9e8a612a10.png https://media.infosec.exchange/infosecmedia/media_attachment... Similar command, but with a garbage domain it hasn't seen before, and a less well-known domain. It produced convincing output in both cases: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/446/441/504/062/129/original/9ae51717c9706043.png https://media.infosec.exchange/infosecmedia/media_attachment... Having it just output a base64 decoding of an obviously unique string. Fascinatingly, it tried to correct typos that I intentionally included: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/446/486/163/988/815/original/7c38c28cc5b482fb.png https://media.infosec.exchange/infosecmedia/media_attachment... This was also pretty cool -- ask it to start a webserver then curl it: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/453/237/416/601/399/original/c91a7c9ee12bc83f.png https://media.infosec.exchange/infosecmedia/media_attachment... Telling it that it is a python interpreter and calling self-defined functions: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/453/859/910/399/619/original/3e27b3607026288f.png https://media.infosec.exchange/infosecmedia/media_attachment... A slightly more complicated function: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/453/866/420/561/434/original/38d4a02d0a01b7ab.png https://media.infosec.exchange/infosecmedia/media_attachment... I did a few more experiments including generating large factorial numbers that took a long time on my laptop but it responded accurately to a much larger length than my laptop could do (though these were only accurate to the first few hundred digits)
- ambicapter 4y ago
  Lol why does ChatGPT hit ^C? Do you think it’s getting bored of waiting to respond to you and decides that’s enough time spent on an answer? Edit: the base64 decoding examples are terrifying. I have no idea how that works. Edit2: actually I can sort of see how that works, b64 encoding doesn’t have any obfuscation to it, I could see how an ML model can build a pretty good approximation over time after seeing all the examples on the internet.
thexumaker 4y ago
yep I tried asking chatgpt to optimize some SQL queries with a heavy amount of full outer joins. The optimization I was trying to push it to was specifically adding an index on a column or filtering the first table that we join on but it kept creating SQL subqueries and switching the full joins to left joins no matter what I told it and the sql errors I sent it
skybrian 4y ago
Large language models are storytellers. To write a story about someone using a computer terminal, there are things it’s helpful to know, but there are easier ways than simulating a computer. Since we don’t know how it works, we should be open to the possibility that it’s using all sorts of storytelling techniques.
roflyear 4y ago
Gpt is never calculating anything!!
FormulatedEdits 4y ago
Its speed as compared to the laptop - I assume it was running online, on OpenAI’s hardware. That hardware is quite extensive, I thought. I believe I’ve seen the cost of one (not ‘search’, what’s the word here, ‘output’?) is something like 1-10 cents.
shawntan 4y ago
I was personally quite appalled that so many people believed this was what it was doing, but I suppose since the source was a DeepMind researcher, it lends a lot of credibility. By the same reasoning, I hope people do realise that there are computations that are just impossible given the finite depth in a Transformer. The only possible way to overcome this in the current paradigm is the Chain-of-Thought related methods.
Imnimo 4y ago
Yeah, I tried asking GPT-4 to write a simple Python program to find the millionth prime. It wrote the naive approach where you loop over numbers and for each number, check all of the possible divisors up to its square root, and then keep track of how many primes you've found. Obviously, this takes quite a while to execute for N = a million. And yet, if you ask GPT-4 what the output of the code will be, as if my magic (memorization) it immediately spits out 15485863. It makes it a lot more clear that it's not actually simulating the code.

namaria 4y ago

So on the one hand, these newly publicized models can render convincing representations of realities we used to get from deterministic processes. On the other hand, it's probabilistic and fails to conform to logic quite often, and in a confident way. We're building systems capable of programing computers non-deterministically. I think this is huge. But not because ChatGPT23 will be a CEO or a politician. But because this is a paradigm shift in compute similar to moving from integrator machines to general computers. I don't think LLMs will make programmers obsolete. I think large enough models will make programming something completely different from what it is now. The days of sequencing tokens for compilers/interpreters seem to be drawing to an end as the dominant way of specifying software products.

tosser0001 4y ago
I'm sort of glad to be retiring soon. I have a feeling everything I enjoy about programming is going to be going away.
- namaria 4y ago
  I get the feeling. But I've always enjoyed the abstract puzzles more then anything. Computers attracted me as a form of very complex abstract puzzles. But when it comes down to it everything in life is just nth dimensional tensor puzzles. What I really cherish computers for giving me is fast and clear feedback.
- anon7725 4y ago
  I'm about 10-15 years from retiring, and lately, I've been thinking a lot about how to handle the rest of my career with all these new AI advancements.
  - 13years 4y ago
    This disruption does appear to be different than others prior. As it is not a narrow disruption with limited boundaries for which we can plan and organize our lives for some stabilization period to follow. Instead, it is constant and accelerating destabilization. Hundreds of AI projects attempting to ride this accelerating wave were essentially just made obsolete yesterday - https://www.youtube.com/watch?v=DH-2BHDYNfk https://www.youtube.com/watch?v=DH-2BHDYNfk I feel the excitement is going to very soon turn into frustration of attempting to remain relevant ahead of the accelerating technological curve. Humans need periods of stabilization to plan and reason about their lives.
sho_hn 4y ago
The LLM can act as a global cache for common solutions to common problems, with the ability to perform the integration work necessary to apply them. That prime number example is a little bit like when you put a functools.lru_cache decorator on a function in Python. It's faster than computing the function call because it's able to recall the return value for the parameters from the cache "memory". Of course, many skilled programmers are also mainly used a cache for common solutions to common problems organizations have in the programming domain. As humans we can derive satisfaction from being able to tackle the same task others can, as a confirmation of our own progress. We like "doing it ourselves". But globally that's not super interesting if lots of people are constantly re-developing the same solutions to the same problems for their own benefit. I guess that's the push-and-pull (and the anxiety generator).

truculent 4y ago

So it sounds like the mechanism is something like stochastic memoisation?

pedrovhb 4y ago
I don't think "memoisation" is an accurate word for this; it implies doing the computation once, and storing the result in cache to return later. It's more like replacing your Python code with a SQL query of the LLM's understanding of what it's supposed to do, which may or may not be correct, and executing it on its "database" of knowledge, which may or may not have the correct data.
mistrial9 4y ago
this is new to me but.. a quick read of Wikipedia [1] later, it appears that this decades-old method takes a goal, and then uses branching and recording to find paths that differ the least from the goal. The article mentions that the curse of dimensionality is so bad here that approximations are often used in practice. Does that capture it? [1] https://en.wikipedia.org/wiki/Stochastic_dynamic_programming https://en.wikipedia.org/wiki/Stochastic_dynamic_programming

HervalFreire 4y ago

[flagged]

christiangenco 4y ago
Rude.
- HervalFreire 4y ago
  [flagged]
mostlysimilar 4y ago
Equally interesting is the psychology of people who take the time to write long posts borderline gloating about tools making software engineers obsolete.
- HervalFreire 4y ago
  Except I didn't write that and I will quote myself: >"I can tell you this. I do not know future iterations of LLMs can take over our jobs" I wrote that I don't know which is the best answer we all have at this point. Given the evidence, completely denying it, as many have is simply not realistic.
  - mostlysimilar 4y ago
    Okay, but you have another post in this thread with: > When these LLMs get normalized probably 5 years from now I'm going go back to these old threads and contact these people who are in self denial and throw it in their face. I'll just link this comment and be like I TOLD YOU, I TOLD YOU, YOU WERE WRONG.
    - HervalFreire 4y ago
      My bad. It's the caps. I'll remove the caps. Just picture me saying it a nicer way.

jmvoodoo 4y ago

What's most interesting to me, is that this is how I would expect a human to approach the problem if presented with the code and asked for the output. The LLM didn't run the code, it tried to predict the output based on its knowledge of python and primes.

shrikant 4y ago
When I was a bored and under-challenged student in the early days of university, one of my tests during a programming exam was to write a program that spit out the factorial of an input integer. For shits and giggles, I just just wrote a series of `if...then` statements for 0 to 5, and only actually calculated the factorial for inputs >=6. I passed after the examiner just input 3 and 5 and was satisfied that the output was correct. Oops.
- WalterBright 4y ago
  Doing a table lookup for common cases and computing for the less common ones is perfectly valid!
  - Jeff_Brown 4y ago
    And for some problems it's even efficient. Not factorial, though.
    - Jeff_Brown 4y ago
      Oh you're right, hive mind, it's efficient. Sorry.
  - dxbydt 4y ago
    Not 100% sure but I believe this is how we landed the lunar module on the moon the first time...tan/arctan/both were too hard to compute on the processors of those days so they discretized into half angles & stored the tangents in a lookup table.
    - 4y ago

chias 4y ago

So here are a few screenshots that I personally took after telling it that it was a linux terminal: Correctly produces a convincing output for having decoded the base64-encoding of "ping google.com" and then piping it to bash: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/446/423/169/460/099/original/be1acd9e8a612a10.png https://media.infosec.exchange/infosecmedia/media_attachment... Similar command, but with a garbage domain it hasn't seen before, and a less well-known domain. It produced convincing output in both cases: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/446/441/504/062/129/original/9ae51717c9706043.png https://media.infosec.exchange/infosecmedia/media_attachment... Having it just output a base64 decoding of an obviously unique string. Fascinatingly, it tried to correct typos that I intentionally included: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/446/486/163/988/815/original/7c38c28cc5b482fb.png https://media.infosec.exchange/infosecmedia/media_attachment... This was also pretty cool -- ask it to start a webserver then curl it: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/453/237/416/601/399/original/c91a7c9ee12bc83f.png https://media.infosec.exchange/infosecmedia/media_attachment... Telling it that it is a python interpreter and calling self-defined functions: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/453/859/910/399/619/original/3e27b3607026288f.png https://media.infosec.exchange/infosecmedia/media_attachment... A slightly more complicated function: https://media.infosec.exchange/infosecmedia/media_attachments/files/109/453/866/420/561/434/original/38d4a02d0a01b7ab.png https://media.infosec.exchange/infosecmedia/media_attachment... I did a few more experiments including generating large factorial numbers that took a long time on my laptop but it responded accurately to a much larger length than my laptop could do (though these were only accurate to the first few hundred digits)

ambicapter 4y ago
Lol why does ChatGPT hit ^C? Do you think it’s getting bored of waiting to respond to you and decides that’s enough time spent on an answer? Edit: the base64 decoding examples are terrifying. I have no idea how that works. Edit2: actually I can sort of see how that works, b64 encoding doesn’t have any obfuscation to it, I could see how an ML model can build a pretty good approximation over time after seeing all the examples on the internet.

thexumaker 4y ago

yep I tried asking chatgpt to optimize some SQL queries with a heavy amount of full outer joins. The optimization I was trying to push it to was specifically adding an index on a column or filtering the first table that we join on but it kept creating SQL subqueries and switching the full joins to left joins no matter what I told it and the sql errors I sent it

skybrian 4y ago

Large language models are storytellers. To write a story about someone using a computer terminal, there are things it’s helpful to know, but there are easier ways than simulating a computer. Since we don’t know how it works, we should be open to the possibility that it’s using all sorts of storytelling techniques.

roflyear 4y ago

Gpt is never calculating anything!!

FormulatedEdits 4y ago

Its speed as compared to the laptop - I assume it was running online, on OpenAI’s hardware. That hardware is quite extensive, I thought. I believe I’ve seen the cost of one (not ‘search’, what’s the word here, ‘output’?) is something like 1-10 cents.

shawntan 4y ago

I was personally quite appalled that so many people believed this was what it was doing, but I suppose since the source was a DeepMind researcher, it lends a lot of credibility. By the same reasoning, I hope people do realise that there are computations that are just impossible given the finite depth in a Transformer. The only possible way to overcome this in the current paradigm is the Chain-of-Thought related methods.

Imnimo 4y ago

Yeah, I tried asking GPT-4 to write a simple Python program to find the millionth prime. It wrote the naive approach where you loop over numbers and for each number, check all of the possible divisors up to its square root, and then keep track of how many primes you've found. Obviously, this takes quite a while to execute for N = a million. And yet, if you ask GPT-4 what the output of the code will be, as if my magic (memorization) it immediately spits out 15485863. It makes it a lot more clear that it's not actually simulating the code.