4 ms·

My interest in LLMs lies in the static program analysis space and these results are genuinely exciting. LLM-guided analysis (e.g. invariant modelling or detecti

My interest in LLMs lies in the static program analysis space and these results are genuinely exciting. LLM-guided analysis (e.g. invariant modelling or detecting bugs by identifying unusual implementation patterns) is going to be a thing. OpenAI is gradually turning into the Fisher-Price of AI. I cancelled my ChatGPT Pro subscription after 12mo due to increasingly low quality answers since ~Nov 2023. I occasionally try the API but the quality there is only slightly better than ChatGPT.

deleted 3y ago
[deleted]
jimmywetnips 3y ago
I keep trying alternatives to gpt 4 based on people shitting on it and you know what, every single time I try something else, I hit a bruh moment on the first try. I asked Gemini v1 2 days ago about programming and the first response was some nonsense about elections being a complex topic. if gpt 4 is Fisher Price then it's competitors are ill fitting kinder egg toys with missing pieces. https://i.imgur.com/gxh0mUZ.jpeg https://i.imgur.com/gxh0mUZ.jpeg
- Jensson 3y ago
  Gemini in their interface has overly strong political correctness and safety which creates such strange responses, but if you avoid triggering that then it is pretty good. Edit: And in your screenshot you did get real responses, you see gemini generates several look at response 2, looks like what you wanted.
  - scotty79 3y ago
    It's so sad that we got AIs only after everybody got so sensitive about some utterances. I think the language itself was never supposed to be that serious as we are currently making it.
  - darepublic 3y ago
    look forward to an LLM that can simulate talking to a mature highly knowledgable adult and not a painfully grinning corporate mannequin.
  - jimmywetnips 3y ago
    I did see the other examples. But the fact is, a question about double metaphone that is 99% code should not trigger some tangent about politics. Not once in my thousands of questions to gpt did it go so far off base. Maybe in the gpt3 days sometimes it would get caught in a repeating character loop. Yet my fourth or so time back to Bard and it shits the bed on the first try makes me think, yeah... Still kinder surprise Also, the suggested changes from the remaining responses are incredibly naive. It suggests pulling numbers out and appending them back in, which breaks if the numbers aren't simply at the end of the strings, as well as not going with the pronunciation spirit of metaphone. "Google search for election info" https://g.co/gemini/share/7afd63de0079 https://g.co/gemini/share/7afd63de0079
- spaceman_2020 3y ago
  Using raw ChatGPT for complex coding queries is just poor tool use. Custom GPTs exist for a reason. Grimoire (custom GPT) outperforms raw ChatGPT by miles. My workflow is usually Grimoire to get a broad overview of things, then a model that can use my repo via tools like Cursor.
  - jimmywetnips 3y ago
    Ooo good to know. I'll check that out.
kolinko 3y ago
+1. I built a smart contract decompiler (eveem.org), and got deep into formal verification / code analysis path during that time. There was a ton more I could do back then if I had access to gpt-4. Some analysis methods were extremely slow using solvers, and it was a pita to write code for dedicated code pattern recognition. Gpt4 improves that, and it's easy to verify if it's guesses are correct using formal methods.
- ryanjshaw 3y ago
  Nice, I've used your decompiler! The smart contract decompilation/binary lifting space is pretty active and I'm not sure how much low hanging fruit is left there. But there are now active bug bounty/audit contests in this space, so my focus is on projects where I already have the source. I'm exploring some unconventional domain-specific ideas that leverage ASTs and (eventually) LLMs. That being said, I recently read a paper whose authors made use of traditional statistical analysis to find unusual implementation patterns in an IR decompilation of smart contracts. I wonder if custom LLMs can be used to perform that analysis.
qsort 3y ago
Agreed. I'm very much not on the LLM hype-train, but I've been saying from day one that I'd really like something like this. Code has strong local similarities, so it's not far fetched to imagine that future LLMs -- if not current ones -- could be used as a stochastic linter of sorts. Programming is hard, it's not insane to make use of all the help we can get.
flessner 3y ago
I feel the same, ChatGPT barely beats Google searches for me nowadays. For example, on a side project I used to explain it shortly and the issue I am facing, mostly looking to discover packages or libraries that can help me build it. Over 3 months ago GPT4 used to jump straight to the point and recommend alternative scenarios and packages. Now it just gives me a giant text with considerations for the whole project, but rarely actually tackles the issue I am having. Even if I ask it to actually go in-depth into whatever I am interested in it usually doesn't and repeats the same surface level information.
- rajnathani 3y ago
  > I feel the same, ChatGPT barely beats Google searches for me nowadays. I started feeling similar recently, but then I added the "Custom Instructions" [0] feature in ChatGPT, and the results are much better. The instruction that I've put there is that I'm a highly information-aware person and that I require detailed answers. The reason that I feel this works is that GPT is essentially trained (caveat: the RLHF part could partially falsify my "theory" here) on internet textual data whereby the usual method of answering on internet forums (and other sites) has a relatively low-to-moderate detailed way of answering things; but given that we know that GPT has in its latent space pretty much has all the useful knowledge on the internet, that if one asks for very detailed (information diverse/inclusive) answers by default that it can perform really well. [0] https://openai.com/blog/custom-instructions-for-chatgpt https://openai.com/blog/custom-instructions-for-chatgpt

deleted 3y ago

[deleted]

jimmywetnips 3y ago

I keep trying alternatives to gpt 4 based on people shitting on it and you know what, every single time I try something else, I hit a bruh moment on the first try. I asked Gemini v1 2 days ago about programming and the first response was some nonsense about elections being a complex topic. if gpt 4 is Fisher Price then it's competitors are ill fitting kinder egg toys with missing pieces. https://i.imgur.com/gxh0mUZ.jpeg https://i.imgur.com/gxh0mUZ.jpeg

Jensson 3y ago
Gemini in their interface has overly strong political correctness and safety which creates such strange responses, but if you avoid triggering that then it is pretty good. Edit: And in your screenshot you did get real responses, you see gemini generates several look at response 2, looks like what you wanted.
- scotty79 3y ago
  It's so sad that we got AIs only after everybody got so sensitive about some utterances. I think the language itself was never supposed to be that serious as we are currently making it.
- darepublic 3y ago
  look forward to an LLM that can simulate talking to a mature highly knowledgable adult and not a painfully grinning corporate mannequin.
- jimmywetnips 3y ago
  I did see the other examples. But the fact is, a question about double metaphone that is 99% code should not trigger some tangent about politics. Not once in my thousands of questions to gpt did it go so far off base. Maybe in the gpt3 days sometimes it would get caught in a repeating character loop. Yet my fourth or so time back to Bard and it shits the bed on the first try makes me think, yeah... Still kinder surprise Also, the suggested changes from the remaining responses are incredibly naive. It suggests pulling numbers out and appending them back in, which breaks if the numbers aren't simply at the end of the strings, as well as not going with the pronunciation spirit of metaphone. "Google search for election info" https://g.co/gemini/share/7afd63de0079 https://g.co/gemini/share/7afd63de0079
spaceman_2020 3y ago
Using raw ChatGPT for complex coding queries is just poor tool use. Custom GPTs exist for a reason. Grimoire (custom GPT) outperforms raw ChatGPT by miles. My workflow is usually Grimoire to get a broad overview of things, then a model that can use my repo via tools like Cursor.
- jimmywetnips 3y ago
  Ooo good to know. I'll check that out.

kolinko 3y ago

+1. I built a smart contract decompiler (eveem.org), and got deep into formal verification / code analysis path during that time. There was a ton more I could do back then if I had access to gpt-4. Some analysis methods were extremely slow using solvers, and it was a pita to write code for dedicated code pattern recognition. Gpt4 improves that, and it's easy to verify if it's guesses are correct using formal methods.

ryanjshaw 3y ago
Nice, I've used your decompiler! The smart contract decompilation/binary lifting space is pretty active and I'm not sure how much low hanging fruit is left there. But there are now active bug bounty/audit contests in this space, so my focus is on projects where I already have the source. I'm exploring some unconventional domain-specific ideas that leverage ASTs and (eventually) LLMs. That being said, I recently read a paper whose authors made use of traditional statistical analysis to find unusual implementation patterns in an IR decompilation of smart contracts. I wonder if custom LLMs can be used to perform that analysis.

qsort 3y ago

Agreed. I'm very much not on the LLM hype-train, but I've been saying from day one that I'd really like something like this. Code has strong local similarities, so it's not far fetched to imagine that future LLMs -- if not current ones -- could be used as a stochastic linter of sorts. Programming is hard, it's not insane to make use of all the help we can get.

flessner 3y ago

I feel the same, ChatGPT barely beats Google searches for me nowadays. For example, on a side project I used to explain it shortly and the issue I am facing, mostly looking to discover packages or libraries that can help me build it. Over 3 months ago GPT4 used to jump straight to the point and recommend alternative scenarios and packages. Now it just gives me a giant text with considerations for the whole project, but rarely actually tackles the issue I am having. Even if I ask it to actually go in-depth into whatever I am interested in it usually doesn't and repeats the same surface level information.

rajnathani 3y ago
> I feel the same, ChatGPT barely beats Google searches for me nowadays. I started feeling similar recently, but then I added the "Custom Instructions" [0] feature in ChatGPT, and the results are much better. The instruction that I've put there is that I'm a highly information-aware person and that I require detailed answers. The reason that I feel this works is that GPT is essentially trained (caveat: the RLHF part could partially falsify my "theory" here) on internet textual data whereby the usual method of answering on internet forums (and other sites) has a relatively low-to-moderate detailed way of answering things; but given that we know that GPT has in its latent space pretty much has all the useful knowledge on the internet, that if one asks for very detailed (information diverse/inclusive) answers by default that it can perform really well. [0] https://openai.com/blog/custom-instructions-for-chatgpt https://openai.com/blog/custom-instructions-for-chatgpt