23 ms·

That's exactly what I want to do next. Any tips on how should I get codebases and real world PRs? Are the ones on popular open source repos on GitHub sufficien

That's exactly what I want to do next. Any tips on how should I get codebases and real world PRs? Are the ones on popular open source repos on GitHub sufficient? I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias. Secondly, after getting all this, how do I evaluate which method gave better results? Should it be done by a human, or should I just plug an LLM to check?

deleted 2y ago
[deleted]
jarebear6expepj 2y ago
“…should it be done by a human?” Sigh.
- namanyayg 2y ago
  I'll do it personally in the beginning but was thinking about doing it on scale
  - deleted 2y ago
    [deleted]
  - JTyQZSnP3cQGa8B 2y ago
    > doing it on scale Like cloud-scale, no-code scale, or NoSQL scale? You are confused, which shows that, maybe, you should not be using such tools with the experience that you don't have.
    - zBard 2y ago
      ‘Like cloud-scale, no-code scale or NoSQL scale’. That is the dumbest statement I have heard this week. You should perhaps refrain from commenting, at-least until you gain the modicum of intelligence that you currently don’t have.
smusamashah 2y ago
Very first thing you can tell us (or try if you haven't) is that if you re-prompt, does it give the same answer? Second can you get it to generate (consistently and repeatedly) the text that gp pointed out? Don't need to switch to a different repo for quick test, just make it reproable on your current repo.
- namanyayg 2y ago
  Not the exact text, but still decent quality. I'll play around with temperature and prompts a bit.
QuadmasterXLII 2y ago
If you could get an LLM to check, you could just spam solutions with any assortment of models and then use your checker to pick the best.
- Yoric 2y ago
  Except LLMs are usually really bad at checking. Perhaps if they're some day augmented by formal methods, that might change.
  - QuadmasterXLII 2y ago
    Sorry, I was unclear- my statement was supposed to be proof that LLMs are bad at checking, because the approach I listed is well known to not work
    - Yoric 2y ago
      Got it, now it makes sense :) I may accidentally have been inspired from your message when I wrote the following piece, yesterday: https://yoric.github.io/post/formal-ai/ https://yoric.github.io/post/formal-ai/
throwup238 2y ago
> I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias. As opposed to what, yet another beginner React app? That’s what everyone seems to be testing with but none of the projects I’ve seen are reflective of a production codebase that’s years old and has been touched by a dozen developers. Throw it at a complicated non-frontend mixed language repo like cxx-qt [1] or something, preferably where the training data doesn’t include the latest API. [1] https://github.com/KDAB/cxx-qt https://github.com/KDAB/cxx-qt
- lukan 2y ago
  "preferably where the training data doesn’t include the latest API" That is the reason LLM's in their current shape are pretty useless to me for most tasks. They happily mix different versions of popular frameworks, so I have to do so much manual work to fix it, I rather do all by myself then. Pure (common) math problems, or other domains where the tech did not change so much, like bash scripts or regex are where I can use them. But my actual code? Not really. The LLM would need to be trained only on the API version I use and that is not a thing yet, as far as I am aware.
  - owenpalmer 2y ago
    Have you tried RAG on the docs?
    - lukan 2y ago
      Not yet, I can imagine it can improve things, but was too sceptical that it is worth the effort. Have you had good succes with it?
JTyQZSnP3cQGa8B 2y ago
[flagged]
- tyrust 2y ago
  They're asking about viewing existing PRs, not creating new ones. No need to be so hostile.
fuzzythinker 2y ago
https://www.kaggle.com/competitions/konwinski-prize https://www.kaggle.com/competitions/konwinski-prize
rHAdG12327 2y ago
Given how eager Microsoft is to steal other people's code, perhaps the leaked Windows source code would be an option. Or perhaps Microsoft will let you train on their internal issue tracker.

deleted 2y ago

[deleted]

jarebear6expepj 2y ago

“…should it be done by a human?” Sigh.

namanyayg 2y ago
I'll do it personally in the beginning but was thinking about doing it on scale
- deleted 2y ago
  [deleted]
- JTyQZSnP3cQGa8B 2y ago
  > doing it on scale Like cloud-scale, no-code scale, or NoSQL scale? You are confused, which shows that, maybe, you should not be using such tools with the experience that you don't have.
  - zBard 2y ago
    ‘Like cloud-scale, no-code scale or NoSQL scale’. That is the dumbest statement I have heard this week. You should perhaps refrain from commenting, at-least until you gain the modicum of intelligence that you currently don’t have.

smusamashah 2y ago

Very first thing you can tell us (or try if you haven't) is that if you re-prompt, does it give the same answer? Second can you get it to generate (consistently and repeatedly) the text that gp pointed out? Don't need to switch to a different repo for quick test, just make it reproable on your current repo.

namanyayg 2y ago
Not the exact text, but still decent quality. I'll play around with temperature and prompts a bit.

QuadmasterXLII 2y ago

If you could get an LLM to check, you could just spam solutions with any assortment of models and then use your checker to pick the best.

Yoric 2y ago
Except LLMs are usually really bad at checking. Perhaps if they're some day augmented by formal methods, that might change.
- QuadmasterXLII 2y ago
  Sorry, I was unclear- my statement was supposed to be proof that LLMs are bad at checking, because the approach I listed is well known to not work
  - Yoric 2y ago
    Got it, now it makes sense :) I may accidentally have been inspired from your message when I wrote the following piece, yesterday: https://yoric.github.io/post/formal-ai/ https://yoric.github.io/post/formal-ai/

throwup238 2y ago

> I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias. As opposed to what, yet another beginner React app? That’s what everyone seems to be testing with but none of the projects I’ve seen are reflective of a production codebase that’s years old and has been touched by a dozen developers. Throw it at a complicated non-frontend mixed language repo like cxx-qt [1] or something, preferably where the training data doesn’t include the latest API. [1] https://github.com/KDAB/cxx-qt https://github.com/KDAB/cxx-qt

lukan 2y ago
"preferably where the training data doesn’t include the latest API" That is the reason LLM's in their current shape are pretty useless to me for most tasks. They happily mix different versions of popular frameworks, so I have to do so much manual work to fix it, I rather do all by myself then. Pure (common) math problems, or other domains where the tech did not change so much, like bash scripts or regex are where I can use them. But my actual code? Not really. The LLM would need to be trained only on the API version I use and that is not a thing yet, as far as I am aware.
- owenpalmer 2y ago
  Have you tried RAG on the docs?
  - lukan 2y ago
    Not yet, I can imagine it can improve things, but was too sceptical that it is worth the effort. Have you had good succes with it?

JTyQZSnP3cQGa8B 2y ago

[flagged]

tyrust 2y ago
They're asking about viewing existing PRs, not creating new ones. No need to be so hostile.

fuzzythinker 2y ago

https://www.kaggle.com/competitions/konwinski-prize https://www.kaggle.com/competitions/konwinski-prize

rHAdG12327 2y ago

Given how eager Microsoft is to steal other people's code, perhaps the leaked Windows source code would be an option. Or perhaps Microsoft will let you train on their internal issue tracker.