4 ms·

Looking at https://stackoverflow.co/labs/ https://stackoverflow.co/labs/ > The future of collective knowledge sharing. The future is feeding all answers into

by chx 3y ago

Looking at https://stackoverflow.co/labs/ https://stackoverflow.co/labs/ > The future of collective knowledge sharing. The future is feeding all answers into a meat grinder and then forming word like nuggets from the paste and hope we like this? I am sorry but you lost your mind. There is a very fundamental disconnect between LLM hype and reality. Here is my latest experiment, I needed to search for a string in dangling commits and asked Bing: https://i.imgur.com/pQtpkH5.png https://i.imgur.com/pQtpkH5.png You will notice my question was a link because I copied the title from https://stackoverflow.com/q/21903972/308851 https://stackoverflow.com/q/21903972/308851 and apparently that made it into a link (I had the question open because every time I need to do this I just jump to this question, I outsourced storing my knowledge to SO ages ago). Every single answer given by Bing is wrong, of course, because git log -g is not appropriate for this as the answer states. It's right there: "Commits that are (still) referenced by the reflog are considered reachable and not dangling". And here's the best part: the Learn More 1. Stackoverflow.com links to this very question. The language model have seen "git log -g" in the vicinity of the string "Search for string in dangling commits in Git" and thought it's appropriate. The irony.

airza 3y ago
I don't like the profit motives, but it's tough for me to argue that the experience of using stackoverflow is better than just jamming my question into chatgpt.
- chx 3y ago
  Except the information value coming out of a chatgpt answer is zero. You are literally best off if you ignore whatever it says.
  - airza 3y ago
    It's really not though. I ask it questions about typescript all the time: "i want to do X with types, how do i express this in TS syntax?" and it answers correctly. No disagreement that i don't always agree with its subjective judgment, but that is on par with subjective opinions on stack overflow.
  - naasking 3y ago
    You're ignoring all the times the stackoverflow answers also provide zero information, so this is not a balanced assessment. Prompting LLMs correctly to get high quality answers is a skill.
    - chx 3y ago
      Seeing value in LLM answers is apophenia. It's an understandable human condition but still. There can be value in SO answers, there can not be in LLM as it's just automated plagiarism without understanding anything.
      - falcor84 3y ago
        There's immense value in ChatGPT's answers, you just need to accept that it's fallible.
        practice9 3y ago
        I think most of the critique is coming from the people that made their mind up already or used only GPT-3.5 in the past. Then it becomes a self-fulfilling prophecy: a) "ChatGPT is stupid! Also, plagiarism! No value!" b) " I wrote a carelessly bad prompt and it responded with a seemingly bad answer!" c) " This confirms my biases!" I find that when I provide it with enough context and a set of requirements (as I would with a non-AI engineer), it's very good at problem solving.
      - dlkf 3y ago
        How does the issue of plagiarism bear on its utility? That’s like saying it’s impossible to get value out of listening to a torrented mp3. I’m not sure understanding is an inherent issue either. When I Google ”weather lisbon,” Google spits out a number. The app doesn’t ”understand” anything, but it still helps me plan my day.
      - ben_w 3y ago
        > automated plagiarism without understanding anything I'm old enough to remember the memes campaigning to rename "software engineering" to "googling StackOverflow", so plagiarism isn't a point against the value of an LLM, and given the case of an LLM made by StackOverflow using StackOverflow data it's also not a legal issue.
  - dwaltrip 3y ago
    This is so wrong it’s not even funny.
  - talleyrand 3y ago
    ChatGPT recently suggested a fix for a buggy service worker I have and it worked perfectly. I am eternally grateful.
  - pizza 3y ago
    Along those lines, how is that position not randomania? You’re basically saying that the signal to noise ratio is zero so its utility is zero. I guess you must think all of the comments here where someone mentioned how it helped them solve a problem, also have a 0 (or at least dismissible) signal to noise ratio? What signal to noise would be needed for you to say it’s not devoid of any information? What would the minimum accuracy need to be? 80%? 95? 99? 100? The thing is there are many questions where it is a short path to a solution that can then be verified. The most useful way to use it productively is to induce it to give you pointers to things you had no idea existed. Then dereference those pointers via traditional means (ie google, man pages). Be your own null ptr (hallucinations) segfault handler. Critical thinking a hundred percent preserved. It’s a little bemusing how people take issue with next token prediction trained with a log likelihood loss. In particular they think that getting a high accuracy extremely low perplexity LLM is unimpressive because of the low accuracy high perplexity autocorrect on their phone. No- the llm’s emergent generalization capabilities as nll drops really is a capital p Phenomenon. But I feel that I don’t need to try to change anyone’s mind anymore. Everyone had a threshold logp where they’ll change their mind about this, whether it’s attained in the next week or long from now. Just state the nonzero perplexity that you would accept from a person thinking, and wait for the loss curve to meet it.
    - hanselot 3y ago
      Don't bargain on that. For some the threshold is way too high given how far they've sunk. As always, the ones most vocal about it haven't even bothered trying out guanaco 70b 2bit. Probably don't know what ggml, gptq, cformers or any of the others are. They still base their assumptions on the openai gold standard.
jimstr 3y ago
The answer I got with the prompt "You are a git expert. How do I search for a string in dangling commits?" Was: Dangling commits in Git are often referred to as commits that are no longer reachable from any branch, tag, or the HEAD. These commits usually occur when you create a commit and then move the branch pointer away from it without creating a new branch or tag. They can also occur during rebases or other history rewriting operations. Git has a garbage collector (`git gc`) that will periodically clean up these dangling commits, but until that happens, they're still in your repository and can be examined. Unfortunately, Git doesn't provide a direct command to search for a string in dangling commits. However, you can find these commits using the `git fsck` command, which checks the filesystem for errors and dangling commits. Here is a way to do it manually: 1. List all the dangling commits using `git fsck --lost-found`. This will print out all the dangling commits and store them in `.git/lost-found/commit/`. 2. For each dangling commit, you can search for a specific string using `git log -p <commit-id>` or `git show <commit-id>` and then pipe the output to `grep`. Here's an example of what this might look like in a script: mkdir -p .git/lost-found git fsck --lost-found for commit in $(ls .git/lost-found/commit/); do git show $commit | grep -q "your string here" && echo "Found in commit $commit" done This script will print the IDs of any dangling commits that contain the string "your string here". Remember to replace "your string here" with the string you are searching for. *NOTE*: This operation could be time-consuming if you have a lot of dangling commits or a large repository. Also, this approach might not work if the garbage collector has already cleaned up the commits. Make sure to be careful when manipulating Git's internals like this, as it could potentially corrupt your repository if done incorrectly.

by chx 3y ago

airza 3y ago

I don't like the profit motives, but it's tough for me to argue that the experience of using stackoverflow is better than just jamming my question into chatgpt.

chx 3y ago
Except the information value coming out of a chatgpt answer is zero. You are literally best off if you ignore whatever it says.
- airza 3y ago
  It's really not though. I ask it questions about typescript all the time: "i want to do X with types, how do i express this in TS syntax?" and it answers correctly. No disagreement that i don't always agree with its subjective judgment, but that is on par with subjective opinions on stack overflow.
- naasking 3y ago
  You're ignoring all the times the stackoverflow answers also provide zero information, so this is not a balanced assessment. Prompting LLMs correctly to get high quality answers is a skill.
  - chx 3y ago
    Seeing value in LLM answers is apophenia. It's an understandable human condition but still. There can be value in SO answers, there can not be in LLM as it's just automated plagiarism without understanding anything.
    - falcor84 3y ago
      There's immense value in ChatGPT's answers, you just need to accept that it's fallible.
      - practice9 3y ago
        I think most of the critique is coming from the people that made their mind up already or used only GPT-3.5 in the past. Then it becomes a self-fulfilling prophecy: a) "ChatGPT is stupid! Also, plagiarism! No value!" b) " I wrote a carelessly bad prompt and it responded with a seemingly bad answer!" c) " This confirms my biases!" I find that when I provide it with enough context and a set of requirements (as I would with a non-AI engineer), it's very good at problem solving.
    - dlkf 3y ago
      How does the issue of plagiarism bear on its utility? That’s like saying it’s impossible to get value out of listening to a torrented mp3. I’m not sure understanding is an inherent issue either. When I Google ”weather lisbon,” Google spits out a number. The app doesn’t ”understand” anything, but it still helps me plan my day.
    - ben_w 3y ago
      > automated plagiarism without understanding anything I'm old enough to remember the memes campaigning to rename "software engineering" to "googling StackOverflow", so plagiarism isn't a point against the value of an LLM, and given the case of an LLM made by StackOverflow using StackOverflow data it's also not a legal issue.
- dwaltrip 3y ago
  This is so wrong it’s not even funny.
- talleyrand 3y ago
  ChatGPT recently suggested a fix for a buggy service worker I have and it worked perfectly. I am eternally grateful.
- pizza 3y ago
  Along those lines, how is that position not randomania? You’re basically saying that the signal to noise ratio is zero so its utility is zero. I guess you must think all of the comments here where someone mentioned how it helped them solve a problem, also have a 0 (or at least dismissible) signal to noise ratio? What signal to noise would be needed for you to say it’s not devoid of any information? What would the minimum accuracy need to be? 80%? 95? 99? 100? The thing is there are many questions where it is a short path to a solution that can then be verified. The most useful way to use it productively is to induce it to give you pointers to things you had no idea existed. Then dereference those pointers via traditional means (ie google, man pages). Be your own null ptr (hallucinations) segfault handler. Critical thinking a hundred percent preserved. It’s a little bemusing how people take issue with next token prediction trained with a log likelihood loss. In particular they think that getting a high accuracy extremely low perplexity LLM is unimpressive because of the low accuracy high perplexity autocorrect on their phone. No- the llm’s emergent generalization capabilities as nll drops really is a capital p Phenomenon. But I feel that I don’t need to try to change anyone’s mind anymore. Everyone had a threshold logp where they’ll change their mind about this, whether it’s attained in the next week or long from now. Just state the nonzero perplexity that you would accept from a person thinking, and wait for the loss curve to meet it.
  - hanselot 3y ago
    Don't bargain on that. For some the threshold is way too high given how far they've sunk. As always, the ones most vocal about it haven't even bothered trying out guanaco 70b 2bit. Probably don't know what ggml, gptq, cformers or any of the others are. They still base their assumptions on the openai gold standard.

jimstr 3y ago

The answer I got with the prompt "You are a git expert. How do I search for a string in dangling commits?" Was: Dangling commits in Git are often referred to as commits that are no longer reachable from any branch, tag, or the HEAD. These commits usually occur when you create a commit and then move the branch pointer away from it without creating a new branch or tag. They can also occur during rebases or other history rewriting operations. Git has a garbage collector (`git gc`) that will periodically clean up these dangling commits, but until that happens, they're still in your repository and can be examined. Unfortunately, Git doesn't provide a direct command to search for a string in dangling commits. However, you can find these commits using the `git fsck` command, which checks the filesystem for errors and dangling commits. Here is a way to do it manually: 1. List all the dangling commits using `git fsck --lost-found`. This will print out all the dangling commits and store them in `.git/lost-found/commit/`. 2. For each dangling commit, you can search for a specific string using `git log -p <commit-id>` or `git show <commit-id>` and then pipe the output to `grep`. Here's an example of what this might look like in a script: mkdir -p .git/lost-found git fsck --lost-found for commit in $(ls .git/lost-found/commit/); do git show $commit | grep -q "your string here" && echo "Found in commit $commit" done This script will print the IDs of any dangling commits that contain the string "your string here". Remember to replace "your string here" with the string you are searching for. *NOTE*: This operation could be time-consuming if you have a lot of dangling commits or a large repository. Also, this approach might not work if the garbage collector has already cleaned up the commits. Make sure to be careful when manipulating Git's internals like this, as it could potentially corrupt your repository if done incorrectly.