4 ms·
Downside of ChatGPT is that it makes stuff up, of course, even when you ask it for URLs. A few days ago I was going to send a patch to Alpine Linux, and I reme
by Arnavion 2y ago
Downside of ChatGPT is that it makes stuff up, of course, even when you ask it for URLs.
A few days ago I was going to send a patch to Alpine Linux, and I remembered they had a document for how they want commit messages to be authored but couldn't remember the URL.
Google search results for "alpine linux commit message style doc" only returned garbage in the first ~50 results - pages from Alpine's wiki unrelated to commit message style, pages from other projects like the Linux kernel about their commit style, and batshit results like https://docs.ansible.com/ansible/latest/collections/community/general/lbu_module.html https://docs.ansible.com/ansible/latest/collections/communit... (happens to contain the word "commit").
So I asked ChatGPT "What is the URL of Alpine Linux's style guide for commit messages?" and it confidently told me it's https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/CONTRIBUTING.md https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/C... , which does not exist. I replied "That is a 404" and it responded "I apologize for the inconvenience. Alpine Linux's style guide for commit messages may have been moved or updated since my last training data. To find the most current style guide for Alpine Linux's commit messages, you can visit their repository directly and look for the CONTRIBUTING.md file or [go read their documentation and figure it out yourself, paraphrased]". The first one is a lie - a file named CONTRIBUTING.md has never existed in the master branch of that repo, as git easily reveals. The second one is schizophrenic - it's telling me to read the same file I just told it doesn't exist. The third one is unhelpful.
I had to find it myself. (It's https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/COMMITSTYLE.md https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/C... )
- Yawrehto 2y agoAnd it doesn't cite sources, and when it does it makes stuff up. It's even worse than Google; at least with Google I know where the information is coming from.
- skeledrew 2y agoI only use ChatGPT for things which don't really need a source, such a code snippets. Either a snippet works or it doesn't. For anything where I think an LLM might mess up without me realizing, I go for Perplexity, so I can then check the sources to confirm.
- viraptor 2y agoYou can get the sweet spot with https://perplexity.ai/ https://perplexity.ai/ for many cases. It does the searches, aggregated answer, and the actual supporting links. It got back with "The URL for Alpine Linux's style guide for commit messages can be found in the README.md file of the aports repository on GitLab. The specific URL is: https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/README.md https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/R..." (+ extra links that include the one from the end of this document, which is what you're after) It could do better by returning the actual link rather than a page where you can find it, but it's still better then Google. (Kagi pointed at the same readme at least)
- remram 2y agoSo it made up BOTH those answers. There was never a CONTRIBUTING.md, and COMMITSTYLE.md was added in 2020. https://gitlab.alpinelinux.org/alpine/aports/-/commit/0b70974e074fa6d961ceded2428b99ddc2cc0ef4 https://gitlab.alpinelinux.org/alpine/aports/-/commit/0b7097...
- karencarits 2y agoThanks for providing an example of the search query you had problems with. Interestingly, I haven't noticed the decay in Google search quality that many are complaining about. The first hit when I use your Google query is https://github.com/alpinelinux/aports/blob/master/COMMITSTYLE.md https://github.com/alpinelinux/aports/blob/master/COMMITSTYL... which seems correct?
- Arnavion 2y agoNot for me it isn't. In fact now the first result for me is my very comment above, but the actual COMMITSTYLE.md is still not in the first 50 results. The only result from even the right domain in the first 50 results is https://gitlab.alpinelinux.org/alpine/docs/user-handbook https://gitlab.alpinelinux.org/alpine/docs/user-handbook (at #12, and of course irrelevant).