4 ms·

I wish ChatGPT would give you the score(s) of the results it returns, so you could understand how correct it "thinks" the response is. I wonder if those confide

I wish ChatGPT would give you the score(s) of the results it returns, so you could understand how correct it "thinks" the response is. I wonder if those confidently wrong answers are also low scoring.

permo-w 4y ago
you could probably ask it
- dpkirchner 4y ago
  I'd considered that -- in some cases it says it's absolutely confident, like with basic arithmetic, and in others it says: > Without more information, I cannot estimate the accuracy or reliability of these claims. It is important to note that there may be conflicting or inaccurate information available online, and it is always a good idea to verify information from multiple sources before accepting it as fact. which seems pretty reasonable.
- dragonwriter 4y ago
  You can ask it, and it may generate text with a score, but there is no particular reason that the score generated in the text would reflect any internal score it has.
geoffreypoirier 4y ago
I had a long requirements chat last night and did just that: There are many self-hosted solutions that could potentially meet the requirements you have listed for your RSS aggregator and reader project. Here is a list of some popular options, ranked in order of their approximate percentage of compatibility with the features you have specified: RSS Aggregator | Percentage Score | URL | | --- | --- | | 90% | https://tt-rss.org/ https://tt-rss.org/ | | 85% | https://miniflux.app/ https://miniflux.app/ | | 80% | https://newsboat.org/ https://newsboat.org/ | | 75% | https://github.com/lobsters/lobsters https://github.com/lobsters/lobsters | | 70% | https://github.com/noahcoad/feeds https://github.com/noahcoad/feeds | | 65% | https://github.com/ssut/py-googlenews https://github.com/ssut/py-googlenews | | 60% | https://git.gnome.org/browse/feedreader/ https://git.gnome.org/browse/feedreader/ | RSS Reader | Percentage Score | URL | | --- | --- | | 90% | https://tt-rss.org/ https://tt-rss.org/ | | 85% | https://miniflux.app/ https://miniflux.app/ | | 80% | https://newsboat.org/ https://newsboat.org/ | | 75% | https://github.com/lobsters/lobsters https://github.com/lobsters/lobsters | | 70% | https://github.com/noahcoad/feeds https://github.com/noahcoad/feeds | | 65% | https://github.com/ssut/py-googlenews https://github.com/ssut/py-googlenews | | 60% | https://git.gnome.org/browse/feedreader/ https://git.gnome.org/browse/feedreader/ | Please note that these percentages are approximate and are intended only as a rough guide. You may want to consider other factors such as the user interface, performance, security, and integration with other tools when choosing a solution for your project.
noitpmeder 4y ago
These scores would mean nothing! At best it would score "how well does my answer fit to my model of the English language". At no point can the current architecture accurately grade "how correct is this answer in the context of the subject domain".
turmeric_root 4y ago
The GPT-3 playground lets you view the probabilities of generated tokens. That's the closest thing to a confidence score you can get from a transformer model. But like the other person said, this isn't very meaningful.

permo-w 4y ago

you could probably ask it

dpkirchner 4y ago
I'd considered that -- in some cases it says it's absolutely confident, like with basic arithmetic, and in others it says: > Without more information, I cannot estimate the accuracy or reliability of these claims. It is important to note that there may be conflicting or inaccurate information available online, and it is always a good idea to verify information from multiple sources before accepting it as fact. which seems pretty reasonable.
dragonwriter 4y ago
You can ask it, and it may generate text with a score, but there is no particular reason that the score generated in the text would reflect any internal score it has.

geoffreypoirier 4y ago

I had a long requirements chat last night and did just that: There are many self-hosted solutions that could potentially meet the requirements you have listed for your RSS aggregator and reader project. Here is a list of some popular options, ranked in order of their approximate percentage of compatibility with the features you have specified: RSS Aggregator | Percentage Score | URL | | --- | --- | | 90% | https://tt-rss.org/ https://tt-rss.org/ | | 85% | https://miniflux.app/ https://miniflux.app/ | | 80% | https://newsboat.org/ https://newsboat.org/ | | 75% | https://github.com/lobsters/lobsters https://github.com/lobsters/lobsters | | 70% | https://github.com/noahcoad/feeds https://github.com/noahcoad/feeds | | 65% | https://github.com/ssut/py-googlenews https://github.com/ssut/py-googlenews | | 60% | https://git.gnome.org/browse/feedreader/ https://git.gnome.org/browse/feedreader/ | RSS Reader | Percentage Score | URL | | --- | --- | | 90% | https://tt-rss.org/ https://tt-rss.org/ | | 85% | https://miniflux.app/ https://miniflux.app/ | | 80% | https://newsboat.org/ https://newsboat.org/ | | 75% | https://github.com/lobsters/lobsters https://github.com/lobsters/lobsters | | 70% | https://github.com/noahcoad/feeds https://github.com/noahcoad/feeds | | 65% | https://github.com/ssut/py-googlenews https://github.com/ssut/py-googlenews | | 60% | https://git.gnome.org/browse/feedreader/ https://git.gnome.org/browse/feedreader/ | Please note that these percentages are approximate and are intended only as a rough guide. You may want to consider other factors such as the user interface, performance, security, and integration with other tools when choosing a solution for your project.

noitpmeder 4y ago

These scores would mean nothing! At best it would score "how well does my answer fit to my model of the English language". At no point can the current architecture accurately grade "how correct is this answer in the context of the subject domain".

turmeric_root 4y ago

The GPT-3 playground lets you view the probabilities of generated tokens. That's the closest thing to a confidence score you can get from a transformer model. But like the other person said, this isn't very meaningful.