4 ms·

I've been trying out various models for summarization and I have to say, on first glance, this davinci-003 doesn't impress compared to davinci-002 or facebook/b

I've been trying out various models for summarization and I have to say, on first glance, this davinci-003 doesn't impress compared to davinci-002 or facebook/bart-large-cnn. If you throw in "summarize the following text:" and copy in the contents of https://en.wikipedia.org/wiki/Hacker_News https://en.wikipedia.org/wiki/Hacker_News you get the following (temperature 0.0 max tokens 100)[0]: davinci-003 Hacker News is a social news website focusing on computer science and entrepreneurship, run by Y Combinator. It was created by Paul Graham in 2007 and is moderated by Daniel Gackle. It is intended to recreate a community similar to the early days of Reddit, but with a karma system that requires users to accumulate 501 points before they can downvote content. It also has automated flame and spam detectors and active human moderation, as well as a "voting rings" detector. It davinci-002 Hacker News is a social news website focusing on computer science and entrepreneurship. It is run by the investment fund and startup incubator Y Combinator. The site was created by Paul Graham in February 2007. Initially called Startup News or occasionally News.YC., it became known by its current name on August 14, 2007. facebook/bart-large-cnn Hacker News is a social news website focusing on computer science and entrepreneurship. It is run by the investment fund and startup incubator Y Combinator. In general, content that can be submitted is defined as anything that gratifies one's intellectual curiosity. The site has a proactive attitude in moderating content, including automated flame and spam detectors and active human moderation. Curious if anyone knows any good alternatives or tricks to improve the summaries? For now I'm using the bart one as it has the advantage of being downloadable from huggingface so you can run it yourself [1]. [0]: https://beta.openai.com/playground?model=text-davinci-003 https://beta.openai.com/playground?model=text-davinci-003 [1]: https://huggingface.co/facebook/bart-large-cnn https://huggingface.co/facebook/bart-large-cnn

obert 4y ago
In general it's better asking questions after the content, and T=0.7 is way too high. Here's HN homepage summarized with T=0 and TopP=1, asking "Summarize the content above.": DV2> The content above is a list of the top 30 stories on Hacker News as of 2 hours ago. Each story includes the number of points it has received, the number of comments, and a brief description. DV3> The above content is a list of 30 posts from Hacker News, featuring topics such as new GPT-3 models, US Reps urging the DOJ to investigate anticompetitive conduct, intentionally making close friends, enabling the future of GitHub's REST API, curation and decentralization, making close friends, learning C, FFmpeg, open-source tabletop board game simulator, hiring engineers to unify ML, illegal hashes, designing Windows 95's user interface, a three-sided coin, high performance data integration framework in Go, Mauna Loa eruption, AirDrop being limited to 10 minutes, 26 lines of code that changed the world, SwiftUI being slow, AVX 512 being the future, a directory of Mac menu bar apps, the ATtiny 2-Series, a Fijian island being strangled by vines, hospice becoming a for-profit hustle, protecting from what one wants, a psychoactive substance, AI WordPress code snippet generator, open source and saying "no", a factory that only builds white Toyota Land Cruisers, widget.json and widget construction set, and lab-grown timber/wood.
- robbomacrae 4y ago
  I followed your advice and tried with Temperature 0.0 and rather than paste more walls of text I edited what I got. Sorry if that's bad form? I also upped the max tokens to 100 to better compare with the fb can model. Interestingly whilst the davinci 003 model changed output and was still subpar IMHO, the davinci 002 didn't change at all. I wonder if its cached internally.
  - Oranguru 4y ago
    If I am not mistaken, the temperature parameter controls the amount of randomness in the output. A temperature of 0 will always produce the same output. It is not caching as far as I know.
    - obert 4y ago
      yes, T=0 means no randomness, and given sufficient tokens the output should always be the same (in such case whether they cache is a matter of hit ratio, I'd guess)
PoignardAzur 4y ago
> and is moderated by Daniel Gackle Wait, so that means dang's username means "Dan G"? I always though it meant the polite expletive, as in "gosh dang it!". Mind. Blown.

obert 4y ago

In general it's better asking questions after the content, and T=0.7 is way too high. Here's HN homepage summarized with T=0 and TopP=1, asking "Summarize the content above.": DV2> The content above is a list of the top 30 stories on Hacker News as of 2 hours ago. Each story includes the number of points it has received, the number of comments, and a brief description. DV3> The above content is a list of 30 posts from Hacker News, featuring topics such as new GPT-3 models, US Reps urging the DOJ to investigate anticompetitive conduct, intentionally making close friends, enabling the future of GitHub's REST API, curation and decentralization, making close friends, learning C, FFmpeg, open-source tabletop board game simulator, hiring engineers to unify ML, illegal hashes, designing Windows 95's user interface, a three-sided coin, high performance data integration framework in Go, Mauna Loa eruption, AirDrop being limited to 10 minutes, 26 lines of code that changed the world, SwiftUI being slow, AVX 512 being the future, a directory of Mac menu bar apps, the ATtiny 2-Series, a Fijian island being strangled by vines, hospice becoming a for-profit hustle, protecting from what one wants, a psychoactive substance, AI WordPress code snippet generator, open source and saying "no", a factory that only builds white Toyota Land Cruisers, widget.json and widget construction set, and lab-grown timber/wood.

robbomacrae 4y ago
I followed your advice and tried with Temperature 0.0 and rather than paste more walls of text I edited what I got. Sorry if that's bad form? I also upped the max tokens to 100 to better compare with the fb can model. Interestingly whilst the davinci 003 model changed output and was still subpar IMHO, the davinci 002 didn't change at all. I wonder if its cached internally.
- Oranguru 4y ago
  If I am not mistaken, the temperature parameter controls the amount of randomness in the output. A temperature of 0 will always produce the same output. It is not caching as far as I know.
  - obert 4y ago
    yes, T=0 means no randomness, and given sufficient tokens the output should always be the same (in such case whether they cache is a matter of hit ratio, I'd guess)

PoignardAzur 4y ago

> and is moderated by Daniel Gackle Wait, so that means dang's username means "Dan G"? I always though it meant the polite expletive, as in "gosh dang it!". Mind. Blown.