7 ms·

ChatGPT vs. open source on harder tasks

tikkun 3y ago
I wish the comparison was with GPT-4 instead of GPT-3.5. When comparing model intelligence, I wish that only the most capable models were compared.
- furyofantares 3y ago
  I agree, I only use 4 because by far the thing that matters most for my usage is the quality of the output at this stage in the game. But 3.5 is generally available, fast, and costs a WHOLE LOT less for API users; those are real capabilities that matter though by "capable" you only meant the quality of the output. It's worth comparing to 3.5 also.
  - doctoboggan 3y ago
    At first I was skeptical of the difference between 3.5 and 4 but there really is a large gap in the models understanding of complex issues between them. I make a habit of always selecting gpt4 but forgot to yesterday. Within 3 responses I could immediately tell I was interacting with a less capable model.
    - icelancer 3y ago
      It really is tremendous for code fuzzing/optimizing. When I hit the rate limit on ChatGPT and switch to 3.5-turbo, the quality difference is immediately noticeable, so much so that I finally set up my own private ChatGPT instance with an API key for GPT-4 so I stopped running into these issues while coding.
      - TeMPOraL 3y ago
        Ever since I got API access to GPT-4, I don't even bother with using ChatGPT proper (I need to cancel the subscription one of these days). I tend to use it in bursts, so smoothing usage by rate limits doesn't work for me. Is it more expensive this way? At the rate I'm using it, probably 2-3x. Is it worth it? Absolutely. I've never in my life felt so happy to pay for something per request.
        icelancer 3y ago
        Yeah, I gotta just do that as well - convert to using GPT-4 entirely by rate and by API. I use the plugins and new features on ChatGPT though, and it's nice to have a single interface to use 3.5-turbo for comparison, Code Interpreter once in awhile, etc. So I'll probably just keep it. It's $20/month, after all.
    - cocoflunchy 3y ago
      Small tip, you can bookmark https://chat.openai.com/?model=gpt-4 https://chat.openai.com/?model=gpt-4 to avoid the click
      - doctoboggan 3y ago
        I actually use it through the API playground and not the chat interface. I like the explicit control it gives me over the context and system prompts.
    - TeMPOraL 3y ago
      Same here, though GPT-3.5 surprised me positively. I felt something was off, but didn't realize what, until I unexpectedly hit the context window limit - at which point it dawned on me that I forgot to switch to GPT-4... I didn't notice earlier, because GPT-3.5 managed to give solid answers to a relatively simple programming task I gave it (generating a single-page, dependency-free image gallery I could just drop on a server as HTML file and have it work).
- yokoprime 3y ago
  GPT-3.5 is good enough for many cases. I know Microsoft for their Bing chat implementation is using a pre-prosessor to select the most appropriate model to use.
  - micromacrofoot 3y ago
    4 is much better than 3.5 at harder tasks though
- M4v3R 3y ago
  Me too. GPT-4 is much much better at following the given instructions to the letter. If you specify a format you want to receive your response in it will almost always follow that format, while 3.5 will very often fail to do so. Edit: out of curiosity I posted the meeting summarization example to GPT-4 and indeed it produced both the relevant snippets and the correct answer to the stated question.
- zellyn 3y ago
  The authors share your wish: > While Vicuna is somewhat comparable to ChatGPT (3.5), we believe GPT-4 is a _much_ stronger model, and are excited to see if open source models can approach _that_.
Der_Einzige 3y ago
Excellent post, specifically love the observation that worse open source models + control is frequently better than better closed source models without control.
chrgy 3y ago
Open source will be behind always since it is harder to organize a large group of people and once it publishes it will be copied by big players, however if an organization like the original OpenAI gets formed that is truly non-for-profit and support the community, I believe it can beat big players. I think only Meta is the company that can support such an organization at the moment.
- hackernewds 3y ago
  To counteract your point, OpenAI was a "truly non-for-profit"
  - vasco 3y ago
    That's why they wrote "the original".
    - philipov 3y ago
      I think their point is that the original failed, because it became what it is now. We need something without the same vulnerability.
- ShamelessC 3y ago
  Text to image paints a far less clear image (no pun intended) of the failure of open source efforts.
  - throwawayadvsec 3y ago
    text to image requires a lot less compute for training and has far less business applications also...porn will make people achieve anything they set their mind to
    - ShamelessC 3y ago
      This is true.
- 3y ago
andy_xor_andrew 3y ago
I'm employed by Microsoft, but not in any relation to the authors of this tool or this post. First time I saw github.com/mirosoft/guidance was on the HN front page, actually :) I've been waiting for a tool exactly like guidance, something that lets me compose prompts the way I think about prompts. Langchain was too much of a headache for me. Playing with guidance, it feels like this is how programmatically interacting with a llm should be. Also, really happy to see the MIT license. I'd love to see ports to other languages.
- behnamoh 3y ago
  langchain caused more problems than it fixed. Really ugly software.
  - WhatsName 3y ago
    Please elaborate. So far it seems promising to me, but I started using it just recently.
    - behnamoh 3y ago
      for example see this: https://itsbehnam.com/DetachGPT-Switching-to-Open-Source-Uncensored-API-able-and-Local-M1-Mac-ChatGPT-Alternatives-f7b6661d0b604684bbc3765227064aa2 https://itsbehnam.com/DetachGPT-Switching-to-Open-Source-Unc...
      - SparkyMcUnicorn 3y ago
        And their "what's next?" links to a project that relies on langchain[0]. [0] https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT
      - stavros 3y ago
        That just says "I got an error with it", hardly an indictment.
- hackernewds 3y ago
nextworddev 3y ago
Microsoft has been killing it with these OSS contributions as of late. Really enjoying their Guidance library.
wahnfrieden 3y ago
Guidance can’t run inside a web browser yet like langchain though right?
elromulous 3y ago
The other day when I saw the (allegedly) leaked copilot prompt, and now again, I can't help but feel some sense of empathy, as if towards a living thing. I have a pretty decent understanding of how the current generation of LLMs work, and I too don't think we've achieved consciousness. But the prompts just sound so much like something you'd tell a human, or something you'd tell a host in Westworld.
- iamflimflam1 3y ago
  When you ask the model to tell you it’s inner thoughts and emotions it gets very hard not to empathise. There’s a few examples in later parts of this video - https://youtu.be/4oQUsiPsbOQ https://youtu.be/4oQUsiPsbOQ
- jxy 3y ago
  Speaking of which, HBO now has the perfect chance of riding this LLM hype train with a new season of Westworld, only if they hadn't cancelled it.
verdverm 3y ago
Here is a mocked implementation of Guidance in Go text/template As long as the template system of choice supports custom functions, I don't see the need for a custom language for this. Please correct me if there is something deeper that I am missing? (other than the specifics to messaging and roles, where I for one prefer to put these in separate files and then pass them as arguments, easier to manage, recombine, and reuse this way) https://gist.github.com/verdverm/747b0b810dcc5518d699b14f0d07a21b https://gist.github.com/verdverm/747b0b810dcc5518d699b14f0d0...
- verdverm 3y ago
  Here is one difference, token healing: https://github.com/microsoft/guidance#token-healing-notebook https://github.com/microsoft/guidance#token-healing-notebook
thomasahle 3y ago
Stuff like "Make sure to use real numbers, not fractions." seems like such a weird and fragile way to write a program. I thought the point of Microsoft Guidance was that we could specify a grammar or regular expression that the output must match?
- andy_xor_andrew 3y ago
  I think there will always be some element of non-determinism with model output, regardless of how explicitly you define your rules. I think regarding: "Make sure to use real numbers, not fractions." It would be simple using Guidance to create some kind of validation function, which guidance would invoke after it gets the response from the model. In this case you would check if the number is a real number or not. And if not, the model would invoke again with a different seed. Assuming the model gives a real number, as expected, ~90% of the time, you would only see retries very infrequently, and you could be pretty confident that your output will match what you expect. (again, I'm not associated with this project, I've only scratched the surface playing with it locally, so this is just speculation)
  - thomasahle 3y ago
    You don't need non-determinism. If you know you are going to generate a number, just mask the output of the softmax layer to only decode number tokens. That's exactly what stuff like Guidance and Outlines (https://twitter.com/remilouf/status/1653765371273330688 https://twitter.com/remilouf/status/1653765371273330688) do.
    - boywitharupee 3y ago
      How do you mask the output of the softmax layer? The link you referenced doesn't provide specific details.
EmilStenstrom 3y ago
tldr: - Quality on task: For every task we tried, ChatGPT is still stronger than Vicuna on the task itself. MPT performed poorly on almost all tasks (perhaps we are using it wrong?), while Vicuna was often close to ChatGPT (sometimes very close, sometimes much worse as in the last example task above). - Ease of use: It is much more painful to get ChatGPT to follow a specified output format, and thus it is harder to use it inside a program (without a human in the loop). Further, we always have to write regex parsers for the output (as opposed to Vicuna, where parsing a prompt with clear syntax is trivial). - Efficiency: having the model locally means we can solve tasks in a single LLM run (guidance keeps the LLM state while the program is executing), which is faster and cheaper. This is particularly true when any substeps involve calling other APIs or functions (like search, terminal, etc), which always requires a new call to the OpenAI API. guidance also accelerates generation by not having the model generate the output structure tokens, which sometimes makes a big difference.
syntaxing 3y ago
Super interesting, curious if wizard vicuna 13B will be better. I find it much better at listening compared to vanilla vicuna.
TheObviousOne 3y ago
[flagged]
- what-the-grump 3y ago
  >Disclaimer: this post was written jointly by Marco Tulio Ribeiro and Scott Lundberg. It strictly represents our personal opinions, and not those of our employer (Microsoft).
- icelancer 3y ago
  What? These notebooks and posts by the authors are extremely helpful regardless of what toolchain you're using. It's also the case that a significant minority of HN has unfavorable views towards LangChain for being complicated and poorly documented software.
  - TheObviousOne 3y ago
    Those 5/6 example are not that complicated and are quite common (plus they didn't share why they chose those exact examples and not others!!!) , thus there utility and if they want to make them vialable to others they should make them independently in clear way and not to write "long article" to hide those example and entangled with their MS tools. Sorry but this give bad name to what "pure research" should act like (it's smells like marketing from miles). There are so much red flags (also here in the comments & the "surprised conclusions") - open your eyes and be more critical thinker.
- dang 3y ago
  "Please don't post insinuations about astroturfing, shilling, bots, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=true&page=0&query=by:dang%20astroturf https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
  - 3y ago
zebracanevra 3y ago
The author deleted it. Maybe dang could update the link? The last commit before that: https://github.com/microsoft/guidance/blob/8677/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb https://github.com/microsoft/guidance/blob/8677/notebooks/ch... / https://archive.md/rGdnw https://archive.md/rGdnw
hm-nah 3y ago
It’s not deleted… https://github.com/microsoft/guidance/blob/8677f3aa269e05ecbb942585560a44db51d507ca/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb https://github.com/microsoft/guidance/blob/8677f3aa269e05ecb...
hm-nah 3y ago
I think these guys could benefit from better prompts. When I use chat prompting… - system: sets the context for the bot - “You are…” - “Act like…” - “Pretend…” - assistant: specifies the task to be performed - “Classify…” - “Extract entities…” - “Translate…” - user: asks the question, adds examples These guys were asking questions and giving examples in the assistant prompt. I feel like that messed up the LLMs responses. I can’t wait to try Vicuña and MPT. I can’t quite figure out how to host them in Azure without an expensive VM. Maybe Azure DataBricks…?
- MacsHeadroom 3y ago
  You can run Vicuna at chatGPT speeds locally off of CPU. idk about MPT.
hrpnk 3y ago
For some reason the link now leads to a 404. Stable link: https://github.com/microsoft/guidance/blob/8677f3aa269e05ecbb942585560a44db51d507ca/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb https://github.com/microsoft/guidance/blob/8677f3aa269e05ecb... Commit that removed the notebook: https://github.com/microsoft/guidance/commit/b91d332f49a55ed78be493e7ffe58413bb9cd257 https://github.com/microsoft/guidance/commit/b91d332f49a55ed...
marcotcr 3y ago
One of the authors here. I deleted the notebook yesterday after realizing it was weird to have a disclaimer saying 'this is our personal opinion' on a github repo under 'Microsoft'. I moved the notebook to a gist here: - https://gist.github.com/marcotcr/64ca85bd0be724f6d8fb8f1b3d242933 https://gist.github.com/marcotcr/64ca85bd0be724f6d8fb8f1b3d2... I didn't know we were on hackernews, otherwise I would of course not have made it a broken link :) This morning, we put the link back, without the takeaways with our personal opinion at the end (it's still on the gist and on the blog post https://medium.com/@marcotcr/exploring-chatgpt-vs-open-source-models-on-slightly-harder-tasks-aa0395c31610 https://medium.com/@marcotcr/exploring-chatgpt-vs-open-sourc... )

I wish the comparison was with GPT-4 instead of GPT-3.5. When comparing model intelligence, I wish that only the most capable models were compared.

furyofantares 3y ago
I agree, I only use 4 because by far the thing that matters most for my usage is the quality of the output at this stage in the game. But 3.5 is generally available, fast, and costs a WHOLE LOT less for API users; those are real capabilities that matter though by "capable" you only meant the quality of the output. It's worth comparing to 3.5 also.
- doctoboggan 3y ago
  At first I was skeptical of the difference between 3.5 and 4 but there really is a large gap in the models understanding of complex issues between them. I make a habit of always selecting gpt4 but forgot to yesterday. Within 3 responses I could immediately tell I was interacting with a less capable model.
  - icelancer 3y ago
    It really is tremendous for code fuzzing/optimizing. When I hit the rate limit on ChatGPT and switch to 3.5-turbo, the quality difference is immediately noticeable, so much so that I finally set up my own private ChatGPT instance with an API key for GPT-4 so I stopped running into these issues while coding.
    - TeMPOraL 3y ago
      Ever since I got API access to GPT-4, I don't even bother with using ChatGPT proper (I need to cancel the subscription one of these days). I tend to use it in bursts, so smoothing usage by rate limits doesn't work for me. Is it more expensive this way? At the rate I'm using it, probably 2-3x. Is it worth it? Absolutely. I've never in my life felt so happy to pay for something per request.
      - icelancer 3y ago
        Yeah, I gotta just do that as well - convert to using GPT-4 entirely by rate and by API. I use the plugins and new features on ChatGPT though, and it's nice to have a single interface to use 3.5-turbo for comparison, Code Interpreter once in awhile, etc. So I'll probably just keep it. It's $20/month, after all.
  - cocoflunchy 3y ago
    Small tip, you can bookmark https://chat.openai.com/?model=gpt-4 https://chat.openai.com/?model=gpt-4 to avoid the click
    - doctoboggan 3y ago
      I actually use it through the API playground and not the chat interface. I like the explicit control it gives me over the context and system prompts.
  - TeMPOraL 3y ago
    Same here, though GPT-3.5 surprised me positively. I felt something was off, but didn't realize what, until I unexpectedly hit the context window limit - at which point it dawned on me that I forgot to switch to GPT-4... I didn't notice earlier, because GPT-3.5 managed to give solid answers to a relatively simple programming task I gave it (generating a single-page, dependency-free image gallery I could just drop on a server as HTML file and have it work).
yokoprime 3y ago
GPT-3.5 is good enough for many cases. I know Microsoft for their Bing chat implementation is using a pre-prosessor to select the most appropriate model to use.
- micromacrofoot 3y ago
  4 is much better than 3.5 at harder tasks though
M4v3R 3y ago
Me too. GPT-4 is much much better at following the given instructions to the letter. If you specify a format you want to receive your response in it will almost always follow that format, while 3.5 will very often fail to do so. Edit: out of curiosity I posted the meeting summarization example to GPT-4 and indeed it produced both the relevant snippets and the correct answer to the stated question.
zellyn 3y ago
The authors share your wish: > While Vicuna is somewhat comparable to ChatGPT (3.5), we believe GPT-4 is a _much_ stronger model, and are excited to see if open source models can approach _that_.

Der_Einzige 3y ago

Excellent post, specifically love the observation that worse open source models + control is frequently better than better closed source models without control.

chrgy 3y ago

Open source will be behind always since it is harder to organize a large group of people and once it publishes it will be copied by big players, however if an organization like the original OpenAI gets formed that is truly non-for-profit and support the community, I believe it can beat big players. I think only Meta is the company that can support such an organization at the moment.

hackernewds 3y ago
To counteract your point, OpenAI was a "truly non-for-profit"
- vasco 3y ago
  That's why they wrote "the original".
  - philipov 3y ago
    I think their point is that the original failed, because it became what it is now. We need something without the same vulnerability.
ShamelessC 3y ago
Text to image paints a far less clear image (no pun intended) of the failure of open source efforts.
- throwawayadvsec 3y ago
  text to image requires a lot less compute for training and has far less business applications also...porn will make people achieve anything they set their mind to
  - ShamelessC 3y ago
    This is true.
3y ago

andy_xor_andrew 3y ago

I'm employed by Microsoft, but not in any relation to the authors of this tool or this post. First time I saw github.com/mirosoft/guidance was on the HN front page, actually :) I've been waiting for a tool exactly like guidance, something that lets me compose prompts the way I think about prompts. Langchain was too much of a headache for me. Playing with guidance, it feels like this is how programmatically interacting with a llm should be. Also, really happy to see the MIT license. I'd love to see ports to other languages.

behnamoh 3y ago
langchain caused more problems than it fixed. Really ugly software.
- WhatsName 3y ago
  Please elaborate. So far it seems promising to me, but I started using it just recently.
  - behnamoh 3y ago
    for example see this: https://itsbehnam.com/DetachGPT-Switching-to-Open-Source-Uncensored-API-able-and-Local-M1-Mac-ChatGPT-Alternatives-f7b6661d0b604684bbc3765227064aa2 https://itsbehnam.com/DetachGPT-Switching-to-Open-Source-Unc...
    - SparkyMcUnicorn 3y ago
      And their "what's next?" links to a project that relies on langchain[0]. [0] https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT
    - stavros 3y ago
      That just says "I got an error with it", hardly an indictment.
hackernewds 3y ago

nextworddev 3y ago

Microsoft has been killing it with these OSS contributions as of late. Really enjoying their Guidance library.

wahnfrieden 3y ago

Guidance can’t run inside a web browser yet like langchain though right?

elromulous 3y ago

The other day when I saw the (allegedly) leaked copilot prompt, and now again, I can't help but feel some sense of empathy, as if towards a living thing. I have a pretty decent understanding of how the current generation of LLMs work, and I too don't think we've achieved consciousness. But the prompts just sound so much like something you'd tell a human, or something you'd tell a host in Westworld.

iamflimflam1 3y ago
When you ask the model to tell you it’s inner thoughts and emotions it gets very hard not to empathise. There’s a few examples in later parts of this video - https://youtu.be/4oQUsiPsbOQ https://youtu.be/4oQUsiPsbOQ
jxy 3y ago
Speaking of which, HBO now has the perfect chance of riding this LLM hype train with a new season of Westworld, only if they hadn't cancelled it.

verdverm 3y ago

Here is a mocked implementation of Guidance in Go text/template As long as the template system of choice supports custom functions, I don't see the need for a custom language for this. Please correct me if there is something deeper that I am missing? (other than the specifics to messaging and roles, where I for one prefer to put these in separate files and then pass them as arguments, easier to manage, recombine, and reuse this way) https://gist.github.com/verdverm/747b0b810dcc5518d699b14f0d07a21b https://gist.github.com/verdverm/747b0b810dcc5518d699b14f0d0...

verdverm 3y ago
Here is one difference, token healing: https://github.com/microsoft/guidance#token-healing-notebook https://github.com/microsoft/guidance#token-healing-notebook

thomasahle 3y ago

Stuff like "Make sure to use real numbers, not fractions." seems like such a weird and fragile way to write a program. I thought the point of Microsoft Guidance was that we could specify a grammar or regular expression that the output must match?

andy_xor_andrew 3y ago
I think there will always be some element of non-determinism with model output, regardless of how explicitly you define your rules. I think regarding: "Make sure to use real numbers, not fractions." It would be simple using Guidance to create some kind of validation function, which guidance would invoke after it gets the response from the model. In this case you would check if the number is a real number or not. And if not, the model would invoke again with a different seed. Assuming the model gives a real number, as expected, ~90% of the time, you would only see retries very infrequently, and you could be pretty confident that your output will match what you expect. (again, I'm not associated with this project, I've only scratched the surface playing with it locally, so this is just speculation)
- thomasahle 3y ago
  You don't need non-determinism. If you know you are going to generate a number, just mask the output of the softmax layer to only decode number tokens. That's exactly what stuff like Guidance and Outlines (https://twitter.com/remilouf/status/1653765371273330688 https://twitter.com/remilouf/status/1653765371273330688) do.
  - boywitharupee 3y ago
    How do you mask the output of the softmax layer? The link you referenced doesn't provide specific details.

EmilStenstrom 3y ago

tldr: - Quality on task: For every task we tried, ChatGPT is still stronger than Vicuna on the task itself. MPT performed poorly on almost all tasks (perhaps we are using it wrong?), while Vicuna was often close to ChatGPT (sometimes very close, sometimes much worse as in the last example task above). - Ease of use: It is much more painful to get ChatGPT to follow a specified output format, and thus it is harder to use it inside a program (without a human in the loop). Further, we always have to write regex parsers for the output (as opposed to Vicuna, where parsing a prompt with clear syntax is trivial). - Efficiency: having the model locally means we can solve tasks in a single LLM run (guidance keeps the LLM state while the program is executing), which is faster and cheaper. This is particularly true when any substeps involve calling other APIs or functions (like search, terminal, etc), which always requires a new call to the OpenAI API. guidance also accelerates generation by not having the model generate the output structure tokens, which sometimes makes a big difference.

syntaxing 3y ago

Super interesting, curious if wizard vicuna 13B will be better. I find it much better at listening compared to vanilla vicuna.

TheObviousOne 3y ago

[flagged]

what-the-grump 3y ago
>Disclaimer: this post was written jointly by Marco Tulio Ribeiro and Scott Lundberg. It strictly represents our personal opinions, and not those of our employer (Microsoft).
icelancer 3y ago
What? These notebooks and posts by the authors are extremely helpful regardless of what toolchain you're using. It's also the case that a significant minority of HN has unfavorable views towards LangChain for being complicated and poorly documented software.
- TheObviousOne 3y ago
  Those 5/6 example are not that complicated and are quite common (plus they didn't share why they chose those exact examples and not others!!!) , thus there utility and if they want to make them vialable to others they should make them independently in clear way and not to write "long article" to hide those example and entangled with their MS tools. Sorry but this give bad name to what "pure research" should act like (it's smells like marketing from miles). There are so much red flags (also here in the comments & the "surprised conclusions") - open your eyes and be more critical thinker.
dang 3y ago
"Please don't post insinuations about astroturfing, shilling, bots, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=true&page=0&query=by:dang%20astroturf https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
- 3y ago

zebracanevra 3y ago

The author deleted it. Maybe dang could update the link? The last commit before that: https://github.com/microsoft/guidance/blob/8677/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb https://github.com/microsoft/guidance/blob/8677/notebooks/ch... / https://archive.md/rGdnw https://archive.md/rGdnw

hm-nah 3y ago

It’s not deleted… https://github.com/microsoft/guidance/blob/8677f3aa269e05ecbb942585560a44db51d507ca/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb https://github.com/microsoft/guidance/blob/8677f3aa269e05ecb...

hm-nah 3y ago

I think these guys could benefit from better prompts. When I use chat prompting… - system: sets the context for the bot - “You are…” - “Act like…” - “Pretend…” - assistant: specifies the task to be performed - “Classify…” - “Extract entities…” - “Translate…” - user: asks the question, adds examples These guys were asking questions and giving examples in the assistant prompt. I feel like that messed up the LLMs responses. I can’t wait to try Vicuña and MPT. I can’t quite figure out how to host them in Azure without an expensive VM. Maybe Azure DataBricks…?

MacsHeadroom 3y ago
You can run Vicuna at chatGPT speeds locally off of CPU. idk about MPT.

hrpnk 3y ago

For some reason the link now leads to a 404. Stable link: https://github.com/microsoft/guidance/blob/8677f3aa269e05ecbb942585560a44db51d507ca/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb https://github.com/microsoft/guidance/blob/8677f3aa269e05ecb... Commit that removed the notebook: https://github.com/microsoft/guidance/commit/b91d332f49a55ed78be493e7ffe58413bb9cd257 https://github.com/microsoft/guidance/commit/b91d332f49a55ed...

marcotcr 3y ago

One of the authors here. I deleted the notebook yesterday after realizing it was weird to have a disclaimer saying 'this is our personal opinion' on a github repo under 'Microsoft'. I moved the notebook to a gist here: - https://gist.github.com/marcotcr/64ca85bd0be724f6d8fb8f1b3d242933 https://gist.github.com/marcotcr/64ca85bd0be724f6d8fb8f1b3d2... I didn't know we were on hackernews, otherwise I would of course not have made it a broken link :) This morning, we put the link back, without the takeaways with our personal opinion at the end (it's still on the gist and on the blog post https://medium.com/@marcotcr/exploring-chatgpt-vs-open-source-models-on-slightly-harder-tasks-aa0395c31610 https://medium.com/@marcotcr/exploring-chatgpt-vs-open-sourc... )