5 ms·

How does this compare to Github Copilot? It's not shown in their comparison

nkozyra 2y ago
Not sure how much current Copilot varies from the original Codex, but another set of benchmarks here: https://paperswithcode.com/sota/code-generation-on-humaneval https://paperswithcode.com/sota/code-generation-on-humaneval
ramon156 2y ago
Knowing the training data GH has I doubt it's comparable, then again I don't have the benchmarks
- ramon156 2y ago
  After typing this I tried the live chat out and it honestly seems a lot more promising than current GH Copilot, very nice!
- ssgodderidge 2y ago
  Are you saying GH has more than Codestral and therefore GH has a better model? Or that Codestral would be better because Codestral is not littered with "bad" code?
  - nkozyra 2y ago
    Bad code is obviously very subjective, but I would wager that GH places a much higher value on feedback mechanisms like stars, issues, PRs, velocity, etc. Their ubiquity likely allows them to automatically cherry-pick less "bad code."
    - nicce 2y ago
      Nothing prevents Mistral do the same if they want to. Issues and and PRs are public information, exposed by APIs, and not that much rate limited.
rohansood15 2y ago
Copilot primarily uses GPT-3.5, which is outclassed by Llama3-70B. And this model claims to be slightly better than Llama3-70B. Edit: For those who don't believe me, https://github.com/microsoft/vscode-copilot-release/issues/664 https://github.com/microsoft/vscode-copilot-release/issues/6.... Gpt-4 for chat, 3.5 for code.
- jasonjmcghee 2y ago
  GitHub Copilot uses GPT-3.5? I was under the impression it was a custom codex model with a surrogate local model as per https://github.blog/2023-02-14-github-copilot-now-has-a-better-ai-model-and-new-capabilities/ https://github.blog/2023-02-14-github-copilot-now-has-a-bett... When did this change?
  - Rastonbury 2y ago
    When it first launched it, I too didn't know they had changed the model from the original codex which came similar time as gpt-3.5
- jasonjmcghee 2y ago
  > Gpt-4 for chat, 3.5 for code That thread is comparing sidebar chat to inline chat. Doesn't discuss code completions afaict.
localfirst 2y ago
It's miles better. In fact I stopped using expensive GPT-4 Codestral just works, its quick, output is accurate its kinda scary.
esafak 2y ago
It is fast all right, but the quality is not there. I asked it to implement OAuth with Stytch and Ktor and it made everything up. I pointed out the correct name for the package and asked if it really knew the SDK, and it apologized and repeated the same made up code after merely changing the name of the package.
- cco 2y ago
  This is actually why we (Stytch) haven't rolled out any of these "chatbots for code". We have a big list of example questions we get from devs trying us out and we've tested several home grown and third party providers and thus far haven't seen anything good enough that we'd put into production. Thanks for testing this out for us! I'll cross it off our list :)

Not sure how much current Copilot varies from the original Codex, but another set of benchmarks here: https://paperswithcode.com/sota/code-generation-on-humaneval https://paperswithcode.com/sota/code-generation-on-humaneval

ramon156 2y ago

Knowing the training data GH has I doubt it's comparable, then again I don't have the benchmarks

ramon156 2y ago
After typing this I tried the live chat out and it honestly seems a lot more promising than current GH Copilot, very nice!
ssgodderidge 2y ago
Are you saying GH has more than Codestral and therefore GH has a better model? Or that Codestral would be better because Codestral is not littered with "bad" code?
- nkozyra 2y ago
  Bad code is obviously very subjective, but I would wager that GH places a much higher value on feedback mechanisms like stars, issues, PRs, velocity, etc. Their ubiquity likely allows them to automatically cherry-pick less "bad code."
  - nicce 2y ago
    Nothing prevents Mistral do the same if they want to. Issues and and PRs are public information, exposed by APIs, and not that much rate limited.

rohansood15 2y ago

Copilot primarily uses GPT-3.5, which is outclassed by Llama3-70B. And this model claims to be slightly better than Llama3-70B. Edit: For those who don't believe me, https://github.com/microsoft/vscode-copilot-release/issues/664 https://github.com/microsoft/vscode-copilot-release/issues/6.... Gpt-4 for chat, 3.5 for code.

jasonjmcghee 2y ago
GitHub Copilot uses GPT-3.5? I was under the impression it was a custom codex model with a surrogate local model as per https://github.blog/2023-02-14-github-copilot-now-has-a-better-ai-model-and-new-capabilities/ https://github.blog/2023-02-14-github-copilot-now-has-a-bett... When did this change?
- Rastonbury 2y ago
  When it first launched it, I too didn't know they had changed the model from the original codex which came similar time as gpt-3.5
jasonjmcghee 2y ago
> Gpt-4 for chat, 3.5 for code That thread is comparing sidebar chat to inline chat. Doesn't discuss code completions afaict.

localfirst 2y ago

It's miles better. In fact I stopped using expensive GPT-4 Codestral just works, its quick, output is accurate its kinda scary.

esafak 2y ago

It is fast all right, but the quality is not there. I asked it to implement OAuth with Stytch and Ktor and it made everything up. I pointed out the correct name for the package and asked if it really knew the SDK, and it apologized and repeated the same made up code after merely changing the name of the package.

cco 2y ago
This is actually why we (Stytch) haven't rolled out any of these "chatbots for code". We have a big list of example questions we get from devs trying us out and we've tested several home grown and third party providers and thus far haven't seen anything good enough that we'd put into production. Thanks for testing this out for us! I'll cross it off our list :)