6 ms·

Sorry, I have to ask this: how does this compare to ChatGPT?

swyx 3y ago
it doesn't. replit-code-v1-3b is a code LLM, ChatGPT is an app on top of LLMs. it compares to OpenAI Codex, a small version of which is behind GitHub Copilot.
- mritchie712 3y ago
  It (replit-code-v1-3b) is already quite good at explaining code: Input: below is a SQL statement: SELECT CAST(DATE_TRUNC('week', "t1"."TIMESTAMP") AS DATE) AS "WEEK_START", COUNT(\*) AS "EVENT_COUNT" FROM "ANALYTICS"."POSTHOG"."POSTHOG_EVENTS" AS "t1" GROUP BY "WEEK_START" ORDER BY "WEEK_START" LIMIT 2000 Explain this SQL. Respond in JSON format with the following keys: TITLE, DESCRIPTION, TABLES JSON response: output: { "title": "Weekly Events Count", "description": "Count of weekly events", "tables": [ { "name": "POSTHOG_EVENTS", "columns": [ "WEEK_START", "EVENT_COUNT" ] } ] }
- cubefox 3y ago
  Free ChatGPT is based on code-davinci-002 (GPT-3.5), which is used in OpenAI Codex. See https://platform.openai.com/docs/model-index-for-researchers https://platform.openai.com/docs/model-index-for-researchers https://help.openai.com/en/articles/6195637-getting-started-with-codex https://help.openai.com/en/articles/6195637-getting-started-...
- pottertheotter 3y ago
  Sorry, but this is wrong. OpenAI's gpt-3.5-turbo and gpt-4 models (aka ChatGPT) are what it should be compared against. Codex has been deprecated since March, something that has been widely discussed in the LLM world and is prominently noted on the main Codex page: "As of March 2023, the Codex models are now deprecated. Please check out our newer Chat models which are able to do many coding tasks with similar capability" See: https://platform.openai.com/docs/guides/code https://platform.openai.com/docs/guides/code
runnerup 3y ago
It's not crucial that it beat ChatGPT this year. That's a pretty unattainable goal for a group like Replit. From the users POV, none of the current copilots compare favorably against ChatGPT, even Microsoft's OpenAI-powered GitHub Copilot. What's important is that they're preparing for the future by building all the tooling/UI/UX around coding copilots. This way, when costs and feasibility of building ChatGPT-quality LLM's drop and multiple open-source models are available, Replit has the ability to immediately drop them into their production environment. They'll also have the skills and systems to finetune any new models and wring extra performance out of them. This is more important to users than it seems at first because current UX of things like GitHub Copilot don't allow me to use their AI against my codebase the way that I want to (the way I use ChatGPT). Right now GitHub Copilot is a glorified auto-complete, but I want it to do widespread scaffolding, refactoring, and analysis across my whole codebase. Microsoft has access to LLM's that can do this through their control of OpenAI -- but Microsoft lacks the tooling/UI/UX to bring the power of ChatGPT to me as a user of VSCode/IntelliJ/PyCharm/Visual Studio. So if Replit can find more innovative, boundary-pushing ways of integrating LLM's, they won't necessarily need the highest quality LLM's to produce a superior user experience. It's a strong signal that Replit is well-positioned for the future, when ChatGPT-like models are democratized. Hopefully JetBrains is paying attention. They definitely have time to wait a bit more (1-2 years?), but not a lot of time. JetBrains shouldn't solely rely on Github Copilot plug-in to provide their users with LLM's, because it's not clear that the user experience of that plug-in will stay competitive with the user experience that GitHub Copilot will offer directly in VSCode. The IntelliJ/PyCharm plugin may remain "just a fancy auto-complete" while VSCode gets more interactive workflows. Future IDE's with LLM integration require novel, smart, clever UX typically invented only by very creative people. It's also worth noting that Replit is not just trying to be an IDE -- they're also building a marketplace to buy/sell coding work, and establishing a small foothold as a niche cloud computing provider.
- valedan 3y ago
  What does this mean for the future of editors like emacs and (neo)vim? Right now the Copilot plugin for Neovim works pretty much the same as the one for VSCode, but as LLMs get integrated more into IDEs and new workflows are built around them, will the old-school editors be able to keep up? I'm a little worried because I just switched from VSCode to Neovim a few months ago!
heliophobicdude 3y ago
Hard to compare them actually. The thing about ChatGPT is the chat part. It was trained to interact and respond with human conversation. This is more like CodePilot, with code complete based off of actual code

swyx 3y ago

it doesn't. replit-code-v1-3b is a code LLM, ChatGPT is an app on top of LLMs. it compares to OpenAI Codex, a small version of which is behind GitHub Copilot.

mritchie712 3y ago
It (replit-code-v1-3b) is already quite good at explaining code: Input: below is a SQL statement: SELECT CAST(DATE_TRUNC('week', "t1"."TIMESTAMP") AS DATE) AS "WEEK_START", COUNT(\*) AS "EVENT_COUNT" FROM "ANALYTICS"."POSTHOG"."POSTHOG_EVENTS" AS "t1" GROUP BY "WEEK_START" ORDER BY "WEEK_START" LIMIT 2000 Explain this SQL. Respond in JSON format with the following keys: TITLE, DESCRIPTION, TABLES JSON response: output: { "title": "Weekly Events Count", "description": "Count of weekly events", "tables": [ { "name": "POSTHOG_EVENTS", "columns": [ "WEEK_START", "EVENT_COUNT" ] } ] }
cubefox 3y ago
Free ChatGPT is based on code-davinci-002 (GPT-3.5), which is used in OpenAI Codex. See https://platform.openai.com/docs/model-index-for-researchers https://platform.openai.com/docs/model-index-for-researchers https://help.openai.com/en/articles/6195637-getting-started-with-codex https://help.openai.com/en/articles/6195637-getting-started-...
pottertheotter 3y ago
Sorry, but this is wrong. OpenAI's gpt-3.5-turbo and gpt-4 models (aka ChatGPT) are what it should be compared against. Codex has been deprecated since March, something that has been widely discussed in the LLM world and is prominently noted on the main Codex page: "As of March 2023, the Codex models are now deprecated. Please check out our newer Chat models which are able to do many coding tasks with similar capability" See: https://platform.openai.com/docs/guides/code https://platform.openai.com/docs/guides/code

runnerup 3y ago

It's not crucial that it beat ChatGPT this year. That's a pretty unattainable goal for a group like Replit. From the users POV, none of the current copilots compare favorably against ChatGPT, even Microsoft's OpenAI-powered GitHub Copilot. What's important is that they're preparing for the future by building all the tooling/UI/UX around coding copilots. This way, when costs and feasibility of building ChatGPT-quality LLM's drop and multiple open-source models are available, Replit has the ability to immediately drop them into their production environment. They'll also have the skills and systems to finetune any new models and wring extra performance out of them. This is more important to users than it seems at first because current UX of things like GitHub Copilot don't allow me to use their AI against my codebase the way that I want to (the way I use ChatGPT). Right now GitHub Copilot is a glorified auto-complete, but I want it to do widespread scaffolding, refactoring, and analysis across my whole codebase. Microsoft has access to LLM's that can do this through their control of OpenAI -- but Microsoft lacks the tooling/UI/UX to bring the power of ChatGPT to me as a user of VSCode/IntelliJ/PyCharm/Visual Studio. So if Replit can find more innovative, boundary-pushing ways of integrating LLM's, they won't necessarily need the highest quality LLM's to produce a superior user experience. It's a strong signal that Replit is well-positioned for the future, when ChatGPT-like models are democratized. Hopefully JetBrains is paying attention. They definitely have time to wait a bit more (1-2 years?), but not a lot of time. JetBrains shouldn't solely rely on Github Copilot plug-in to provide their users with LLM's, because it's not clear that the user experience of that plug-in will stay competitive with the user experience that GitHub Copilot will offer directly in VSCode. The IntelliJ/PyCharm plugin may remain "just a fancy auto-complete" while VSCode gets more interactive workflows. Future IDE's with LLM integration require novel, smart, clever UX typically invented only by very creative people. It's also worth noting that Replit is not just trying to be an IDE -- they're also building a marketplace to buy/sell coding work, and establishing a small foothold as a niche cloud computing provider.

valedan 3y ago
What does this mean for the future of editors like emacs and (neo)vim? Right now the Copilot plugin for Neovim works pretty much the same as the one for VSCode, but as LLMs get integrated more into IDEs and new workflows are built around them, will the old-school editors be able to keep up? I'm a little worried because I just switched from VSCode to Neovim a few months ago!

heliophobicdude 3y ago

Hard to compare them actually. The thing about ChatGPT is the chat part. It was trained to interact and respond with human conversation. This is more like CodePilot, with code complete based off of actual code