18 ms·

GitHub Copilot Workspace: Technical Preview

308 points by davidbarker 2y ago

LeicaLatte 2y ago
I am still waiting on copilot 2. Instead we got another product, another idiom.
azhenley 2y ago
I just wrote up my experiences of using it for a week: https://austinhenley.com/blog/copilotworkspace.html https://austinhenley.com/blog/copilotworkspace.html TL;DR: Copilot Workspace is a great concept. But the UX and framing is entirely wrong and sets the wrong expectation for users. The current prototype is very slow (5+ minutes for one-line code changes) and the generated code is often buggy or has nothing to do with the specification. It doesn’t help me understand the code. The generated plan is usually good. ChatGPT does a much better job in my head-to-head comparisons (assuming I already know exactly what code is relevant). I'm still optimistic of where it can go from here. I recommend everyone sign up to try it and give the team feedback.
- rgbrenner 2y ago
  Thanks for writing that up. It's what I suspected from my use of CoPilot. I love copilot as an autocomplete tool... but it frequently gets things wrong, and using the chat feature to ask it to complete some task usually just generates code that breaks things. So until that improves, Im skeptical a workspace tool would work. Workspace seems like an awesome idea though.. once the tech is further along.
  - davidbarker 2y ago
    Have you tried Cursor? (https://cursor.sh https://cursor.sh) It's a fork of VS Code with some AI features sprinkled in. It writes around 80% of my code, these days. It also has a few useful features: - a chat interface where you can @-mention files, folders, and even documentation - if you edit a line of code, it suggests edits around that line that are useful (e.g. you change a variable name and it will suggest updating the other uses, which you accept just by pressing Tab) - as you're writing/editing code, it will suggest where your cursor might go next — press Tab and your cursor jumps there
    - cedws 2y ago
      Looks interesting, but I don't really want my code to go via some unknown company. As far as I can tell in "Privacy Mode" code still goes via their servers, they just promise not to store anything (with the caveat that OpenAI retain stuff for 30d).
      - davidbarker 2y ago
        They give you the option to use your own OpenAI/Anthropic/Azure API keys, but in all honesty, I don't know if they still gather information about your code even using your own API keys. You could use something like Little Snitch (on Mac) to check if it makes any calls to their servers. They also allow you to override the URL for the OpenAI models, so although I haven't tried, perhaps you can use local models on your own machine.
        cedws 2y ago
        Unfortunately, it looks like code still goes via their servers: https://cursor.sh/privacy https://cursor.sh/privacy > Even if you use your API key, your requests will still go through our backend! > That's where we do our final prompt building.
        davidbarker 2y ago
        Ah, that's unfortunate.
studenthrow3831 2y ago
Student here: I legitimately cannot understand how senior developers can dismiss these LLM tools when they've gone from barely stringing together a TODO app to structuring and executing large-scale changes in entire repositories in 3 years. I'm not a singulatarian, but this seems like a brutal S-curve we're heading into. I also have a hard time believing that there is enough software need to make such an extreme productivity multiplier not be catastrophic to labor demand. Are there any arguments that could seriously motivate me to continue with this career outside of just blind hope that it will be okay? I'm not a total doomer, currently 'hopium' works and I'm making progress, but I wish my hopes could at least be founded.
- zackmorris 2y ago
  After programming for 35 years since I was 12 and learning everything from the transistor level up through highly abstracted functional programming, I'm a total doomer. I put the odds of programming being solved by AI by 2030 at > 50%, and by 2040 at > 95%. It's over. Programming (automating labor) is the hardest job there is IMHO, kinda by definition. Just like AGI is the last problem is computer science. You noticed the 3 year pace of exponential growth, and now that will compound, so we'll see exponential-exponential growth. AIs will soon be designing their own hardware and playgrounds to evolve themselves perhaps 1 million times faster than organic evolution. Lots has been written about this by Ray Kurzweil and others. The problem with this is that humans can't adapt that fast. We have thought leaders and billionaires in total detail of the situation. Basically that late stage capitalism becomes inevitable once the pace of innovation creates a barrier to entry that humans can't compete with. The endgame will be one trillionaire or AI owning the world, with all of humanity forced to perform meaningless work, because the entity in charge will lack all accountability. We're already seeing that now with FAANG corporations that are effectively metastasized AIs using humans as robots. They've already taken over most governments through regulatory capture. My personal experience in this was that I performed a lifetime of hard work over about a 25 year period, participating in multiple agencies and startups, but never getting a win. So I've spent my life living in poverty. It doesn't matter how much I know or how much experience I have when my mind has done maybe 200 years of problem solving at a 10x accelerated rate working in tech - see the 40 years of work in 4 years quote by Paul Graham. I'm burned out basically at a level of no return now. I'm not sure that I will be able to adapt to delegating my labor to younger workers like yourself and AI without compromising my integrity by basing my survival on the continued exploitation of others. I'd highly recommend to any young people reading this to NOT drink the kool-aid. Nobody knows what's going to happen, and if they say they do then they are lying. Including me. Think of tech as one of the tools in your arsenal that you can use to survive the coming tech dystopia. Only work with and for people that you can see being someday. Don't lose your years like I did, building out someone else's dream. Because the odds of failure which were once 90% in the first year are perhaps 99% today. That's why nobody successful pursues meaningful work now. The successful people prostitute themselves as influencers. I'm still hopeful that we can all survive this by coming together and organizing. Loosely that will look like capturing the means of production in a distributed way at the local level in ways that can't be taken by concentrated wealth. Not socialism or communism, but a post-scarcity distribution network where automation provides most necessities for free. So permaculture, renewable energy, basically solarpunk. I think of this as a resource economy that self-evidently provides more than an endlessly devaluing and inflating money supply. But hey, what do I know. I'm the case story of what not to do. You can use my story to hopefully avoid my fate. Good luck! <3
mistrial9 2y ago
Github copilot trained on GPL-licensed code? Brad Smith you continue to outdo yourself
- naikrovek 2y ago
  Do you see that stated somewhere or are you just trying to stir people up?
  - mistrial9 2y ago
    can we review all current lawsuits and their discovery status? pointers welcome
idan 2y ago
Hi All! GitHub Next here, happy to answer questions about Copilot Workspace and Next in general <3
- JTyQZSnP3cQGa8B 2y ago
  Can I install it locally without this tool having an internet connection?
  - idan 2y ago
    No, it's a webapp, like github.com.
- Yenrabbit 2y ago
  Looks extremely cool. Have you shared / do you plan to share examples of this in practice? I would be great to read some case studies or even just browse some actual PRs rather than trying to judge from the catchy video :)
  - idan 2y ago
    GitHub Stars have had access to this since last week, and a few of them have made in-use videos: - https://www.youtube.com/watch?v=FARf9emEPjI https://www.youtube.com/watch?v=FARf9emEPjI by Dev Leonardo - https://www.youtube.com/watch?v=XItuTFn4PWU https://www.youtube.com/watch?v=XItuTFn4PWU by Ahmad Awais And keep an eye on https://x.com/githubnext https://x.com/githubnext, we'll be sharing / linking to more in-action things. Any PR created with Workspace will have a link to a readonly copy of the workspace so you can see how it happened. We expect those to start circulating as people get access!
cryptoz 2y ago
This is quite similar to my idea and prototype I applied to YC with just last week. Uh-oh! Haha. I did call out GitHub as a primary competitor so I did see this coming - and I guess on second looks I'm still quite differentiated. All these new tools will overlap some but also have their own pros/cons since there are so many different ways to approach building an AI code tool! Just yesterday I put up a cheap $5 linode running a prototype but not ready for signups yet. https://codeplusequalsai.com/ https://codeplusequalsai.com/ I think I have some good ideas about how to get the the LLM to modify code - specifically working with ASTs. I wonder how GitHub prompts the LLM to get usable code modifications out of it. How often does it encounter errors?
- sqeaky 2y ago
  Good luck! I hope to see many competitors in this space because hegemony in this space would inhibit the software industry as a whole. Figure out your differentiators and figure what copilot sucks at. Succeeding here can be done.
esafak 2y ago
I don't use Copilot in particular but in the software space, everything is constantly changing, and the models are trained infrequently so they are never familiar with the things I ask about. Reality is outrunning the models for now but that may change.
- throwup238 2y ago
  There's a Black Mirror-esque movie plot idea in there! Everyone starts using bionic implants to filter out offensive parts of reality like people they don't like and stuff, until one day all the filters start falling out of sync with reality... ...and then aliens invade.
  - jl6 2y ago
    They Live... kinda?
PoignardAzur 2y ago
I think Microsoft is going the wrong direction with Copilot (though it's a reasonable direction given their incentives). Right now Copilot is terrible at large changes to complex codebases; the larger and more complex, the worse. But it's great at suggesting very short snippets that guess exactly what you were in the middle of writing and write it for you. I wish Copilot focused more on the user experience at the small scale: faster and smaller completions, using the semantic info provided by the IDE, new affordances besides "press tab to complete" (I'd love a way for Copilot to tell me "your cursor should jump to this line next"), etc. This whole focus on making the AI do the entire end-to-end process for you seems like a dead end to me.
- konschubert 2y ago
  Totally.
- vizzier 2y ago
  proper insertion points in completions akin to older Visual Studio templates would probably be ideal, but you can use ctrl + right arrow in vscode to accept completions one word at a time, rather than a whole block with tab.
- bloomfieldj 2y ago
  I find Cursor’s Copilot++ is miles ahead of GitHub’s in terms of speed and autocomplete helpfulness. They’re also working on the “your cursor should jump to this line next" feature, but I haven’t relied on many of its suggestions yet. It’s available in their vscode fork, but doesn’t seem to be in their docs yet.
  - PoignardAzur 2y ago
    Had a glance. Copilot++ looks intriguing, though the landing page is terrible (I can't even see what the demos are trying to show). I might give it a try.
- lenerdenator 2y ago
  > But it's great at suggesting very short snippets that guess exactly what you were in the middle of writing and write it for you. I found it reasonably good when I described what code should do through comments and let it generate based on that, evaluated the output, moved on to the next piece of logic, and repeated. Basically, chaining the snippets together. Then again I'm doing Python back-end web development, so not something terrifically hard.
rahulpandita 2y ago
https://githubnext.com/projects/copilot-workspace/ https://githubnext.com/projects/copilot-workspace/
echelon 2y ago
I'm a fan of AI-empowered coding, but I'm not a fan of a single SaaS vendor dominating the entire software lifecycle. That's an incredibly dangerous proposition. I'd love to see anyone but Microsoft push ahead, and ideally, lots of players.
Protostome 2y ago
Curious to see how it goes. For 20 years I have been working with Vim and recently added the Co-pilot plugin which is awesome. I can't however see myself using a different editor/IDE than the good'ol vim. It's not that I haven't tried, but all those IDEs tend to be bloated and consume tons of memory.
- idan 2y ago
  When video terminals first came out, everyone started out using line editors even though line editors make no sense when you can display arbitrary buffer. It took a while until editors changed to be "screen native". But they did change, meaningfully. When GUIs first came out, editors were just "terminal editors in a window". Took a while for the modern concept of an IDE to happen, with hovers, red squigglies, sidebars, jump to definition. All of that was possible on the first day of the GUI editor! But it took a while to figure out what everyone wanted it to be. I think we're at a similar inflection point. Yeah, everyone today (myself included) is comfortable in the environment we know. VS Code is lovely. And AI (plus realtime multiplayer) is not a display technology. But I think it's a material technology shift in the same vein as those two moments in history. I would not bet that the next thirty years is going to continue look like today's VS Code. I don't know to say what it WILL look like — we have to keep prototyping to find out.
  - Protostome 2y ago
    I mostly agree with all of your points. But IMHO, if (and that's a big if nowadays, with modern web and mobile dev) text editing is the core task of the app your'e using as your IDE, then all the graphics are just redundant. All the highlighting, hovers / etc can be done via much simpler graphics that is memory and CPU efficient. As a dev, I would like to IDE developers to invest in core functionality, rather than a fancy GUI. Take the electron IDE for example. It embeds the chrome runtime which is a total waste, given that i just want to edit some text files.
idan 2y ago
Hello! GitHub Next here, happy to answer questions and unpack how we think about AI tools for developers (spoiler: it's less about codegen and more about helping with the rest of the dev cycle — building an understanding of how the system works, clearly specifying how it should change, etc)
- ahnix 2y ago
  [dead]
- roboyoshi 2y ago
  Interesting stuff! I've been trying ollama/gpt + continue.dev and copilot in VSCode for a bit now and the chat-style assistant is a huge plus. Especially in DevOps (my main work) the codegen is rather unhelpful, but the ability to let LLMs explain some sections and help in rubber-ducking is huge. I see that a good output requires a good input (prompt). How does copilot workspace determine a good input for the prompt? I see that in the github repo there is already a bunch of "Tips and Tricks" to get better results. What is your experience so far? Should we change our way of creating issues (user-stories / bug-reports, change-requests) to a format that is better understood by AI/Copilot? (half-joking, half-serious).
  - idan 2y ago
    Well, that's basically the heart of Copilot Workspace! The whole UX is structured to make it easy for the human to steer. - Alter the "current" bullet points in the spec to correct the AI's understanding of the system today - Alter the "proposed" bullet points in the spec to correct the AI's understanding of what SHOULD be - Alter files/bullets in the plan in order to correct the AI's understanding of how to go from current to proposed. That said, I think there's definitely a future where we might want to explore how we nudge humans into better issue-writing habits! A well-specified issue is as important to other humans as it is to AI. And "well-specified" is not about "more", it's about clarity. Having the right level of detail, clear articulation of what success means, etc.
- rjindael 2y ago
  If it is more about code planning, how much different is it than simply telling ChatGPT the overall structure of your code and asking it to give you a rudimentary plan on what to do next? Would it be able to actually execute steps of the plan by generating code, then creating PRs for it? I feel like this is a great tool for our team since my understanding of the announcement is that it is more or less functionally equivalent to hiring another programmer on your team (or, if not that, at the very least having a really useful assistant.) Kudos to the GitHub team and I have immensely enjoyed using Copilot thus far to increase my productivity :-)
msoad 2y ago
This should've been done in VSCode. Most developers don't want to write code in a browser. Once VSCode extension is built they could generalize it to other editors. I'm sure they are thinking of VSCode integration and I am hopping for that to be available soon
- idan 2y ago
  We are indeed thinking about how to do that in a way that feels natural in VS Code. :)
- pluc 2y ago
  It's very different to have the code and suggest things than it is to obtain the code and suggest things, without introducing noticeable delay in processing that code
- lispisok 2y ago
  I wonder if AI tools is going to kill editor diversity. Every one of these tools VS Code is the first class citizen and other editors are an afterthought. Sure people can write their own Emacs package but that's only if the tool developers enable it and the experience is usually not as good as the official VS Code version. I can also see the future where not using VS Code is a signal you are less reliant on AI tools to code
  - padthai 2y ago
    A thousands of small cuts, almost everything comes first to VSCode before other editors, even mainstream things are underrepresented in most other editors (Jupyter, commercial integrations, etc…). Although that was true about Atom too though?
  - hackermatic 2y ago
    We used to say the same about Eclipse!
  - freedomben 2y ago
    This is a great fear of mine as a (neo)vim user. I've repeatedly encountered situations already where it was assumed everyone was using VS code, and I only see that getting worse as more and more integrations get built into IDEs.
Lucasoato 2y ago
Am I the only one disappointed by Github copilot quality lately? I get much better responses by copying and pasting on ChatGPT, rather than the suggestions in the vscode plugin.
- smrtinsert 2y ago
  I'm pretty sure they toggle between gpt 3 and 4 depending on how much traffic they're getting in order to make sure they can meet all suggestions.
  - frontalier 2y ago
    the machines were not supposed to get tired
- hackermatic 2y ago
  People have been anecdotally reporting and investigating similar problems since at least last year[0], and it's entirely possible that changes to improve one aspect of a model could make it much worse at other aspects without careful regression testing and gradual rollout. I think models intended to solve every problem make it very hard to guarantee they can solve any particular problem reliably over time. Imagine if a million developers simultaneously got much worse at their jobs! [0] https://arstechnica.com/information-technology/2023/07/is-chatgpt-getting-worse-over-time-study-claims-yes-but-others-arent-sure/ https://arstechnica.com/information-technology/2023/07/is-ch...
deleted 2y ago
[deleted]
sdesol 2y ago
Interesting. I don't think the AI code generation will live up to developer expectations, but I do see the value in "project management for developers". The value I see with workspace is not the code generation, but the ability to help developers organize their thoughts.
- alfalfasprout 2y ago
  The real problem is a large part of the enormous hype train is investors thinking they can replace talented engineers w/ AI. That's fundamentally doomed for failure. What AI can really do well is take an already competent engineer and suddenly get rid of a lot of the annoying tedium they had to deal with. Whether it's writing boilerplate, doing basic project management, organizing brain dumps/brainstorming, etc.
  - nprateem 2y ago
    Yes. And they've also introduced a new kind of tedium in going round in circles while it doesn't understand you, then you give up and go back to SO.
  - sdesol 2y ago
    I do think workspace by GitHub is a smart move. What is clear as day is, we don't have enough data to model how developers think and iterate to solve problems. With workspace, I see it as a way to capture structured data on how developers think. This is certainly a long game though. I think GitHub with MS money can continue to lose money on Copilot for the next 5 years to gather data. For other VC ventures, I don't think they can wait that long.
  - godzillabrennus 2y ago
    These AI coding assistants already do what you are describing. At some point in the future, AI should be capable enough to do the job of a competent developer. This is not a short-term milestone, IMHO.
  - idan 2y ago
    Bingo.
happypumpkin 2y ago
> At last count, Copilot had over 1.8 million paying individual and 50,000 enterprise customers. > Copilot loses an average of $20 a month per user, according to a Wall Street Journal report, with some customers costing GitHub as much as $80 a month. Presumably at some point they need to actually make money on this? That is a $432 million / yr loss just on individual users.
- Yenrabbit 2y ago
  Despite being widely repeated they've publically denied this claim a few times (for e.g. https://x.com/natfriedman/status/1712140497127342404 https://x.com/natfriedman/status/1712140497127342404, https://x.com/simonw/status/1712165081327480892 https://x.com/simonw/status/1712165081327480892)
danenania 2y ago
For anyone who might be interested in an open source, terminal-based approach to using AI on larger tasks and real-world projects, I'm building Plandex: https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex I've tried to create a very tight feedback loop between the developer and the LLM. I wanted something that feels similar to git. Apart from the planning and code generation itself, Plandex is also strongly focused on version control. Every interaction with the model is versioned so that it's easy to try out different strategies and backtrack/revise when needed. I just released a new update (literally a few minutes ago) that includes some major improvements to reliability, as well as support for all kinds of models (previously it had been OpenAI-only). It's been fun trying out Claude Opus and Mixtral especially. I'd love to hear people's thoughts!
- Bnjoroge 2y ago
  just played with it. pretty solid! questions for me, how do you validate the new changes will work without any issues after applying them? the biggest issue I have with most code generators is the feedback loop from suggested code -> testing it out -> failing, doing it again. would be great if this was more seamless. additionally, would be helpful to control the code generation process in real-time.
  - rahulpandita 2y ago
    GitHubNext Dev here: We have a terminal feature that allows you to connect to sandbox in the cloud for you to validate these changes before pushing them out to your repo
  - danenania 2y ago
    Thanks for trying it! So far, the the testing and validation stage has been left to the developer. While this could change in the future, my experience has been the models aren't quite good enough yet to make an auto-test/auto-fix loop like you've described the default behavior. You end up with a lot of tire-spinning where you're burning a lot of tokens to fix issues that a human can resolve trivially. I think it's better for now to use LLMs to generate the bulk of a task, then have the developer clean up and integrate rather than trying to get the LLM to do 100%. That said, you can accomplish a workflow like this with Plandex already by piping output into context. It would look something like: plandex new plandex load relevant_context.ts some_more_context.ts plandex tell 'some kind of complex task' # ...Plandex does its thing, but doesn't get it 100% right npm test | plandex load plandex tell 'please fix the problems causing the failed tests' As the models improve, I'm definitely interested in baking this in to make it more automated.
7thpower 2y ago
I use Copilot in VS code for work and it is great for auto complete, but the chat experience is very poor compared to tools like Cursor. The inline edits are even worse. Long story short, is I don’t have a lot of confidence in the product right now. That being said, I am very optimistic on the product long term and I generally like the vision.
- l5870uoo9y 2y ago
  I cancelled my subscription mainly because it made me neglect overall structure and instead autocomplete quick inline and repetitive solutions. Haven’t missed it.
rgs224 2y ago
was able to get around waitlist: copilot-workspace.githubnext.com /<owner>/<repo>?task=<description>
- throwup238 2y ago
  Great job! Worked for me to create the plan and now slowly generating the file changes for the plan.
- jckwind 2y ago
  i think its patched :/
  - neuralnerd 2y ago
    Yup. Got patched :/
singluere 2y ago
While I've not used this product, I've created somewhat similar setup using open source LLMs that runs locally. After having used it for about three months, I can say that debugging LLM prompts was far more annoying than debugging code. Ultimately, I ended up abandoning my setup and going in favor of writing code the good old fashioned way. YMMV
- BoorishBears 2y ago
  Creating this using Open Source LLMs would be like saying you tried A5 Wagyu by going to Burger King, respectfully. I think benchmarks are severely overselling what open source models are capable of compared to closed source models.
  - Zambyte 2y ago
    I really don't think they're being over sold that much. I'm running llama 3 8b on my machine, and it feels a lot like running claude 3 haiku with a much lower context window. Quality wise it is surprisingly nice.
    - BoorishBears 2y ago
      Llama 3 just came out so they couldn't have used it, and Claude Haiku is the smallest cheapest closed source model out there from what I've seen. Github is likely using a GPT-4 class model which is two (massive) steps up in capabilities in Anthropic's offerings alone
      - Zambyte 2y ago
        Yeah I just mentioned Llama to point out that the open weight models have been really catching up. Microsoft is almost certainly using GPT-4 given their relationship with ClosedAI, but I would definitely not put GPT-4 (nor Turbo) "two massive steps up" from Claude 3 Opus. I have access to both through Kagi, and I have found myself favoring the responses of Claude to the point where I almost never use GPT(TM) anymore.
nextworddev 2y ago
RIP Devin
sqs 2y ago
Cool idea! Agree on the end vision. The alpha is in figuring out how to get there and making it work really well along the way. Also, why isn’t anybody connecting the Copilot Workspace announcement to Devin? Biggest company in the world announces a release of a competitor to the most widely seen dev product announcement of the year? Only saw one incidental mention of the connection.
kylecarbs 2y ago
I was able to try it (without being approved) here: https://copilot-workspace.githubnext.com/ https://copilot-workspace.githubnext.com/
- asadm 2y ago
  hmm doesnt work anymore
mlhpdx 2y ago
I’m letting it sink in that GitHub thinks it would be a win if 10%+ of the world population is involved in creating software. I have no hope of hiring an electrician in that world.
- idan 2y ago
  You have succeeded at hiring an electrician in THIS world? What's their number? Do they actually show up when they say they will?!?!?!!1!one
ilaksh 2y ago
I think these types of tools will become really effective when they stop trying to target every single programming language and platform ever created and focus on one specific version of an integrated platform. Most businesses don't actually need software to be in whatever flavor of the month the dev team happens to be interested in. They just want to be able customize their software. By restricting the scope somewhat, you make it much more feasible to make sure the model has the training and/or knowledge ready for retrieval to fulfill tasks using a specific stack. So I see this type of thing as quickly evolving into a tool for non-developers. And within a few years, these tools will cut into software engineering jobs. It will become part of the evolution of no-code and low-code.
- jstummbillig 2y ago
  As long as it does the language that my business currently uses, I am all for that! Jokes aside, I am not sure I buy the premise. I have read somewhere (citation required) that LLMs get better at other things after having learned how to code. And maybe they also get better at coding in one language after having learned a bunch.
colemannerd 2y ago
I wish there were fixes to devcontainers before doing adding copilot. I really want declarative, repeatable builds that are easily used in both Codespaces AND Actions. I know all of the functionality is theoretically there in devcontainers.json, it is so manual to configure and confusing, that anytime I've done it, I use it for 2 weeks and then just go back to developing on local because I don't have time to keep that up. ESPECIALLY if you're deploying to AWS cloud and also, want to use alternative package managers like poetry, uv, yarn, jsr, etc.
nojvek 2y ago
I use Github co-pilot but 55% productive is a bullshit number. Perhaps 1% may be. Most of co-pilot suggestions are either simple pattern matches or subtle hallucinations where I have to catch and fix silly bugs. Github Chat is not very useful at understanding what the code is doing. Tried it once or twice and gave up. The hype will help with Microsoft Stock though. Seems like bean counter management is taking over.
- Bjorkbat 2y ago
  I was actually about to make a comment on that. They got that number from some dumb "experiment" they did where told two groups of developers to implement an HTTP server in Javascript. The group with Copilot got the job done 55% faster (https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/?ocid=AIDcmmc3fhtaow_SEM__k_Cj0KCQjwir2xBhC_ARIsAMTXk85qF--5Z_RO3fCcRnFE6itQ3cKFQvIGLnsyaF-Gwd4FdZFoisM7GtUaAou4EALw_wcB_k_#finding-2-but-speed-is-important-too https://github.blog/2022-09-07-research-quantifying-github-c...) So yeah, they did an experiment with <100 developers asking them to create implement something that only took the control group 3 hours to finish from scratch, and from this we got the "55% more productive" statistic.
  - chasd00 2y ago
    That's really disappointing and a little bit scary. That seems like a task tailor fit for an LLM, it's been done a million times before, follows an established protocol, and the implementations are all basically the same. Granted a lot of code is like that but a more realistic task may have been something like "create an online ordering system for a cake company that sells different kinds of cakes, has a sale every so often, and uses coupons printed in the local paper". That task is more ambiguous and better reflects what software devs are tasked with day to day IMO.
  - deleted 2y ago
    [deleted]
daviding 2y ago
One side effect that I don't think I like of these tools is the inevitable push for developers to just use the most popular languages. Javascript and python are the main LLM trained languages, so it just self reenforces that it is best to use those for everything. There is something a bit sad about that. I guess the dream is that they are just intermediatory data anyway, like a chatty bytecode layer.
elwell 2y ago
> and is expressly designed to deliver–not replace–developer creativity I'm not fooled by such corp speak.
- strix_varius 2y ago
  If they could, they would!
Bjorkbat 2y ago
You can see my earlier comment somewhere here, but I feel obliged to remind everyone that the claim that Github Copilot made developers "55% more productive" came from a study where they asked 100 developers to implement an HTTP server in Javascript, split the group up roughly 50/50, and gave one group Github Copilot. The Copilot group did it in an hour and 11 minutes, whereas the control group got it done 2 hours and 41 minutes. https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/?ocid=AIDcmmc3fhtaow_SEM__k_Cj0KCQjwir2xBhC_ARIsAMTXk85qF--5Z_RO3fCcRnFE6itQ3cKFQvIGLnsyaF-Gwd4FdZFoisM7GtUaAou4EALw_wcB_k_#finding-2-but-speed-is-important-too https://github.blog/2022-09-07-research-quantifying-github-c... That's where the 55% number is coming from. It's coming from this experiment, and only this experiment. So yeah, if you're wondering why you aren't somehow 50% more productive when using Github Copilot, it's probably because you're not implementing a simple task that's been done to death in countless tutorials.
- mucle6 2y ago
  Do you not have a good experience with github copilot? I recently started using it and it will write functions that are 90% right based on comments, and I just modify what I want. I've felt at least a 2x speedup, maybe even 4x. That said I'm working on a new project where I'm writing a lot of code. For making small changes I could see how its much less valuable.
  - chasd00 2y ago
    "I just modify what I want." i think that's key, you use the llm to get something started and then change/fix/enhance to get what you want. That works for me too but the folks that want to prompt an llm from nothing to a finished application are in for a rough ride.
mucle6 2y ago
I wonder if we can use "copilot-ability" as a proxy for code base complexity. In other words, if copilot can't help you write code, then could that mean your code base is too complicated for a beginner to understand and modify?
- chasd00 2y ago
  I hope not but i bet you're right. I'm sure there's lots of people working on llms to generate a "cognitive complexity" score like some of the static code analyzers we have. It will be so ripe with false positives as to be effectively worthless however the score is an easy thing to build a metric, report, and policy on... just like with the analyzers today.
- layer8 2y ago
  That assumes that it would be reasonable to expect a beginner to understand the code base of any nontrivial application or system. I don’t think that’s a reasonable expectation, if only because there are countless requirements, assumptions, experiences made, priorities and design decision that have to be known to understand the code. It’s also a question of economy. You can always try to make a code base easier to understand and to document anything and everything, but it comes at a cost. It doesn’t help if a code base is in principle understandable by a beginner if they have to spend months reading documentation first, and someone had to spend months if not years to write it.
iknownthing 2y ago
I noticed there were a lot of YC companies trying to do similar things. I'm not sure how they think they have a shot against Copilot.
jaredcwhite 2y ago
So GitHub has outright declared war on competence in programming. Great, just great. The code examples is this blog post are horrible…I'd be embarrassed to share any such demonstration of generated code offering these solutions. Reject that PR in production, for the love of god! Funny that I already started migrating my personal projects off of GitHub and onto Codeberg over the weekend. I'll be rapidly accelerating that process after today. Copilot has gone from vaguely irritating (and of course I refuse to use it on ethical grounds) to an overall direction for the company I *intensely* disagree with.
sqs 2y ago
You can try out GitHub Copilot Workspace now in read-only mode: https://copilot-workspace.githubnext.com/AnandChowdhary/anandchowdhary.com?shareId=fe652b2c-e6a3-4ab9-841f-a071f5789c1b https://copilot-workspace.githubnext.com/AnandChowdhary/anan...
Cilvic 2y ago
I'm not sure why they insist on code spaces instead of running this inside vs code? Setups/working looks pretty similar to aider [1] that I've been using and liking, Aider is smaller steps than plandex, but plandex went into some kind of loops a couple of times so I stoped using it for now. [1] https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider
- lostintangent 2y ago
  We definitely intend to explore a VS Code extension in the not too distant future. And we decided to build a web client + integrated cloud terminal, simply because that allowed us to create an experience that is one-click away from an issue, and could be accessed from anywhere and any device (i.e. we’ve deeply optimized CW for mobile, and I do quite a bit of code thinking in that modality). In the meantime, when you open a Codespace from Copilot Workspace, you could open that Codespace in VS Code desktop. And use that as a companion editor to the web client (since we bi-directionally sync file changes between them). But totally agreed that a more integrated VS Code experience will be compelling!
  - Cilvic 2y ago
    > simply because that allowed us to create an experience that is one-click away from an issue, That's indeed pretty cool and as you said it's not an either-or. Thanks for providing more background.
- danenania 2y ago
  > plandex went into some kind of loops a couple of times so I stoped using it for now. Hey, Plandex creator here. I just pushed a release today that includes fixes for exactly this kind of problem - https://github.com/plandex-ai/plandex/releases/tag/cli%2Fv0.9.0 https://github.com/plandex-ai/plandex/releases/tag/cli%2Fv0.... -- Plandex now has a much better 'working memory' that helps it not to go into loops, repeat steps it's already done, or give up too early. I'd love to hear whether it's working better for you now.
  - 2y ago
lostintangent 2y ago
If folks want to see what Copilot Workspace looks like, here’s a sample session where I addressed a feature request in an OSS VS Code extension I maintain: https://copilot-workspace.githubnext.com/lostintangent/gitdoc/issues/52?shareId=8a0244c3-95d7-4bfc-8b85-f1a9d7c77189 https://copilot-workspace.githubnext.com/lostintangent/gitdo.... You can see the originating issue and the resulting PR from there. And note that while the initial spec/plan/code was mostly good, I iterated on a couple parts of the plan, and then made a minor tweak to the code manually (everything in CW is editable). Which is a key part of our goal with CW: to help bootstrap you with a task (or think out loud with AI), and then provide the iteration primitives to explore further.
steve_adams_86 2y ago
Watching the examples and other people's demos, I get the sense that this product completely ignores what makes good software. It has no "big picture" plan or contextual awareness required to make those nuanced yet critical decisions in the far ends of applications. Those cases where you badly need domain expertise, understanding of how the user interfaces with the product, awareness of existing technical debt, and so on. The scope it can assist in is too limited. It is exactly the kind of thing I'd use to generate self-destructing spaghetti. The thing is, people with less experience will eat this up because it can do some tasks with ease which might not be familiar to them yet. Over time, the code they produce will be a mine field of incongruent patterns and solutions. It appears to support the hypothesis that these tools may actually create more work, but I worry that the work will be much like that created by outsourced developers shipping duct-taped nonsense back to North America throughout the last 20 years or so.
- mlsu 2y ago
  And, on top of that, the first few fixes will actually work! They will look good to PMs and to the business! Look at the velocity! The velocity is good enough to scale back the size of the team, it's good enough to mandate a pace that is only possible with AI doing automated submission and review of PRs, it's good enough to not have any kind of formal design work. Just ship. After several months of this, any additional bug fix by the AI is just adding to the existing morass of tech debt created by the AI. Slowly but surely, the AI will stop being able to submit a PR that doesn't result in a regression. The people who actually care about software quality aren't on new projects; those are run by juniors and copilot. Nope, they're going to be stuck on maintenance of this garbage. And meanwhile, that new team working on new feature X is moving so quickly... better give them a raise! The people who suffer here are, of course, the non-marginal user, who will have to contend with every new feature breaking every other existing feature that they like in the software.
- idan 2y ago
  That hasn't been our experience using it in-house! It's not perfect, but not every software engineering task is some galaxy-brain architectural shit. Sometimes, you have to lay down some bricks. And sometimes, to lay down bricks, you need to touch more than one tiny patch of code. Having something round up the likely areas of the codebase that needs touching feels magical. It doesn't always succeed! But it feels pretty magical to get that boost when you're new to some part of a codebase (which, real talk, code I wrote > 1 month ago, I must page back into memory). Making it easy for me to progressively add context for the model is an accurate analogue for how I think as a developer when tackling a task. I have to build a mental model of how things work. And then a plan for how I'm going to change it. Maybe for the kinds of tasks you usually tackle, it won't have value. But the amount of context it's attempting to bring to bear on whatever task you give it is categorically more — and better — than any other tool I've seen. I have seen (and been the author of) spaghetti. Could I make CW generate spaghetti? Surely. That's why it's a tool for developers, not a substitute for developers.
vegancap 2y ago
Jesus christ, I don't know whether to be excited or terrified by some of these new AI releases...
burntcaramel 2y ago
Pretty sure I wouldn’t want to work on a team using this. “Who wrote this code, it has a serious flaw?” “The customer needed that feature quickly so we used Copilot. No sorry, I don’t know how it works, I trust that it does. I glanced over the code and it looked about right.” “Did you read the checklist that it generated? The flaw stems from an assumption it made.” “Oh no, I didn’t see that. To be honest I don’t read the checklists it makes.”
benzguo 2y ago
it's so over for Devin
TriNetra 2y ago
can we use gpt to convert say, a Xamarin Android app code into React-Native? has anyone tried similar conversion successfully?
rany_ 2y ago
Is this sort of like a response to Devin? I'm having a hard time understanding what this is.
ianbutler 2y ago
A little late to the party here, but we (https://www.bismuthos.com https://www.bismuthos.com) offer something already to scratch this itch. We provide a workspace to build Python backends. Chat on the left, code and visual editors on the right. However, we also handle deployments, data storage (we have a blob store), serving (we built a home grown function runtime) and logging. The experience is tightly integrated with the copilot and the idea is to get ideas off the ground as quickly as possible. We're early days and you can try it for free right now.
joandaniel497 2y ago
[dead]

308 points by davidbarker 2y ago

I am still waiting on copilot 2. Instead we got another product, another idiom.

I just wrote up my experiences of using it for a week: https://austinhenley.com/blog/copilotworkspace.html https://austinhenley.com/blog/copilotworkspace.html TL;DR: Copilot Workspace is a great concept. But the UX and framing is entirely wrong and sets the wrong expectation for users. The current prototype is very slow (5+ minutes for one-line code changes) and the generated code is often buggy or has nothing to do with the specification. It doesn’t help me understand the code. The generated plan is usually good. ChatGPT does a much better job in my head-to-head comparisons (assuming I already know exactly what code is relevant). I'm still optimistic of where it can go from here. I recommend everyone sign up to try it and give the team feedback.

rgbrenner 2y ago
Thanks for writing that up. It's what I suspected from my use of CoPilot. I love copilot as an autocomplete tool... but it frequently gets things wrong, and using the chat feature to ask it to complete some task usually just generates code that breaks things. So until that improves, Im skeptical a workspace tool would work. Workspace seems like an awesome idea though.. once the tech is further along.
- davidbarker 2y ago
  Have you tried Cursor? (https://cursor.sh https://cursor.sh) It's a fork of VS Code with some AI features sprinkled in. It writes around 80% of my code, these days. It also has a few useful features: - a chat interface where you can @-mention files, folders, and even documentation - if you edit a line of code, it suggests edits around that line that are useful (e.g. you change a variable name and it will suggest updating the other uses, which you accept just by pressing Tab) - as you're writing/editing code, it will suggest where your cursor might go next — press Tab and your cursor jumps there
  - cedws 2y ago
    Looks interesting, but I don't really want my code to go via some unknown company. As far as I can tell in "Privacy Mode" code still goes via their servers, they just promise not to store anything (with the caveat that OpenAI retain stuff for 30d).
    - davidbarker 2y ago
      They give you the option to use your own OpenAI/Anthropic/Azure API keys, but in all honesty, I don't know if they still gather information about your code even using your own API keys. You could use something like Little Snitch (on Mac) to check if it makes any calls to their servers. They also allow you to override the URL for the OpenAI models, so although I haven't tried, perhaps you can use local models on your own machine.
      - cedws 2y ago
        Unfortunately, it looks like code still goes via their servers: https://cursor.sh/privacy https://cursor.sh/privacy > Even if you use your API key, your requests will still go through our backend! > That's where we do our final prompt building.
        davidbarker 2y ago
        Ah, that's unfortunate.

studenthrow3831 2y ago

Student here: I legitimately cannot understand how senior developers can dismiss these LLM tools when they've gone from barely stringing together a TODO app to structuring and executing large-scale changes in entire repositories in 3 years. I'm not a singulatarian, but this seems like a brutal S-curve we're heading into. I also have a hard time believing that there is enough software need to make such an extreme productivity multiplier not be catastrophic to labor demand. Are there any arguments that could seriously motivate me to continue with this career outside of just blind hope that it will be okay? I'm not a total doomer, currently 'hopium' works and I'm making progress, but I wish my hopes could at least be founded.

zackmorris 2y ago
After programming for 35 years since I was 12 and learning everything from the transistor level up through highly abstracted functional programming, I'm a total doomer. I put the odds of programming being solved by AI by 2030 at > 50%, and by 2040 at > 95%. It's over. Programming (automating labor) is the hardest job there is IMHO, kinda by definition. Just like AGI is the last problem is computer science. You noticed the 3 year pace of exponential growth, and now that will compound, so we'll see exponential-exponential growth. AIs will soon be designing their own hardware and playgrounds to evolve themselves perhaps 1 million times faster than organic evolution. Lots has been written about this by Ray Kurzweil and others. The problem with this is that humans can't adapt that fast. We have thought leaders and billionaires in total detail of the situation. Basically that late stage capitalism becomes inevitable once the pace of innovation creates a barrier to entry that humans can't compete with. The endgame will be one trillionaire or AI owning the world, with all of humanity forced to perform meaningless work, because the entity in charge will lack all accountability. We're already seeing that now with FAANG corporations that are effectively metastasized AIs using humans as robots. They've already taken over most governments through regulatory capture. My personal experience in this was that I performed a lifetime of hard work over about a 25 year period, participating in multiple agencies and startups, but never getting a win. So I've spent my life living in poverty. It doesn't matter how much I know or how much experience I have when my mind has done maybe 200 years of problem solving at a 10x accelerated rate working in tech - see the 40 years of work in 4 years quote by Paul Graham. I'm burned out basically at a level of no return now. I'm not sure that I will be able to adapt to delegating my labor to younger workers like yourself and AI without compromising my integrity by basing my survival on the continued exploitation of others. I'd highly recommend to any young people reading this to NOT drink the kool-aid. Nobody knows what's going to happen, and if they say they do then they are lying. Including me. Think of tech as one of the tools in your arsenal that you can use to survive the coming tech dystopia. Only work with and for people that you can see being someday. Don't lose your years like I did, building out someone else's dream. Because the odds of failure which were once 90% in the first year are perhaps 99% today. That's why nobody successful pursues meaningful work now. The successful people prostitute themselves as influencers. I'm still hopeful that we can all survive this by coming together and organizing. Loosely that will look like capturing the means of production in a distributed way at the local level in ways that can't be taken by concentrated wealth. Not socialism or communism, but a post-scarcity distribution network where automation provides most necessities for free. So permaculture, renewable energy, basically solarpunk. I think of this as a resource economy that self-evidently provides more than an endlessly devaluing and inflating money supply. But hey, what do I know. I'm the case story of what not to do. You can use my story to hopefully avoid my fate. Good luck! <3

mistrial9 2y ago

Github copilot trained on GPL-licensed code? Brad Smith you continue to outdo yourself

naikrovek 2y ago
Do you see that stated somewhere or are you just trying to stir people up?
- mistrial9 2y ago
  can we review all current lawsuits and their discovery status? pointers welcome

idan 2y ago

Hi All! GitHub Next here, happy to answer questions about Copilot Workspace and Next in general <3

JTyQZSnP3cQGa8B 2y ago
Can I install it locally without this tool having an internet connection?
- idan 2y ago
  No, it's a webapp, like github.com.
Yenrabbit 2y ago
Looks extremely cool. Have you shared / do you plan to share examples of this in practice? I would be great to read some case studies or even just browse some actual PRs rather than trying to judge from the catchy video :)
- idan 2y ago
  GitHub Stars have had access to this since last week, and a few of them have made in-use videos: - https://www.youtube.com/watch?v=FARf9emEPjI https://www.youtube.com/watch?v=FARf9emEPjI by Dev Leonardo - https://www.youtube.com/watch?v=XItuTFn4PWU https://www.youtube.com/watch?v=XItuTFn4PWU by Ahmad Awais And keep an eye on https://x.com/githubnext https://x.com/githubnext, we'll be sharing / linking to more in-action things. Any PR created with Workspace will have a link to a readonly copy of the workspace so you can see how it happened. We expect those to start circulating as people get access!

cryptoz 2y ago

This is quite similar to my idea and prototype I applied to YC with just last week. Uh-oh! Haha. I did call out GitHub as a primary competitor so I did see this coming - and I guess on second looks I'm still quite differentiated. All these new tools will overlap some but also have their own pros/cons since there are so many different ways to approach building an AI code tool! Just yesterday I put up a cheap $5 linode running a prototype but not ready for signups yet. https://codeplusequalsai.com/ https://codeplusequalsai.com/ I think I have some good ideas about how to get the the LLM to modify code - specifically working with ASTs. I wonder how GitHub prompts the LLM to get usable code modifications out of it. How often does it encounter errors?

sqeaky 2y ago
Good luck! I hope to see many competitors in this space because hegemony in this space would inhibit the software industry as a whole. Figure out your differentiators and figure what copilot sucks at. Succeeding here can be done.

esafak 2y ago

I don't use Copilot in particular but in the software space, everything is constantly changing, and the models are trained infrequently so they are never familiar with the things I ask about. Reality is outrunning the models for now but that may change.

throwup238 2y ago
There's a Black Mirror-esque movie plot idea in there! Everyone starts using bionic implants to filter out offensive parts of reality like people they don't like and stuff, until one day all the filters start falling out of sync with reality... ...and then aliens invade.
- jl6 2y ago
  They Live... kinda?

PoignardAzur 2y ago

I think Microsoft is going the wrong direction with Copilot (though it's a reasonable direction given their incentives). Right now Copilot is terrible at large changes to complex codebases; the larger and more complex, the worse. But it's great at suggesting very short snippets that guess exactly what you were in the middle of writing and write it for you. I wish Copilot focused more on the user experience at the small scale: faster and smaller completions, using the semantic info provided by the IDE, new affordances besides "press tab to complete" (I'd love a way for Copilot to tell me "your cursor should jump to this line next"), etc. This whole focus on making the AI do the entire end-to-end process for you seems like a dead end to me.

konschubert 2y ago
Totally.
vizzier 2y ago
proper insertion points in completions akin to older Visual Studio templates would probably be ideal, but you can use ctrl + right arrow in vscode to accept completions one word at a time, rather than a whole block with tab.
bloomfieldj 2y ago
I find Cursor’s Copilot++ is miles ahead of GitHub’s in terms of speed and autocomplete helpfulness. They’re also working on the “your cursor should jump to this line next" feature, but I haven’t relied on many of its suggestions yet. It’s available in their vscode fork, but doesn’t seem to be in their docs yet.
- PoignardAzur 2y ago
  Had a glance. Copilot++ looks intriguing, though the landing page is terrible (I can't even see what the demos are trying to show). I might give it a try.
lenerdenator 2y ago
> But it's great at suggesting very short snippets that guess exactly what you were in the middle of writing and write it for you. I found it reasonably good when I described what code should do through comments and let it generate based on that, evaluated the output, moved on to the next piece of logic, and repeated. Basically, chaining the snippets together. Then again I'm doing Python back-end web development, so not something terrifically hard.

rahulpandita 2y ago

https://githubnext.com/projects/copilot-workspace/ https://githubnext.com/projects/copilot-workspace/

echelon 2y ago

I'm a fan of AI-empowered coding, but I'm not a fan of a single SaaS vendor dominating the entire software lifecycle. That's an incredibly dangerous proposition. I'd love to see anyone but Microsoft push ahead, and ideally, lots of players.

Protostome 2y ago

Curious to see how it goes. For 20 years I have been working with Vim and recently added the Co-pilot plugin which is awesome. I can't however see myself using a different editor/IDE than the good'ol vim. It's not that I haven't tried, but all those IDEs tend to be bloated and consume tons of memory.

idan 2y ago
When video terminals first came out, everyone started out using line editors even though line editors make no sense when you can display arbitrary buffer. It took a while until editors changed to be "screen native". But they did change, meaningfully. When GUIs first came out, editors were just "terminal editors in a window". Took a while for the modern concept of an IDE to happen, with hovers, red squigglies, sidebars, jump to definition. All of that was possible on the first day of the GUI editor! But it took a while to figure out what everyone wanted it to be. I think we're at a similar inflection point. Yeah, everyone today (myself included) is comfortable in the environment we know. VS Code is lovely. And AI (plus realtime multiplayer) is not a display technology. But I think it's a material technology shift in the same vein as those two moments in history. I would not bet that the next thirty years is going to continue look like today's VS Code. I don't know to say what it WILL look like — we have to keep prototyping to find out.
- Protostome 2y ago
  I mostly agree with all of your points. But IMHO, if (and that's a big if nowadays, with modern web and mobile dev) text editing is the core task of the app your'e using as your IDE, then all the graphics are just redundant. All the highlighting, hovers / etc can be done via much simpler graphics that is memory and CPU efficient. As a dev, I would like to IDE developers to invest in core functionality, rather than a fancy GUI. Take the electron IDE for example. It embeds the chrome runtime which is a total waste, given that i just want to edit some text files.

idan 2y ago

Hello! GitHub Next here, happy to answer questions and unpack how we think about AI tools for developers (spoiler: it's less about codegen and more about helping with the rest of the dev cycle — building an understanding of how the system works, clearly specifying how it should change, etc)

ahnix 2y ago
[dead]
roboyoshi 2y ago
Interesting stuff! I've been trying ollama/gpt + continue.dev and copilot in VSCode for a bit now and the chat-style assistant is a huge plus. Especially in DevOps (my main work) the codegen is rather unhelpful, but the ability to let LLMs explain some sections and help in rubber-ducking is huge. I see that a good output requires a good input (prompt). How does copilot workspace determine a good input for the prompt? I see that in the github repo there is already a bunch of "Tips and Tricks" to get better results. What is your experience so far? Should we change our way of creating issues (user-stories / bug-reports, change-requests) to a format that is better understood by AI/Copilot? (half-joking, half-serious).
- idan 2y ago
  Well, that's basically the heart of Copilot Workspace! The whole UX is structured to make it easy for the human to steer. - Alter the "current" bullet points in the spec to correct the AI's understanding of the system today - Alter the "proposed" bullet points in the spec to correct the AI's understanding of what SHOULD be - Alter files/bullets in the plan in order to correct the AI's understanding of how to go from current to proposed. That said, I think there's definitely a future where we might want to explore how we nudge humans into better issue-writing habits! A well-specified issue is as important to other humans as it is to AI. And "well-specified" is not about "more", it's about clarity. Having the right level of detail, clear articulation of what success means, etc.
rjindael 2y ago
If it is more about code planning, how much different is it than simply telling ChatGPT the overall structure of your code and asking it to give you a rudimentary plan on what to do next? Would it be able to actually execute steps of the plan by generating code, then creating PRs for it? I feel like this is a great tool for our team since my understanding of the announcement is that it is more or less functionally equivalent to hiring another programmer on your team (or, if not that, at the very least having a really useful assistant.) Kudos to the GitHub team and I have immensely enjoyed using Copilot thus far to increase my productivity :-)

msoad 2y ago

This should've been done in VSCode. Most developers don't want to write code in a browser. Once VSCode extension is built they could generalize it to other editors. I'm sure they are thinking of VSCode integration and I am hopping for that to be available soon

idan 2y ago
We are indeed thinking about how to do that in a way that feels natural in VS Code. :)
pluc 2y ago
It's very different to have the code and suggest things than it is to obtain the code and suggest things, without introducing noticeable delay in processing that code
lispisok 2y ago
I wonder if AI tools is going to kill editor diversity. Every one of these tools VS Code is the first class citizen and other editors are an afterthought. Sure people can write their own Emacs package but that's only if the tool developers enable it and the experience is usually not as good as the official VS Code version. I can also see the future where not using VS Code is a signal you are less reliant on AI tools to code
- padthai 2y ago
  A thousands of small cuts, almost everything comes first to VSCode before other editors, even mainstream things are underrepresented in most other editors (Jupyter, commercial integrations, etc…). Although that was true about Atom too though?
- hackermatic 2y ago
  We used to say the same about Eclipse!
- freedomben 2y ago
  This is a great fear of mine as a (neo)vim user. I've repeatedly encountered situations already where it was assumed everyone was using VS code, and I only see that getting worse as more and more integrations get built into IDEs.

Lucasoato 2y ago

Am I the only one disappointed by Github copilot quality lately? I get much better responses by copying and pasting on ChatGPT, rather than the suggestions in the vscode plugin.

smrtinsert 2y ago
I'm pretty sure they toggle between gpt 3 and 4 depending on how much traffic they're getting in order to make sure they can meet all suggestions.
- frontalier 2y ago
  the machines were not supposed to get tired
hackermatic 2y ago
People have been anecdotally reporting and investigating similar problems since at least last year[0], and it's entirely possible that changes to improve one aspect of a model could make it much worse at other aspects without careful regression testing and gradual rollout. I think models intended to solve every problem make it very hard to guarantee they can solve any particular problem reliably over time. Imagine if a million developers simultaneously got much worse at their jobs! [0] https://arstechnica.com/information-technology/2023/07/is-chatgpt-getting-worse-over-time-study-claims-yes-but-others-arent-sure/ https://arstechnica.com/information-technology/2023/07/is-ch...

deleted 2y ago

[deleted]

sdesol 2y ago

Interesting. I don't think the AI code generation will live up to developer expectations, but I do see the value in "project management for developers". The value I see with workspace is not the code generation, but the ability to help developers organize their thoughts.

alfalfasprout 2y ago
The real problem is a large part of the enormous hype train is investors thinking they can replace talented engineers w/ AI. That's fundamentally doomed for failure. What AI can really do well is take an already competent engineer and suddenly get rid of a lot of the annoying tedium they had to deal with. Whether it's writing boilerplate, doing basic project management, organizing brain dumps/brainstorming, etc.
- nprateem 2y ago
  Yes. And they've also introduced a new kind of tedium in going round in circles while it doesn't understand you, then you give up and go back to SO.
- sdesol 2y ago
  I do think workspace by GitHub is a smart move. What is clear as day is, we don't have enough data to model how developers think and iterate to solve problems. With workspace, I see it as a way to capture structured data on how developers think. This is certainly a long game though. I think GitHub with MS money can continue to lose money on Copilot for the next 5 years to gather data. For other VC ventures, I don't think they can wait that long.
- godzillabrennus 2y ago
  These AI coding assistants already do what you are describing. At some point in the future, AI should be capable enough to do the job of a competent developer. This is not a short-term milestone, IMHO.
- idan 2y ago
  Bingo.

happypumpkin 2y ago

> At last count, Copilot had over 1.8 million paying individual and 50,000 enterprise customers. > Copilot loses an average of $20 a month per user, according to a Wall Street Journal report, with some customers costing GitHub as much as $80 a month. Presumably at some point they need to actually make money on this? That is a $432 million / yr loss just on individual users.

Yenrabbit 2y ago
Despite being widely repeated they've publically denied this claim a few times (for e.g. https://x.com/natfriedman/status/1712140497127342404 https://x.com/natfriedman/status/1712140497127342404, https://x.com/simonw/status/1712165081327480892 https://x.com/simonw/status/1712165081327480892)

danenania 2y ago

For anyone who might be interested in an open source, terminal-based approach to using AI on larger tasks and real-world projects, I'm building Plandex: https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex I've tried to create a very tight feedback loop between the developer and the LLM. I wanted something that feels similar to git. Apart from the planning and code generation itself, Plandex is also strongly focused on version control. Every interaction with the model is versioned so that it's easy to try out different strategies and backtrack/revise when needed. I just released a new update (literally a few minutes ago) that includes some major improvements to reliability, as well as support for all kinds of models (previously it had been OpenAI-only). It's been fun trying out Claude Opus and Mixtral especially. I'd love to hear people's thoughts!

Bnjoroge 2y ago
just played with it. pretty solid! questions for me, how do you validate the new changes will work without any issues after applying them? the biggest issue I have with most code generators is the feedback loop from suggested code -> testing it out -> failing, doing it again. would be great if this was more seamless. additionally, would be helpful to control the code generation process in real-time.
- rahulpandita 2y ago
  GitHubNext Dev here: We have a terminal feature that allows you to connect to sandbox in the cloud for you to validate these changes before pushing them out to your repo
- danenania 2y ago
  Thanks for trying it! So far, the the testing and validation stage has been left to the developer. While this could change in the future, my experience has been the models aren't quite good enough yet to make an auto-test/auto-fix loop like you've described the default behavior. You end up with a lot of tire-spinning where you're burning a lot of tokens to fix issues that a human can resolve trivially. I think it's better for now to use LLMs to generate the bulk of a task, then have the developer clean up and integrate rather than trying to get the LLM to do 100%. That said, you can accomplish a workflow like this with Plandex already by piping output into context. It would look something like: plandex new plandex load relevant_context.ts some_more_context.ts plandex tell 'some kind of complex task' # ...Plandex does its thing, but doesn't get it 100% right npm test | plandex load plandex tell 'please fix the problems causing the failed tests' As the models improve, I'm definitely interested in baking this in to make it more automated.

7thpower 2y ago

I use Copilot in VS code for work and it is great for auto complete, but the chat experience is very poor compared to tools like Cursor. The inline edits are even worse. Long story short, is I don’t have a lot of confidence in the product right now. That being said, I am very optimistic on the product long term and I generally like the vision.

l5870uoo9y 2y ago
I cancelled my subscription mainly because it made me neglect overall structure and instead autocomplete quick inline and repetitive solutions. Haven’t missed it.

rgs224 2y ago

was able to get around waitlist: copilot-workspace.githubnext.com /<owner>/<repo>?task=<description>

throwup238 2y ago
Great job! Worked for me to create the plan and now slowly generating the file changes for the plan.
jckwind 2y ago
i think its patched :/
- neuralnerd 2y ago
  Yup. Got patched :/

singluere 2y ago

While I've not used this product, I've created somewhat similar setup using open source LLMs that runs locally. After having used it for about three months, I can say that debugging LLM prompts was far more annoying than debugging code. Ultimately, I ended up abandoning my setup and going in favor of writing code the good old fashioned way. YMMV

BoorishBears 2y ago
Creating this using Open Source LLMs would be like saying you tried A5 Wagyu by going to Burger King, respectfully. I think benchmarks are severely overselling what open source models are capable of compared to closed source models.
- Zambyte 2y ago
  I really don't think they're being over sold that much. I'm running llama 3 8b on my machine, and it feels a lot like running claude 3 haiku with a much lower context window. Quality wise it is surprisingly nice.
  - BoorishBears 2y ago
    Llama 3 just came out so they couldn't have used it, and Claude Haiku is the smallest cheapest closed source model out there from what I've seen. Github is likely using a GPT-4 class model which is two (massive) steps up in capabilities in Anthropic's offerings alone
    - Zambyte 2y ago
      Yeah I just mentioned Llama to point out that the open weight models have been really catching up. Microsoft is almost certainly using GPT-4 given their relationship with ClosedAI, but I would definitely not put GPT-4 (nor Turbo) "two massive steps up" from Claude 3 Opus. I have access to both through Kagi, and I have found myself favoring the responses of Claude to the point where I almost never use GPT(TM) anymore.

nextworddev 2y ago

RIP Devin

sqs 2y ago

Cool idea! Agree on the end vision. The alpha is in figuring out how to get there and making it work really well along the way. Also, why isn’t anybody connecting the Copilot Workspace announcement to Devin? Biggest company in the world announces a release of a competitor to the most widely seen dev product announcement of the year? Only saw one incidental mention of the connection.

kylecarbs 2y ago

I was able to try it (without being approved) here: https://copilot-workspace.githubnext.com/ https://copilot-workspace.githubnext.com/

asadm 2y ago
hmm doesnt work anymore

mlhpdx 2y ago

I’m letting it sink in that GitHub thinks it would be a win if 10%+ of the world population is involved in creating software. I have no hope of hiring an electrician in that world.

idan 2y ago
You have succeeded at hiring an electrician in THIS world? What's their number? Do they actually show up when they say they will?!?!?!!1!one

ilaksh 2y ago

I think these types of tools will become really effective when they stop trying to target every single programming language and platform ever created and focus on one specific version of an integrated platform. Most businesses don't actually need software to be in whatever flavor of the month the dev team happens to be interested in. They just want to be able customize their software. By restricting the scope somewhat, you make it much more feasible to make sure the model has the training and/or knowledge ready for retrieval to fulfill tasks using a specific stack. So I see this type of thing as quickly evolving into a tool for non-developers. And within a few years, these tools will cut into software engineering jobs. It will become part of the evolution of no-code and low-code.

jstummbillig 2y ago
As long as it does the language that my business currently uses, I am all for that! Jokes aside, I am not sure I buy the premise. I have read somewhere (citation required) that LLMs get better at other things after having learned how to code. And maybe they also get better at coding in one language after having learned a bunch.

colemannerd 2y ago

I wish there were fixes to devcontainers before doing adding copilot. I really want declarative, repeatable builds that are easily used in both Codespaces AND Actions. I know all of the functionality is theoretically there in devcontainers.json, it is so manual to configure and confusing, that anytime I've done it, I use it for 2 weeks and then just go back to developing on local because I don't have time to keep that up. ESPECIALLY if you're deploying to AWS cloud and also, want to use alternative package managers like poetry, uv, yarn, jsr, etc.

nojvek 2y ago

I use Github co-pilot but 55% productive is a bullshit number. Perhaps 1% may be. Most of co-pilot suggestions are either simple pattern matches or subtle hallucinations where I have to catch and fix silly bugs. Github Chat is not very useful at understanding what the code is doing. Tried it once or twice and gave up. The hype will help with Microsoft Stock though. Seems like bean counter management is taking over.

Bjorkbat 2y ago
I was actually about to make a comment on that. They got that number from some dumb "experiment" they did where told two groups of developers to implement an HTTP server in Javascript. The group with Copilot got the job done 55% faster (https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/?ocid=AIDcmmc3fhtaow_SEM__k_Cj0KCQjwir2xBhC_ARIsAMTXk85qF--5Z_RO3fCcRnFE6itQ3cKFQvIGLnsyaF-Gwd4FdZFoisM7GtUaAou4EALw_wcB_k_#finding-2-but-speed-is-important-too https://github.blog/2022-09-07-research-quantifying-github-c...) So yeah, they did an experiment with <100 developers asking them to create implement something that only took the control group 3 hours to finish from scratch, and from this we got the "55% more productive" statistic.
- chasd00 2y ago
  That's really disappointing and a little bit scary. That seems like a task tailor fit for an LLM, it's been done a million times before, follows an established protocol, and the implementations are all basically the same. Granted a lot of code is like that but a more realistic task may have been something like "create an online ordering system for a cake company that sells different kinds of cakes, has a sale every so often, and uses coupons printed in the local paper". That task is more ambiguous and better reflects what software devs are tasked with day to day IMO.
- deleted 2y ago
  [deleted]

daviding 2y ago

One side effect that I don't think I like of these tools is the inevitable push for developers to just use the most popular languages. Javascript and python are the main LLM trained languages, so it just self reenforces that it is best to use those for everything. There is something a bit sad about that. I guess the dream is that they are just intermediatory data anyway, like a chatty bytecode layer.

elwell 2y ago

> and is expressly designed to deliver–not replace–developer creativity I'm not fooled by such corp speak.

strix_varius 2y ago
If they could, they would!

Bjorkbat 2y ago

You can see my earlier comment somewhere here, but I feel obliged to remind everyone that the claim that Github Copilot made developers "55% more productive" came from a study where they asked 100 developers to implement an HTTP server in Javascript, split the group up roughly 50/50, and gave one group Github Copilot. The Copilot group did it in an hour and 11 minutes, whereas the control group got it done 2 hours and 41 minutes. https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/?ocid=AIDcmmc3fhtaow_SEM__k_Cj0KCQjwir2xBhC_ARIsAMTXk85qF--5Z_RO3fCcRnFE6itQ3cKFQvIGLnsyaF-Gwd4FdZFoisM7GtUaAou4EALw_wcB_k_#finding-2-but-speed-is-important-too https://github.blog/2022-09-07-research-quantifying-github-c... That's where the 55% number is coming from. It's coming from this experiment, and only this experiment. So yeah, if you're wondering why you aren't somehow 50% more productive when using Github Copilot, it's probably because you're not implementing a simple task that's been done to death in countless tutorials.

mucle6 2y ago
Do you not have a good experience with github copilot? I recently started using it and it will write functions that are 90% right based on comments, and I just modify what I want. I've felt at least a 2x speedup, maybe even 4x. That said I'm working on a new project where I'm writing a lot of code. For making small changes I could see how its much less valuable.
- chasd00 2y ago
  "I just modify what I want." i think that's key, you use the llm to get something started and then change/fix/enhance to get what you want. That works for me too but the folks that want to prompt an llm from nothing to a finished application are in for a rough ride.

mucle6 2y ago

I wonder if we can use "copilot-ability" as a proxy for code base complexity. In other words, if copilot can't help you write code, then could that mean your code base is too complicated for a beginner to understand and modify?

chasd00 2y ago
I hope not but i bet you're right. I'm sure there's lots of people working on llms to generate a "cognitive complexity" score like some of the static code analyzers we have. It will be so ripe with false positives as to be effectively worthless however the score is an easy thing to build a metric, report, and policy on... just like with the analyzers today.
layer8 2y ago
That assumes that it would be reasonable to expect a beginner to understand the code base of any nontrivial application or system. I don’t think that’s a reasonable expectation, if only because there are countless requirements, assumptions, experiences made, priorities and design decision that have to be known to understand the code. It’s also a question of economy. You can always try to make a code base easier to understand and to document anything and everything, but it comes at a cost. It doesn’t help if a code base is in principle understandable by a beginner if they have to spend months reading documentation first, and someone had to spend months if not years to write it.

iknownthing 2y ago

I noticed there were a lot of YC companies trying to do similar things. I'm not sure how they think they have a shot against Copilot.

jaredcwhite 2y ago

So GitHub has outright declared war on competence in programming. Great, just great. The code examples is this blog post are horrible…I'd be embarrassed to share any such demonstration of generated code offering these solutions. Reject that PR in production, for the love of god! Funny that I already started migrating my personal projects off of GitHub and onto Codeberg over the weekend. I'll be rapidly accelerating that process after today. Copilot has gone from vaguely irritating (and of course I refuse to use it on ethical grounds) to an overall direction for the company I *intensely* disagree with.

sqs 2y ago

You can try out GitHub Copilot Workspace now in read-only mode: https://copilot-workspace.githubnext.com/AnandChowdhary/anandchowdhary.com?shareId=fe652b2c-e6a3-4ab9-841f-a071f5789c1b https://copilot-workspace.githubnext.com/AnandChowdhary/anan...

Cilvic 2y ago

I'm not sure why they insist on code spaces instead of running this inside vs code? Setups/working looks pretty similar to aider [1] that I've been using and liking, Aider is smaller steps than plandex, but plandex went into some kind of loops a couple of times so I stoped using it for now. [1] https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider

lostintangent 2y ago
We definitely intend to explore a VS Code extension in the not too distant future. And we decided to build a web client + integrated cloud terminal, simply because that allowed us to create an experience that is one-click away from an issue, and could be accessed from anywhere and any device (i.e. we’ve deeply optimized CW for mobile, and I do quite a bit of code thinking in that modality). In the meantime, when you open a Codespace from Copilot Workspace, you could open that Codespace in VS Code desktop. And use that as a companion editor to the web client (since we bi-directionally sync file changes between them). But totally agreed that a more integrated VS Code experience will be compelling!
- Cilvic 2y ago
  > simply because that allowed us to create an experience that is one-click away from an issue, That's indeed pretty cool and as you said it's not an either-or. Thanks for providing more background.
danenania 2y ago
> plandex went into some kind of loops a couple of times so I stoped using it for now. Hey, Plandex creator here. I just pushed a release today that includes fixes for exactly this kind of problem - https://github.com/plandex-ai/plandex/releases/tag/cli%2Fv0.9.0 https://github.com/plandex-ai/plandex/releases/tag/cli%2Fv0.... -- Plandex now has a much better 'working memory' that helps it not to go into loops, repeat steps it's already done, or give up too early. I'd love to hear whether it's working better for you now.
- 2y ago

lostintangent 2y ago

If folks want to see what Copilot Workspace looks like, here’s a sample session where I addressed a feature request in an OSS VS Code extension I maintain: https://copilot-workspace.githubnext.com/lostintangent/gitdoc/issues/52?shareId=8a0244c3-95d7-4bfc-8b85-f1a9d7c77189 https://copilot-workspace.githubnext.com/lostintangent/gitdo.... You can see the originating issue and the resulting PR from there. And note that while the initial spec/plan/code was mostly good, I iterated on a couple parts of the plan, and then made a minor tweak to the code manually (everything in CW is editable). Which is a key part of our goal with CW: to help bootstrap you with a task (or think out loud with AI), and then provide the iteration primitives to explore further.

steve_adams_86 2y ago

Watching the examples and other people's demos, I get the sense that this product completely ignores what makes good software. It has no "big picture" plan or contextual awareness required to make those nuanced yet critical decisions in the far ends of applications. Those cases where you badly need domain expertise, understanding of how the user interfaces with the product, awareness of existing technical debt, and so on. The scope it can assist in is too limited. It is exactly the kind of thing I'd use to generate self-destructing spaghetti. The thing is, people with less experience will eat this up because it can do some tasks with ease which might not be familiar to them yet. Over time, the code they produce will be a mine field of incongruent patterns and solutions. It appears to support the hypothesis that these tools may actually create more work, but I worry that the work will be much like that created by outsourced developers shipping duct-taped nonsense back to North America throughout the last 20 years or so.

mlsu 2y ago
And, on top of that, the first few fixes will actually work! They will look good to PMs and to the business! Look at the velocity! The velocity is good enough to scale back the size of the team, it's good enough to mandate a pace that is only possible with AI doing automated submission and review of PRs, it's good enough to not have any kind of formal design work. Just ship. After several months of this, any additional bug fix by the AI is just adding to the existing morass of tech debt created by the AI. Slowly but surely, the AI will stop being able to submit a PR that doesn't result in a regression. The people who actually care about software quality aren't on new projects; those are run by juniors and copilot. Nope, they're going to be stuck on maintenance of this garbage. And meanwhile, that new team working on new feature X is moving so quickly... better give them a raise! The people who suffer here are, of course, the non-marginal user, who will have to contend with every new feature breaking every other existing feature that they like in the software.
idan 2y ago
That hasn't been our experience using it in-house! It's not perfect, but not every software engineering task is some galaxy-brain architectural shit. Sometimes, you have to lay down some bricks. And sometimes, to lay down bricks, you need to touch more than one tiny patch of code. Having something round up the likely areas of the codebase that needs touching feels magical. It doesn't always succeed! But it feels pretty magical to get that boost when you're new to some part of a codebase (which, real talk, code I wrote > 1 month ago, I must page back into memory). Making it easy for me to progressively add context for the model is an accurate analogue for how I think as a developer when tackling a task. I have to build a mental model of how things work. And then a plan for how I'm going to change it. Maybe for the kinds of tasks you usually tackle, it won't have value. But the amount of context it's attempting to bring to bear on whatever task you give it is categorically more — and better — than any other tool I've seen. I have seen (and been the author of) spaghetti. Could I make CW generate spaghetti? Surely. That's why it's a tool for developers, not a substitute for developers.

vegancap 2y ago

Jesus christ, I don't know whether to be excited or terrified by some of these new AI releases...

burntcaramel 2y ago

Pretty sure I wouldn’t want to work on a team using this. “Who wrote this code, it has a serious flaw?” “The customer needed that feature quickly so we used Copilot. No sorry, I don’t know how it works, I trust that it does. I glanced over the code and it looked about right.” “Did you read the checklist that it generated? The flaw stems from an assumption it made.” “Oh no, I didn’t see that. To be honest I don’t read the checklists it makes.”

benzguo 2y ago

it's so over for Devin

TriNetra 2y ago

can we use gpt to convert say, a Xamarin Android app code into React-Native? has anyone tried similar conversion successfully?

rany_ 2y ago

Is this sort of like a response to Devin? I'm having a hard time understanding what this is.

ianbutler 2y ago

A little late to the party here, but we (https://www.bismuthos.com https://www.bismuthos.com) offer something already to scratch this itch. We provide a workspace to build Python backends. Chat on the left, code and visual editors on the right. However, we also handle deployments, data storage (we have a blob store), serving (we built a home grown function runtime) and logging. The experience is tightly integrated with the copilot and the idea is to get ideas off the ground as quickly as possible. We're early days and you can try it for free right now.

joandaniel497 2y ago

[dead]