9 ms·
Using Google’s code history to write more code
- hinkley 3y agoI probably spend 5-10% of my time on average, and more in bad situations, going through code trying to purge past mistakes. Part of why this takes so much time is that there’s always some special idiot who is writing new code in the bad way and won’t stop. But they are typically lazy and copy and paste that code from existing places. If you can remove the last copy from the code, they will finally stop. And now that special idiot is a piece of software that never sleeps, never eats, and never forgets? What could go wrong?
- gjadi 3y agoWisdom of the crowd is the hypothesis made here I guess.
- hinkley 3y ago"Think about how dumb the average person is, and then realize that half of them are dumber than that" -George Carlin The wisdom of crowds, if it applies to teams, is in communication. I talk you out of cutting corners, and sometimes you talk me out of it. There's no communication in a training corpus. You just get the average of all of the code that wasn't so horrible that people stopped what they were doing to fix it. Which is to say, half of the code is barely tolerable.
- lopkeny12ko 3y agoThese are exactly my thoughts. This just sounds like a bot to regurgitate the same mistakes over and over again. I'm struggling to understand who at Google actually thought this was a good idea? Or is this just a novelty research project with no practical use?
- passwordoops 3y ago"Did you say this is using an LLM? SHIP IT!!" is my guess.
- Arainach 3y agoGoogler, opinions merely my own: Google's monorepo and tooling for changes across the entire repo make it possible and common to fix entire classes of mistakes at once. Bad patterns lingering in copy/paste is not an issue in practice due to a combination of such changes and linter presubmits that identify and flag any recurrence in the future.
- redleggedfrog 3y agoWhat I want is to write less code. I'd like automated analysis help with finding better more concise ways to render the logic. More code != better.
- JustSomeNobody 3y ago>More code != better. >more concise ways to render the logic This isn't better either. Edit before the shitstorm: This isn't necessarily better either.
- hinkley 3y agoWhen I'm looking at code, I don't want to read War and Peace. But I also don't want to read Haiku either. All the burden is on the reader, and there are many readers. Usually what I want is a short story. Long enough to be accurate, short enough to lose distracting and unnecessary information.
- hiddencost 3y agoI suppose I'm excited for the opposite reason: It's much easier to get one model to change its behavior, than 1000 software engineers.
- qup 3y agoThe whinging we hear is firmly anchored to the timeline on which humans improve, and the extent to which they improve.
- avgcorrection 3y agoWhat I’ll decline: Using LLMs to write more code What I’ll buy: Using LLMs to write less code (maybe even delete a lot)
- jvalencia 3y agothis
- codethief 3y ago…or to do annoying refactorings!
- TremendousJudge 3y agoI would not trust an non-deterministic system to do a refactor on a production system -- I would have to review the whole thing. Isn't it quicker to just do it myself with my regular deterministic tools?
- onjectic 3y agoI’m going to start calling all of my co-workers non-deterministic systems.
- TremendousJudge 3y agoThe difference is that your co-workers sign on their own work -- if it breaks, it's their responsibility. If you commit some code, and it breaks, it's on you, no matter which fancy tool you used to write it.
- 8note 3y agoGenerally if you write some code and it breaks, your whole team has some fault in its release mechanism, and that's group owned
- deleted 3y ago[deleted]
- macjohnmcc 3y agoWhen I find one bug I assume it is likely repeated all over the code and I search it out. I have been burned enough times to not consider it a must.
- hinkley 3y agoThere's always a subtle difference in one of the 3 implementations. Less subtle when it's 5+. That's where the time adds up.
- macjohnmcc 3y agoI also found during these kinds of sweeps large copy and pastes of code. It really lowered my opinion of those working on the same project.
- hinkley 3y agoIt's taught me to worry about people who happily volunteer for difficult tasks. I used to think they were troopers, now I see more and more of them as having a high pain tolerance, and I don't mean that as a compliment. Pain in information and they throw away a lot. It's a form of job security, and not a particularly benign kind. Our job is to replace painful processes with code. If you can't do that you aren't fulfilling the basic premise of your profession.
- zaphirplane 3y agoNot talking to you, just on the topic of DRY fundamentalism I find people creating these DRY abstractions with a lot of knobs to be strange. End up with dryWithRain, drySpicy, dryStrict, dryTurbo then the abstraction is a maze of confusion
- d3nj4l 3y agoPeople love to rag on OpenAI for not being open (and they're right to do so!) but when I read posts like this I'm still thankful that OpenAI at least has APIs. What Google AI product can you actually use for your day to day programming? I don't care if you have a fancy blog post and your system is 100x better than Copilot, at least I can use copilot!
- Crash0v3rid3 3y ago> What Google AI product can you actually use for your day to day programming? Well there is bard that is available in many locations. ...and if you want APIs there is a plethora of them on GCP.
- kccqzy 3y agoEspecially that Bard was recently updated explicitly to support helping with coding tasks. https://blog.google/technology/ai/code-with-bard/amp/ https://blog.google/technology/ai/code-with-bard/amp/
- Jleagle 3y agoHasn't the latest batch of AI's (gpt etc) all stemmed from work Google did years ago (transformers), and released white papers for, for free? No company has to offer APIs for you to play with, most don't in fact.
- lopkeny12ko 3y ago> DIDACT ( Dynamic Integrated Developer ACTivity) This is quite a stretch of an acronym, and makes it feel like they came up with the acronym first and filled out the meaning later...
- jvolkman 3y agohttps://en.wikipedia.org/wiki/Backronym https://en.wikipedia.org/wiki/Backronym
- johnzim 3y agoA classic backronym.
- thraxscipio 3y agoThey're just waiting to be able to call it autodidact in 1-2 year's time
- sangnoir 3y agoAutoDIDACT will be an improvement: it will independently author code from a ticket and update the diff in response to code review feedback.
- xeromal 3y agoTime was your ally, human, but now it has abandoned you. The Forerunners... have returned. This tomb… is now yours.
- sorokod 3y agoA decent use case of some sort of statistical language model.
- sdwr 3y agoAbsolutely incredible! This feels like 2/3rds of the way towards being the dev equivalent of a person from WALL-E ( or Homer Simpson with his drinking bird). Hopefully in 6 months there will be a generally-available tool for this that isn't too expensive. The dream is written tasks -> code, with the ability to interrupt and tweak. Copilot and ChatGPT are "yeah, but..." for coding - they do some of the heavy lifting but don't get all the way there. IMO, people who are poo-pooing this are threatened, and have too much of their ego wrapped up in being good at code. Code is a tool to serve a purpose, most of the meaning is at the social or product level. Let the computer win the game!
- alpb 3y agoThis is signature Google blog post. Great research, not a product, so no you cannot use it. Do they have no intention to make a product out of it? This is not something Microsoft would've done. They directly announced Copilot as a product from the get go instead of parading their tech.
- kccqzy 3y agoIt's a marketing device for people to apply to Google and become employees. Remember Google puts its employees way above its customers, and Kurian is only beginning to change that culture.
- vlovich123 3y agoHasn’t he been there for about 5 years now?
- gowld 3y agoKurian only runs Cloud.
- VirusNewbie 3y agoCloud is like half the company now. It isn’t just GCP.
- quaintdev 3y agoShare holders > Employees > Users Users and employees can swap places on case to case basis
- jeffbee 3y agoI think this really pops the bubble of the fantasy of the ultra-productive scrappy individual running circles around stodgy old corporations, because the little hacker is exploiting Codex or Copilot. It seems in reality the outcome is the opposite: AI is amplifying the advantages of large incumbent software companies. They have the data and the resources to train and deploy this stuff. They've been using it in practice for years and they aren't planning to share it with you.
- sebzim4500 3y agoYeah but at least one of those incumbents seem allergic to releasing actual products, so I wouldn't count the scrappy individuals out just yet.
- jeffbee 3y agoGoogle has more—much, much more—to gain from being a highly productive software developer than they stand to gain from selling software development tools to outside users.
- dwaltrip 3y agoMy intuition is that the new crop of AI tools is still higher leverage for individuals than for large organizations. And given how software development historically has scaled sub-linearly with org size, I expect we will see interesting developments as these tools continue to improve. Advancements should magnify the leverage differential. However, if tools like those discussed in the article can change the equation of how software development scales with org size, then maybe that changes things. I do agree that large orgs will gain a lot here as well. But I’m not sold that they will gain more than skilled individuals, proportionally.
- oceanplexian 3y ago> AI is amplifying the advantages of large incumbent software companies. In large software companies (I've worked for a few) the bureaucracy always seems to scale exponentially more than the 'productivity'. So you end up waiting on a build system that takes 5 hours, you have to use some old library because it's pinned in a monorepo, reviews from other engineers that take days, a Staff Engineer wants to give his opinion so the project gets delayed another week, another manager feels threatened by your idea so he shuts you down, etc. AI won't solve any of this, it will just allow increasingly more convoluted bureaucracy to be created. Compare that to a scrappy developer or startup, they can run literally run circles around big tech. The only thing is they don't have is as much money or resources, but they have a ton of advantages getting things to market, where AI can be used to increase productivity without all the red tape.
- j16sdiz 3y ago- Is this available to public? No. - Available to Enterprise customers? No. - Can other researcher replicate their result? No. - Is this a recruitment blog? No. I have no idea what is the target audience for this post.
- m3kw9 3y agoYeah release something we can test, stop talking so much man
- jeffrallen 3y agoWhat we really need is an AI to delete code.
- treprinum 3y agoThis will allow the following strategy: 1. Hire a talented developer 2. Store all their development interactions over the years 3. Train a model on these interactions 4. Fire the developer 5. Do this for thousands of top-end developers 6. Start assembling virtual AI teams trained on these developers, see which teams perform the best 7. ... something insane is going to happen
- UncleEntity 3y agoStep 1 to 3 complete. Has google started layoffs yet?
- potbelly83 3y agoHonestly Google's a joke. They spend so much time on pie in the sky stuff, but simple stuff (which makes my life miserable), like buttons getting hidden in Google Docs get completely ignored. Guess that's what happens when you only hire smart people, boring details get overlooked and deemed unimportant.
- purplecats 3y agoit isn't a scalable solution to expect every product in which you have no direct control, to do everything precisely tailored to your desires. i recommend building extensions to bridge that gap from what the product affords vs what you seek.
- potbelly83 3y agoI'm not talking about UI things I'd like, I'm talking about existing UI that is broken/buggy. i.e. This kind of thing https://support.google.com/docs/thread/89723040/bug-docs-drawing-save-and-close-button-disappears-if-i-click-and-drag-on-it?hl=en https://support.google.com/docs/thread/89723040/bug-docs-dra...
- still_grokking 3y agoMore code? Code is a liability, not an asset.
- anticensor 3y agoCode is both a liability and an asset.
- still_grokking 3y agoNo, it isn't. The benefit that some code (maybe) provides is an asset. The code as such is strictly not. Nobody would even look at your code if you don't compensate them with some benefits. Exactly that's why nobody touches random "free code" even with a ten foot pole if there aren't very strong reasons to do so. If you could have the benefits without the code any sane person would prefer that. Code is just cost. You don't want code as such as already just owning it has steadily increasing costs. You just want the benefits out of it. If there's a solutions to a problem where writing new code can be avoided this is almost in any case the cheapest and most preferable solution. Only junior devs think code has value. But the truth is you get more value if you're senior skilled and smart enough to avoid wasting your time on writing (more) code. Code is a means to an end, not a means in itself! Given that, the best approximate measure of technical dept (and with it security issues, bugs, and in the end of the day cost) is: Lines of code! https://www.jamesshore.com/v2/blog/2008/an-approximate-measure-of-technical-debt https://www.jamesshore.com/v2/blog/2008/an-approximate-measu... Now, a smart and really helpful AI for programmers would try very hard to achieve the stated goals but delete on it's way as much code as possible. Because that's the only way to lower cost in the long run and keep things maintainable. Any AI that does the opposite, and just creates mostly completely mindless piles of more and more code, is like an oven heated by dollar bills… I really don't get how something such obvious isn't seen by so many people. Especially people that have to handle all the pains created by more an more code on a daily basis.
- anticensor 3y agoThe code is knowledge, which is a non-fungible, trivially reproducible asset (apart from the bottleneck caused by intellectual property).
- buildbot 3y agoThis research is neat, but would be 100x more impactful if they also said "and you can try it out today in Google Colab!"
- gipp 3y agoEh, the whole thing seems very coupled to the Google universe, I doubt it would generalize well outside of it. People really underestimate how divergent the tech is in there. It's really a whole parallel tech ecosystem that's been evolving independently for 20 years. And I'm not aware of anything outside that ecosystem that has anything close to the level of instrumentation and integration of the entire dev process.
- snewman 3y agoSo much negativity on this thread. I think it's at least worth mentioning that the paper does present many genuinely interesting ideas; for instance: 1. The change history of a repository contains more information than just the final state of the code. The change log constitutes useful training data which can allow a model to learn things it might not get from just being trained on completed code. 2. Software engineering consists of a lot more than just writing code. They've been able to train a model to do other tasks, such as reviewing code and responding to review comments. 3. Operations such as renaming a variable can be modeled as a single high-level action, rather than as a set of raw code edits. A model trained to generate such higher-level actions can potentially achieve better results. "For example, a rename might touch a file in dozens of places, but a model can predict a single rename action." 4. A developer's recent activity can be used to help predict next steps. They give the example that if you add a parameter to a function and then move the cursor into the function documentation, the model can predict that your next step is to document the new parameter. "Without history, the model wouldn’t know whether the missing function parameter is intentional (because the developer is in the process of a longer edit to remove it) or accidental (in which case the model should re-add it to fix the problem)." A longer example: > ...we started with a blank file and asked the model to successively predict what edits would come next until it had written a full code file. The astonishing part is that the model developed code in a step-by-step way that would seem natural to a developer: It started by first creating a fully working skeleton with imports, flags, and a basic main function. It then incrementally added new functionality, like reading from a file and writing results, and added functionality to filter out some lines based on a user-provided regular expression, which required changes across the file, like adding new flags. This ability to emulate a natural sequence of authorship steps to create a new code file seems similar to "reason step by step", but specialized for coding.
- fragmede 3y agoThe dream I have, is for an LLM to commit to source control, every time I save a file and run whatever test I'm doing. Me: Save -> run LLM in background: commit "added function foo that does X" Even for when my code doesn't work, squashing commits is way easier than unsquashing commits.
- gtani 3y agosimilar at Meta/facebook https://arxiv.org/abs/2305.12050 https://arxiv.org/abs/2305.12050