13 ms·

Hey, GitHub – Waiting list signup

316 points by rcshubhadeep 4y ago

P5fRxh5kUvp2th 4y ago
Programming Perl with speech recognition (an oldie but goodie) https://www.youtube.com/watch?v=vPXEDW30qBA https://www.youtube.com/watch?v=vPXEDW30qBA
Tade0 4y ago
I hope to see the click consonant "‖" adopted as "||" one day.
ggerganov 4y ago
Very interesting - I was sort of expecting it to happen soon. I have been playing with using Whisper + Github Copilot in Vim [0]. The Whisper text transcription runs offline with a custom C/C++ inference and I use Copilot through the copilot.nvim plugin for Neovim. The results were very satisfying. Edit: And just in case there is interest in this, the code is available [1]. It would be very awesome if someone helps to wrap this functionality in a proper Vim plugin. [0] https://youtu.be/3flN9kTcZJY https://youtu.be/3flN9kTcZJY [1] https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.nvim https://github.com/ggerganov/whisper.cpp/tree/master/example...
chocolatkey 4y ago
If this works well, I would pay a seriously high amount of money. My daily coding time is currently limited by the pain in my hand/fingers that eventually becomes too uncomfortable, and I have to wait for a "cooldown" period of days to "reset" my hands back to normal. I can't even code on a normal keyboard or trackpad for a long time anymore. The problem with current voice programming systems is they're just too slow so I end up getting impatient and using my fingers anyway
- eajakobsen 4y ago
  I imagine you have done extensive research on your own, but in any case I found this article by Josh Comeau on coding with voice commands and eye tracking very interesting: https://www.joshwcomeau.com/blog/hands-free-coding/ https://www.joshwcomeau.com/blog/hands-free-coding/
  - langsoul-com 4y ago
    I wonder if the quest pro could do the eye tracking as well? It has pretty extensive eye tracking cameras, not sure how precise they're though. Could it also do a virtual keyboard, but a custom layout to not trigger arm, elbow and hand pains?
    - atonalfreerider 4y ago
      We built a VR prototype a year ago for voice to code using Codex in VR: https://www.youtube.com/watch?v=icHLoxOFerk https://www.youtube.com/watch?v=icHLoxOFerk
- langsoul-com 4y ago
  I'm assuming you have an ergonomic keyboard? If so which one?
  - tgv 4y ago
    And a movable monitor, decent chair and desk at the appropriate height...
    - Comevius 4y ago
      An ergonomic keyboard layout like Workman doesn't hurt either. QWERTY was made with typewriter jamming in mind, sacrificing ergonomics completely, especially for English.
  - chocolatkey 4y ago
    Ergodox EZ
- Hesinde 4y ago
  Have you tried an organ MIDI pedalboard and a script to translate MIDI to keystrokes? You could also put a micro controller between the pedalboard and your computer so that it looks like a normal keyboard to the computer. I do not know whether that would be pratical, but some sort of feet keyboard is in my idea space for what if.
- ron22 4y ago
  If you want to code with your voice, also checkout https://github.com/cursorless-dev/cursorless https://github.com/cursorless-dev/cursorless
- rahulpandita 4y ago
  GitHubNext here! We appreciate your support. Please consider helping us by signing up for the experiment on the website and providing feedback. :)
singularity2001 4y ago
Drop the "Hey Github" nonsense (hopefully it's only for illustration purposes anyways) and … this will be a generational paradigm change in how to write code… if it works. The hard part will be editing code with your voice too. Like "no, I meant …" etc. VERY PROMISING, in any case you can just manually fill the gaps with the keyboard!
- wiseowise 4y ago
  > this will be a generational paradigm change in how to write code… if it works Why?
  - mkmk3 4y ago
    I could see it maybe being important once github codepilot is embedded in it? You tell it roughly what you want and then adapt by hand. But it is kinda funny seeing parent make such claims so early
    - singularity2001 4y ago
      > once github codepilot is embedded That's exactly the point of the demo, no?
      - mkmk3 4y ago
        Yup my bad for jumping to conclusions. This certainly seems worthwhile exploring
  - wokwokwok 4y ago
    How can it not be a paradigm change when it changes the way people write code from “write by hand” to “generated by ai with natural language”? The problem with speech to code has always been that precise syntax is hard, but AI codegen solves that. So, no, it might not take off, but I feel like if it does, then it means ai-codegen will become the dominant way code is crafted. That would be paradigm shifting. It’s inconceivable that it wouldn’t be.
mezobeli 4y ago
Copilot -> Pilot
ausudhz 4y ago
Next is thoughts to code. Just read my mind I'm gonna seat there and think
- nnurmanov 4y ago
  Song to code, we shall be singing our next systems:)
  - singularity2001 4y ago
    dance to code. transform the esthetics of your movements into … whatever your boss requires.
hcnews 4y ago
To note, there's a class action lawsuit against GitHub Co-Pilot since it learns from a bunch of open source code with very specific licenses. It's very interesting from establishing copyright in an AI training perspective. Hopefully it goes the distance and some nuanced arguments come out in the court case. https://www.theverge.com/2022/11/8/23446821/microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data https://www.theverge.com/2022/11/8/23446821/microsoft-openai...
kgrax01 4y ago
People can’t seriously believe this is going to be useful at all? I can see this helping as an accessibility tool, but beyond that I don’t think it will be useful. This kind of assumes you know everything about what you’re doing, most of the time you don’t.
Quequau 4y ago
I remember a talk given some years ago by a man who was using voice to text for creating source code. The key point I remember from his talk & demonstration is that it was not casual ordinary speech but instead a very weird mashup of sounds intended to represent the various symbols which we use in source code.
- simme_ 4y ago
  I think you're talking about this video: https://youtu.be/8SkdfdXWYaI?t=1049 https://youtu.be/8SkdfdXWYaI?t=1049
  - Quequau 4y ago
    Yes! That's the talk.
jasonlfunk 4y ago
I probably wouldn’t use this to write code, but I could see it being really useful for navigating around a project. “Go to line 35” “Open the model controller” “Show the get method and set method side by side”
- anshumankmr 4y ago
  if you remember the keyboard shortcuts, you can be quite fast while working with VSCode. Voice will never be perfect.
nightski 4y ago
Spoken language is incredibly ambiguous. It's one thing to generate a drawing which can vary wildly in output and still be acceptable. It's another to specify something precisely to a computer. Working with non-programmers on a daily basis it is incredible how difficult it is to communicate even relatively simple things without confusion. So all the more power to them, but I am very skeptical. Especially since co-pilot has zero knowledge of the formal semantics of programming languages. This is a lot different than the half ass auto complete that it already does since that at least has some context.
- singularity2001 4y ago
  With copilot ambiguous language gets transformed into concrete syntax. If the implementation doesn't fit your ambiguous request, you should be able to refine … with ambiguous language. Theoretically this would create a "programming dialog" environment.
  - nightski 4y ago
    So you are going to have to verbalize something, interpret the code, build a mental model of how it works, and then if it does not match what you want go back to step 1? That sounds exhausting when we have spent countless human years developing languages which let us communicate our intentions precisely to computers. If you don't do this there is no ambiguity detector. Meaning it's entirely possible for the computer to interpret what you are saying completely different than intended, yet it is a perfectly valid interpretation. So the only one who can qualify if it got it right is you.
- tluyben2 4y ago
  It's the same with copilot; you have to know how to implement things to implement things with copilot (for the most part), but when you are a programmer and you could write the code, then you know the prompt to write to generate 10+ lines of code for 1 comment of text all day long. Especially for data transformation, copilot has been a real magic tool; if you put in a comment: /* this functions transforms this json from: { ... some complex structure in json } to this json: { ... some different structure in json } */ ... copilot comes up with the function that takes in the first and spits out the latter. Even if the fieldnames do not match etc, it usually 'guesses' right what fits on what (so it does have some context from it's learning phase what 'looks alike' or 'might be the same thing'. Example: I had a structure with firstName: string, lastName: string and a target structure with name: string; it just did name: firstName+' '+lastName, which was indeed what I wanted. But it comes up with more intricate stuff as well that is pretty much surprising (too human basically). What is another bonus; if you generated function transfromAtoB(a: A) above, then you only have to do: /* do the reverse of function transfromAtoB, accept json structure B as input and return structure A */ And it'll come up with the reverse. It's not hard to write yourself, but it's boring and error prone (some of these structures are huge). Now I press tab a bunch of times, and run the tests to see if it worked. I am also not that worried i'm infringing someone's open source code; this is all way to custom to look like anything else. That's where this shines; things where it verbatim copies something, you should've been using a library anyway. Statically typing and using typescript definitely works better than other combinations I have tried (C# was pretty bad last I tried it, JS is good but often subtly wrong because of type issues).
teratron27 4y ago
I'm sure this will work well with my Scottish accent... (or any non-US accent)
crucialfelix 4y ago
I spent half an hour today trying to convince the O2 voice agent to get me a real person. Conversational AI is a special kind of hell filled with unhappy paths. But for a glimpse of the future watch The Expanse or read William Gibson's Agency.
iillexial 4y ago
Those who say it's useless, what do you think about blind people using this, or those who couldn't type?
MauranKilom 4y ago
Related: https://www.youtube.com/watch?v=MzJ0CytAsec https://www.youtube.com/watch?v=MzJ0CytAsec It does look like we've made some progress in the 15 years since. I do wonder how this would work in an office setting though - so much noise, so much distraction, and so much crosstalk between programmers...
- avian 4y ago
  > I do wonder how this would work in an office setting Everyone gets a throat mic and the cubicle farm is full of unintelligible whispering instead of clacking of keyboards? Can't wait for the future. /s
- Hortinstein 4y ago
  Hahahah thank you for posting this, I was about to go look for this because I remember being in tears laughing when I saw it this first time and immediately thinonof this whenever I see voice controlled things
dimazhlobo 4y ago
Why does the oauth scope requires to “operate on your behalf” but the app is “not owned or operated by GitHub”. :/
tweetle_beetle 4y ago
From memory, there was a time (end of millenium?) when using voice recognition to write documents was the next big thing. There was a pricey bit of software for Windows that was popular with power users and they would spend hours training it to their voice. Then it seemed to just die off. I don't think it was bad technology, because I don't think novelty value was enough to account for its popularity - you had to put hours in to get it to work well, it wasn't a casual toy. What's changed since then in terms of technology? Unless it's very significant, I suspect it will go the same way. Apart from an assistive technology viewpoint, my gut instinct is that it's not that satisfying or rewarding talking to a computer all day.
- msaharia 4y ago
  Dragon speech? I used it quite a bit 10 years ago!
- kreddor 4y ago
  It might have been Dragon NaturallySpeaking. I remember toying around with it 20 years ago or so. Apparently it has just been bought by Microsoft.
  - tjixxu 4y ago
    I have good memories spending 4+ hours training Dragon to end up with what seemed like 30% accuracy.
- bagels 4y ago
  Many people dictate messages on their phones. Doctors use it extensively.
  - towawy 4y ago
    Not a doctor, but I also started to using the dictation feature on my iPhone more and more recently. It's often more convenient than typing when I'm walking (which I do a lot) and the pickup of voice messaging, talking to your phone like it's a mic, made me more comfortable doing it in public.
anshumankmr 4y ago
No. Thanks.
deleted 4y ago
[deleted]
susrev 4y ago
All i could think of while looking at this was having to tell Siri where every comma and period should go while texting with it. "insert curly brace", "insert semicolon", "insert insertion", etc. does not sound to fun.
glenjamin 4y ago
There was an excellent talk at Strange Loop a few years ago by Emily Shea about how she'd learnt to code vim using her voice to combat RSI. https://www.youtube.com/watch?v=YKuRkGkf5HU https://www.youtube.com/watch?v=YKuRkGkf5HU The demos are in Ruby, but I could imagine that languages with strong type-aware auto-completion could be easier to do.
dustedcodes 4y ago
When I talk to my Google Home then 50% of my brain power is engaged in predicting and working out how to best phrase something so that the "AI" understands what I mean and the other 50% is used to actually think about what I want to accomplish in the first place. This is just about okay for things like switching lights on/off or requesting a nice song I want to listen to, but I could never be productive programming like this. When I'm in the zone I don't want to have to waste any mental capacity on supplementing an imperfect AI, I want to be thinking 100% about what I want to code and just let my fingers do the work. For that reason I think this will be less appealing to developers than GitHub may think, otherwise I think it's a cool idea.
- wizardofmysore 4y ago
  It's really useful for those who have challenges typing (arthritis, disabilities etc..), perhaps not best for general audience as typing with auto complete is faster.
- danielbln 4y ago
  The assistants (Google, Alexa, Siri) are not great at NLP. Compare how you speak to them vs speaking to a LLM like gpt-3, there is a world of difference. The latter feels like speaking to a human, the former more like your trying to get your voice commands into a state machine.
- raylad 4y ago
  Around 1998 I broke my collarbone and had to use Dragon Dictate. I found that for general subjects it was quite difficult to use because of the fairly poor recognition rate. But when I talked about computers, it got almost everything right. I assumed it must have been trained by the developers, who talked about computers mostly. This is another special purpose vocabulary, so it seems as if it would have a good chance of a high recognition rate.
  - eurasiantiger 4y ago
    It’s most likely just Cortana bolted on to Copilot.
onion2k 4y ago
I've tried writing documentation and fiction using text-to-speech and, for me, it doesn't work because the apparently the of my part brain I use to think about what I'm going to say is the same part I use to actually say it, so I can't do both things at once. I end up writing far more slowly than I can type.
polishdude20 4y ago
Thank god we're remote. An open office space with this would suck.
deleted 4y ago
[deleted]
gopheryourshelf 4y ago
Imagine an office where everyone is sitting screaming at their computer.
singularity2001 4y ago
In case anyone else stopped after watching the video, if you scroll down a bit further you see the list of FEATURES Write/edit code Just state your intent in natural language and let Hey, GitHub! do the heavy lifting of suggesting a code snippet. And if you don't like what was generated, ask for a change in plain English. Go to the next method Code navigation No more using mouse and arrow keys. Ask Hey, GitHub! to... go to line 34 go to method X go to next block Control the IDE "Toggle zen mode", “run the program”, or use any other VisualStudio Code command. Code Summarization Don’t know what a piece of code does? No problem! Ask Hey, GitHub! to explain lines 3-10 and get a summary of what the code does. Explain lines 3 - 10
evnix 4y ago
Eye strain is one reason I have been waiting for something like this. If I could close my eyes and just navigate the codebase through a mental modal and some voice commands, I really wouldn't mind paying! I have looked at some tools for the blind, but you need just way too much dedication for it to work for you and since you have working eyes it is usually easier to just open your eyes.
tabasselejambon 4y ago
Let's try to picture the noise in an openspace full of people using that ... focusing is going to be difficult, well at least for people like me who are easily distracted by background noise/conversations.
ddevault 4y ago
Someone emailed me the other day to share their FOSS voice control system. I was really impressed. It seems to map syllables onto actions in a modal sense ala vim. If I were to build a voice control system, it would look much like this. https://numen.johngebbie.com/index.html https://numen.johngebbie.com/index.html It's free software, it's local to your machine, you don't have to sign up for it, and it works today.
mtkhaos 4y ago
Nice attempt and interesting workflow using a prompt based transformer. I would prefer being able to spawn a command palette and skip over the voice, alongside having the choice between different variations.
okasaki 4y ago
Great for accessibility, but I don't see this would work well in an open office, or even at home if other people are around. Seems really annoying.
pmontra 4y ago
My reactions to the demo (when all is good there is no reaction, so here are only the problematic ones, sorry) 1) import matplotlib.pyplot as plt Why "as plt"?! Let the import alone. But this is a matter of style. 2) Get titanic csv data from the web [...] Surprise, it turns out that "the web" is an URL on raw.githubusercontent.com Hopefully I'll be able to spell an URL of my choice 3) clean records from titanic data where age is null Somehow I already know that there is an Age field and somehow it knows that it must capitalize age into Age 4) fill null values of column Fare with average column values The generated code looks great but somehow I managed to spell a capitalized Fare this time :-) (this is probably a typo in the demo) 5) Hey,Github! New line Inserting a new line can't take so many words. We're going to do without new lines or rely on a formatter or something equivalent. 6) plot line graph of age vs fare column This is where it becomes evident that there was no need to import as plt because I'm not pressing those keys anyway. But this is style and it's going to be uniform across all the users of these tools. 7) Hey, Github! Run program Good. Considerations: A) Why do commands (new line, run) need "Hey, Github!" which is pretty long and terrible to repeat all the day long (just imagine having to say Hey Joe every time we have to say a sentence to Joe, withing a long conversation with Joe) and text-to-code doesn't? B) We got a graph at the end. Now what should I do to edit the code in those 99% of cases where I got the graph wrong? An acceptable answer could be mouse and keyboard. It's a little underwhelming but voice to code already gave me the structure of the code. C) Does that mean that Microsoft and GitHub are going to know all the closed source code we'll write for our customers (there might be contractual implications) or is this something that will be self hosted in our machines?
- rahulpandita 4y ago
  GitHubNext here! Here is a little writeup that explains a bit more about the project https://github.com/githubnext/githubnext/tree/main/HeyGitHub https://github.com/githubnext/githubnext/tree/main/HeyGitHub Hope this is helpful :)
kajaktum 4y ago
This feels like Github expanding because it can't find anything else to do...It being a for profit organization means that it's unable to say "you know what we pretty much have everything we wanted so we're just going into a maintenance/optimization mode". This happens all the time in open source project where they simply tell their users to move elsewhere for the better alternatives but will never happen to a for profit organization.
lakomen 4y ago
Imagine sitting there, talking to your computer, and trying to get the notations right. If err unequal nil opening bracket, no no don't open the racket opening bracket... BRACKET, do you know what a bracket is No don't do a do while, delete delete. Don't delete everything... sigh Well something like that, I imagine it being a very painful experience.
- 3D30497420 4y ago
  Or debugging. Goodness. I can only imagine.
- dorkwood 4y ago
  Maybe you could use little clicks and pops with your mouth to signify different characters. The computer could learn which one is which. That way instead of typing "if (int i = 0; i < count; i++)" you could say something like "if beep int i click zero boop i bop count boop i pop pop zing" This would achieve the same thing, but much faster and with less effort than typing.
  - xigoi 4y ago
    If saying that is faster than typing it, you're really slow at typing.
nxpnsv 4y ago
GitHub is doing a whole lot. I think I prefer to edit my code in an editor, not on the website where it's hosted. And I think I don't want fancy AI driven code editor features using my code either. But I guess it is nice they are considering solutions for vision impaired users.
hbarka 4y ago
How does it do with SQL?
prima-facie 4y ago
Hey Github what did the previous developer actually _mean_ with this piece of legacy code?
pcj-github 4y ago
I could see it being useful for things like "goto line 42" or "rename this file as...", or very simple things like that, otherwise, I don't want the cognitive overhead of having to translate coding intent through a voice interpreter.
Cort3z 4y ago
However weird and seemingly useless this might appear to the normal programmer on here, I see this as a huge accomplishment and an incredibly important tool. Why? Accessibility. Let’s hope that I never get in a serious accident or get an disabling disease, but if I do I am not planning on giving up coding. What would you do if you lost your hands, or became permanently paralyzed. This is the tool we need to combat that. Hats off to github on this one.
- bamboozled 4y ago
  People who cannot see or use a keyboard already use tools like this to code. Been doing this for a long time.
eurasiantiger 4y ago
I do not want this.
deleted 4y ago
[deleted]
karmasimida 4y ago
Actually would be useful. If this is reliable I would pay to use it to some capacity, like add an argument.
birriel 4y ago
In the meantime, Talon is pretty good. You can use Vim motions and commands as you normally would, except using your voice (this applies to any editor, really): https://talonvoice.com/ https://talonvoice.com/
- willjp 4y ago
  Talon is exceptional, I only wish it was more natural to drive cli commands, I find I need to spell them out which I’m still quite slow at.
hdjjhhvvhga 4y ago
I'd like to see how they do with my creative variable and function names.
raidicy 4y ago
I really hope this is very easy to use. I have severe RSI and can barely surf the web. I tried using other voice to code stuff and it just hurt my voice so I'm hoping I can speak very naturally. I'm really looking forward to seeing if this can help me code again.
karmasimida 4y ago
One concern is in office space, saying things aloud is ... awkward to say the least.
univue 4y ago
sgdf
univue 4y ago
Addd some comments
Sheeny96 4y ago
I feel like this is being misunderstood - the long term view of this wouldn't be for code scribing, it'd be for non technical people to be able to instantly create things. Imagine being able to say outloud to your phone "hey, create me a view of all the weather data from the past year correlated against x and y in z view format". The code is the means, not the product.
- urthor 4y ago
  Exactly. The crazy thing is... this probably will work. In 20 years. But, it probably will work. There is absolutely no reason you cannot use a neural network to transcribe appropriately phrased requirements into an AST.
  - BiteCode_dev 4y ago
    Before copilot, we were far from it. But now, given how magical this thing is, it opens many doors to what's possible with no code. I never really believe in anything no code before apart from Excel and RAD. But basic tasks are going to get accessible to a lot of people sooner than I expected
    - urthor 4y ago
      I'm almost certain a trained professional is still going to do a far better job than an amateur. That part is unlikely to change much, if it all. Trained professionals will always outperform the latest fellow dragged off the street.
      - BiteCode_dev 4y ago
        That doesn't matter much. You free part of the industry from needing professionals for some simple tasks, which is enough to empower users with a lot of possibilities, and focus pros on where they are really needed. "Hey phone, next time mum send me a text about voting, you can send back a 'ok boomer'?" or "hey phone, can you setup a webpage that list my tiktok videos up to last year?" Most people are not going to hire a pro for that, but we could end up with a AI general enough to be able to do that for users. It's ok if the result is not extensible, maintainable or modular.
pfd1986 4y ago
I think commenters here are -- as usual -- missing the point. This is the training ground (literally) for better models able to respond to commands like "take the CSV from me desktop, plot columns A and D and check if the KL divergence os close to zero". And from that to more complex tasks. You always need the first step and this is it. I'm bullish.
- BiteCode_dev 4y ago
  Exactly. Copilot is getting better everyday, because it's learning from the way we are using it.
  - rahulpandita 4y ago
    GitHubNext here! We appreciate your support.
- rahulpandita 4y ago
  GitHubNext here! We appreciate your support.
deleted 4y ago
[deleted]
akuji1993 4y ago
export const ButtonComponent; FunctionComponent no Github no semicolon i meant colon Github backspace 5 times no backspace delete delete Github Arrrgh goddammit
- cdrini 4y ago
  What you're describing is more like dictation. What you'd probably say is "export the button component", and it would determine the syntax.
  - akuji1993 4y ago
    Which will probably, outside of small, perfectly planned experiments, work similarly well
    - cdrini 4y ago
      You're free to speculate, but I would respectfully disagree. Extrapolating from how effective GPT3/Copilot are at interpreting text and generating code, I think it has more of a chance of working well than any other tech has ever had. I definitely hope it succeeds, anyhow!
philmander 4y ago
This is effectively a new higher level programming language without a fixed syntax. Describing more the "what", not "how", and being much closer to natural language over computer language. The voice part seems like an (albeit important) accessibility add on. I'm sure it won't be perfect but an amazing step forward in the evolution of programming languages
- meowface 4y ago
  I could be wrong, but I think (minus editor commands) much of this can be emulated in existing Copilot by writing a comment symbol followed by natural language. I wouldn't even be surprised if under the hood "Hey, GitHub!" is basically doing exactly that with the voice input.
WormholeCreator 4y ago
it is not practical if we have to describe each and every line. Also, imagine you are sitting in an office with other team mates - what happens if all of them talk together but are working on different projects. It will disturb others in terms of noise pollution. but it will definitely be a fun project and might work perfectly when you are working alone from home.
jenscow 4y ago
bool success equals user dot no i mean ah fuck stop stop quit
danjc 4y ago
And you thought open plan offices were bad already!
mindvirus 4y ago
This is awesome. I could see using this to write code on my phone even.
tempodox 4y ago
Imagine using this in a setting where you're not alone in the room. Imagine using this surrounded by other developers who do the same.
squarefoot 4y ago
If translation is semantic and not literally identical, chances are that the user asks for a piece of code and it outputs something that is 100% identical to code that is copyrighted elsewhere. Big "blame the AI" legal loophole waiting to happen?
falcor84 4y ago
I think this, or a future version of this, would have real potential. I'm thinking about this in terms of the navigator-pilot pair programming approach, and believe that as a senior, if it's even half-as-good as working with a fresh out of uni hire, then it could have real value. When there's a piece of code that I would like written, when I have good test cases in mind, but would prefer to offload it on someone, I could perhaps write the test cases and function signatures (maybe with the bot's help), get the bot to fill in the blanks until it passes the tests, and then give it direct feedback on how to refactor the code. I've signed up for the waiting list and am excited to try this out.
- knutzui 4y ago
  What you are describing is more akin to what GitHub Copilot already does. It is really good at taking a description and a function signature and producing a solution. Paired with a solid test suite it can definitely speed up development in my experience.
deleted 4y ago
[deleted]
lleontop 4y ago
We have come a long way. I remember when announcements like this one were done by companies on April 1st!
kdmytro 4y ago
This is not going to play well with open-space offices.
amarant 4y ago
Oh cool, my brother used to wish out loud something like this existed a few years back when his wrists were really killing him. He's wrists were so far gone he couldn't even type on a ergonomical keyboard for any greater duration of time, so he used to wish he could just talk instead. For me, I got a ergonomical keyboard before my wrists went bad, and so far they seem to be holding up! Moral of the story: get a good keyboard early, or you might need a tool like this one someday!
darepublic 4y ago
Execution is everything with this. I've wanted something like this so I could actually code while performing other activities or in various states of intoxication. Don't code and drive. Don't drink and code
FloatArtifact 4y ago
The challenges that remain in speech coding is not generating code as much as it is navigating through existing code or an application. There's only two ways to do this effectively and unfortunately no one has taken the true path to accessibility. The more common way is plugins/extensions to grab a information from the editor. Accessibility is more than just one editor. It's the OS and all the applications. Microsoft needs to take the hard route to make an accessibility UI automation server to grab that information and only make up the difference through plugins as needed. It's all about grabbing information from the application and generating on the fly commands, not just parsing free dictation in order to get the best accuracy. It takes a lot of expertise to make any sort of UI automation, fast and efficient for navigating and selecting text or out of focus menu items. I've fussed around and managed to get tree sitter to navigate across code. For example generic commands are like 'next function'. Code simply isn't pronounceable when it's written by others. Therefore, navigating across generic tokens is really the best method. Then other methods can be used for fine navigation if needed. My hope is that they develop a grammar system that is open source and integrates with accessibility frameworks focused on performance. I wish I could have a phone call with the development team.
- skydhash 4y ago
  I think an accessibility a la vim or with something like tree sitter, would help immensely like: “Top of file Down 5 lines Modify import source to … inside first class Down 5 methods Insert new method after Inside arg list Append arg named … of type … … …” And add a way to indentify types and parameters with special pronunciation.
  - FloatArtifact 4y ago
    I recognize that all applications are not accessible through accessibility APIs. However, there is no high level access to accessibility APIs. There are quite a few for automated testing UI. However, none of them are performant enough for speech to code or screen readers. Testing automation frameworks don't really require high performance. Accessibility accessing the content of the application and the context is what's important. It's more important than the speech recognition backend. Speech recognition shines work best with a narrow context. (when those commands are available) The type of performance we need as a speech recognition community and screen reader community is quite high. By the beginning of speech and just before decode time information needs to be available to be parsed for navigation/editing. That way these tokens can be weighted as commands for recognition. Commands could be modeled after vim functionality though. Outside of tree sitter it would be interesting to hook into hooking into as a client a language protocol server. However, I think they only expect one client. In addition, I still see that as a lesser approach without dedicated support for high performance UI automation server for speech recognition engine to leverage.
troelsSteegin 4y ago
What if the code in question is a DSL? Something say that is syntactically python, but with a namespace defined through a narrow set of imports. This would be interesting to explore for end-user scripting.
deleted 4y ago
[deleted]
boredumb 4y ago
As someone who works remotely from home, the last thing I need is to start babbling to myself in code for 8 hours a day. I imagine that's a one way ticket to developing some sort of disorder.
silverlake 4y ago
I’m working on something similar. The target market is the 99% of people who want to program ad-hoc domain-specific problems. For example, generating charts w/o having to dig through all the data sources (Wolfram Alpha does a simple version of this). Building a financial risk model for a client’s specific request (you have to be a whiz at Excel, python or some internal ide). Even for home automation, my mom can’t use Alexa’s awful app to customize routines. I don’t think the voice part is necessary. It’s easy enough to slap ASR on the front. But going from natural language -> full problem spec -> code is hard in the general case, but doable in well-understood domains. Why can’t Scotty talk to a computer? (https://youtube.com/watch?v=hShY6xZWVGE&feature=share https://youtube.com/watch?v=hShY6xZWVGE&feature=share)
kevmo314 4y ago
Why are all the comments here so negative? Maybe typing is a hard sell, but some of the navigation stuff seems quite useful. Even being able to invoke VS Code's command palette would be really cool with this. Something like "Open Dockerfile" would be useful and maybe faster than typing.
kashanjunaid 4y ago
very intresting!
manesioz 4y ago
Interesting. I would find this annoying because its so different from what I'm used to, but the potential it has for people with disabilities is huge.
qntmfred 4y ago
see also https://serenade.ai/ https://serenade.ai/ https://news.ycombinator.com/item?id=22404264 https://news.ycombinator.com/item?id=22404264
geewee 4y ago
Having programmed and navigated my PC via voice exclusively for about 6 months, done a ton of research and written several articles about it and what options are out there [0][1], I think might be pretty ground-breaking stuff. Inputting code with voice is generally difficult, often due to variable names, casing, punctuation etc being hard to get right in voice-to-text. I think this might help quite a lot with that. _However_, some of the hardest things in voice coding isn't necessarily just the input. Navigating large codebases is hard, and particularly editing existing code can be extremely difficult, probably much more difficult than just inputting new code. I have my doubt that with the demonstration shown here, that it's able to make complex editing tasks simple, but if it does - I cannot overstate how huge of a leap forward it is. [0]: https://www.gustavwengel.dk/state-of-voice-coding-2017/ https://www.gustavwengel.dk/state-of-voice-coding-2017/ [1]: https://www.gustavwengel.dk/state-of-voice-coding-2019/ https://www.gustavwengel.dk/state-of-voice-coding-2019/
- jovial_cavalier 4y ago
  >Having programmed and navigated my PC via voice exclusively for about 6 months... I'm curious, why have you done this?
  - FloatArtifact 4y ago
    I can't speak for his use case. However, people with medical conditions like RSI, stroke or anything that limits their action between keyboard and mouse. However, the average developer doesn't need those fine-grained navigation controls but can still benefit from enhanced input through voice. Some have mental disabilities who interface differently. Others are simply supplement their input as an average developer by voice as a preventative measure for repetitive strain RSI. The day the hope is develop something that every developer could see the value and leverage. In a way accessibility is for everyone. In general I see accessibility as a hierarchy that could benefit everyone. Accessibility APIs, close to real time OCR, Eye tracking, alternative inputs (eg pedal, touch pad, stylus) allowing for the broadest possible input and APIs to extract information from applications. Extraction of information from applications and input to applications allows the user to specialize for their use case. My experience as people will become experts in voice their command vernacular shortens as they carve out their niche use case. It goes beyond singular shortcuts too series of actions to get stuff done. However, what really means to happen is voice systems need access to the OS and to the application to really shine. That would empower not only navigation for those that are disabled but context-specific commands that are intuitive and abstracted like next function or parameter.
nicolas_lorenzi 4y ago
I imagine happiness in the open space
danwee 4y ago
Curious: In "Clean records from titanic data where age is null", how does it know that the age field is exactly `Age` and not just `age`? You cannot know this without examing the data set (the headers), so is the software inspecting the loaded CSV "on the fly" before us telling it to actually execute the code?
lkrubner 4y ago
My worst prediction ever was at the end of my book, when I struck a positive note about voice interfaces. The startup I was at in 2015 had the pitch "Let your sales people talk directly to Salesforce" and we pushed the limits of what we could do with NLP. That particular startup had spectacularly bad management and so it flamed out in a series of screaming, raging fights, which I documented here: https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/0998997617/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1668096943&sr=8-1 https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/09... But at the end of the book I struck an upbeat note, about how the technology was advancing quickly and within 3 or 4 years someone would achieve something much greater than our own limited successes. But I was wrong. 7 years later I'm surprised at how little progress there has been. I don't see any startup that's done much better than what we did in 2015. Voice interfaces remain limited in accuracy and use.
polyterative 4y ago
I have rsi, github please make it work well
dang 4y ago
This is the third such thread in the last 24 hours which consists of nothing but an elaborate waiting list signup. I've changed the titles to make that clear. GitHub Blocks – waiting list signup - https://news.ycombinator.com/item?id=33537706 https://news.ycombinator.com/item?id=33537706 - Nov 2022 (41 comments) GitHub code search – waiting list signup - https://news.ycombinator.com/item?id=33537614 https://news.ycombinator.com/item?id=33537614 - Nov 2022 (48 comments) A good HN discussion needs more than a waiting list signup. A good time to have a thread would be when something is actually available.
hintymad 4y ago
So this is a frontend of Copilot. The example of "import pandas" getting translated into "import pandas as pd" is pretty convincing, as the tool helps developers to state their intentions. On the other hand, "hey, github, a new line" kills me.
v3ss0n 4y ago
So software development houses will become call centers.

316 points by rcshubhadeep 4y ago

Programming Perl with speech recognition (an oldie but goodie) https://www.youtube.com/watch?v=vPXEDW30qBA https://www.youtube.com/watch?v=vPXEDW30qBA

Tade0 4y ago

I hope to see the click consonant "‖" adopted as "||" one day.

ggerganov 4y ago

Very interesting - I was sort of expecting it to happen soon. I have been playing with using Whisper + Github Copilot in Vim [0]. The Whisper text transcription runs offline with a custom C/C++ inference and I use Copilot through the copilot.nvim plugin for Neovim. The results were very satisfying. Edit: And just in case there is interest in this, the code is available [1]. It would be very awesome if someone helps to wrap this functionality in a proper Vim plugin. [0] https://youtu.be/3flN9kTcZJY https://youtu.be/3flN9kTcZJY [1] https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.nvim https://github.com/ggerganov/whisper.cpp/tree/master/example...

chocolatkey 4y ago

If this works well, I would pay a seriously high amount of money. My daily coding time is currently limited by the pain in my hand/fingers that eventually becomes too uncomfortable, and I have to wait for a "cooldown" period of days to "reset" my hands back to normal. I can't even code on a normal keyboard or trackpad for a long time anymore. The problem with current voice programming systems is they're just too slow so I end up getting impatient and using my fingers anyway

eajakobsen 4y ago
I imagine you have done extensive research on your own, but in any case I found this article by Josh Comeau on coding with voice commands and eye tracking very interesting: https://www.joshwcomeau.com/blog/hands-free-coding/ https://www.joshwcomeau.com/blog/hands-free-coding/
- langsoul-com 4y ago
  I wonder if the quest pro could do the eye tracking as well? It has pretty extensive eye tracking cameras, not sure how precise they're though. Could it also do a virtual keyboard, but a custom layout to not trigger arm, elbow and hand pains?
  - atonalfreerider 4y ago
    We built a VR prototype a year ago for voice to code using Codex in VR: https://www.youtube.com/watch?v=icHLoxOFerk https://www.youtube.com/watch?v=icHLoxOFerk
langsoul-com 4y ago
I'm assuming you have an ergonomic keyboard? If so which one?
- tgv 4y ago
  And a movable monitor, decent chair and desk at the appropriate height...
  - Comevius 4y ago
    An ergonomic keyboard layout like Workman doesn't hurt either. QWERTY was made with typewriter jamming in mind, sacrificing ergonomics completely, especially for English.
- chocolatkey 4y ago
  Ergodox EZ
Hesinde 4y ago
Have you tried an organ MIDI pedalboard and a script to translate MIDI to keystrokes? You could also put a micro controller between the pedalboard and your computer so that it looks like a normal keyboard to the computer. I do not know whether that would be pratical, but some sort of feet keyboard is in my idea space for what if.
ron22 4y ago
If you want to code with your voice, also checkout https://github.com/cursorless-dev/cursorless https://github.com/cursorless-dev/cursorless
rahulpandita 4y ago
GitHubNext here! We appreciate your support. Please consider helping us by signing up for the experiment on the website and providing feedback. :)

singularity2001 4y ago

Drop the "Hey Github" nonsense (hopefully it's only for illustration purposes anyways) and … this will be a generational paradigm change in how to write code… if it works. The hard part will be editing code with your voice too. Like "no, I meant …" etc. VERY PROMISING, in any case you can just manually fill the gaps with the keyboard!

wiseowise 4y ago
> this will be a generational paradigm change in how to write code… if it works Why?
- mkmk3 4y ago
  I could see it maybe being important once github codepilot is embedded in it? You tell it roughly what you want and then adapt by hand. But it is kinda funny seeing parent make such claims so early
  - singularity2001 4y ago
    > once github codepilot is embedded That's exactly the point of the demo, no?
    - mkmk3 4y ago
      Yup my bad for jumping to conclusions. This certainly seems worthwhile exploring
- wokwokwok 4y ago
  How can it not be a paradigm change when it changes the way people write code from “write by hand” to “generated by ai with natural language”? The problem with speech to code has always been that precise syntax is hard, but AI codegen solves that. So, no, it might not take off, but I feel like if it does, then it means ai-codegen will become the dominant way code is crafted. That would be paradigm shifting. It’s inconceivable that it wouldn’t be.

mezobeli 4y ago

Copilot -> Pilot

ausudhz 4y ago

Next is thoughts to code. Just read my mind I'm gonna seat there and think

nnurmanov 4y ago
Song to code, we shall be singing our next systems:)
- singularity2001 4y ago
  dance to code. transform the esthetics of your movements into … whatever your boss requires.

hcnews 4y ago

To note, there's a class action lawsuit against GitHub Co-Pilot since it learns from a bunch of open source code with very specific licenses. It's very interesting from establishing copyright in an AI training perspective. Hopefully it goes the distance and some nuanced arguments come out in the court case. https://www.theverge.com/2022/11/8/23446821/microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data https://www.theverge.com/2022/11/8/23446821/microsoft-openai...

kgrax01 4y ago

People can’t seriously believe this is going to be useful at all? I can see this helping as an accessibility tool, but beyond that I don’t think it will be useful. This kind of assumes you know everything about what you’re doing, most of the time you don’t.

Quequau 4y ago

I remember a talk given some years ago by a man who was using voice to text for creating source code. The key point I remember from his talk & demonstration is that it was not casual ordinary speech but instead a very weird mashup of sounds intended to represent the various symbols which we use in source code.

simme_ 4y ago
I think you're talking about this video: https://youtu.be/8SkdfdXWYaI?t=1049 https://youtu.be/8SkdfdXWYaI?t=1049
- Quequau 4y ago
  Yes! That's the talk.

jasonlfunk 4y ago

I probably wouldn’t use this to write code, but I could see it being really useful for navigating around a project. “Go to line 35” “Open the model controller” “Show the get method and set method side by side”

anshumankmr 4y ago
if you remember the keyboard shortcuts, you can be quite fast while working with VSCode. Voice will never be perfect.

nightski 4y ago

Spoken language is incredibly ambiguous. It's one thing to generate a drawing which can vary wildly in output and still be acceptable. It's another to specify something precisely to a computer. Working with non-programmers on a daily basis it is incredible how difficult it is to communicate even relatively simple things without confusion. So all the more power to them, but I am very skeptical. Especially since co-pilot has zero knowledge of the formal semantics of programming languages. This is a lot different than the half ass auto complete that it already does since that at least has some context.

singularity2001 4y ago
With copilot ambiguous language gets transformed into concrete syntax. If the implementation doesn't fit your ambiguous request, you should be able to refine … with ambiguous language. Theoretically this would create a "programming dialog" environment.
- nightski 4y ago
  So you are going to have to verbalize something, interpret the code, build a mental model of how it works, and then if it does not match what you want go back to step 1? That sounds exhausting when we have spent countless human years developing languages which let us communicate our intentions precisely to computers. If you don't do this there is no ambiguity detector. Meaning it's entirely possible for the computer to interpret what you are saying completely different than intended, yet it is a perfectly valid interpretation. So the only one who can qualify if it got it right is you.
tluyben2 4y ago
It's the same with copilot; you have to know how to implement things to implement things with copilot (for the most part), but when you are a programmer and you could write the code, then you know the prompt to write to generate 10+ lines of code for 1 comment of text all day long. Especially for data transformation, copilot has been a real magic tool; if you put in a comment: /* this functions transforms this json from: { ... some complex structure in json } to this json: { ... some different structure in json } */ ... copilot comes up with the function that takes in the first and spits out the latter. Even if the fieldnames do not match etc, it usually 'guesses' right what fits on what (so it does have some context from it's learning phase what 'looks alike' or 'might be the same thing'. Example: I had a structure with firstName: string, lastName: string and a target structure with name: string; it just did name: firstName+' '+lastName, which was indeed what I wanted. But it comes up with more intricate stuff as well that is pretty much surprising (too human basically). What is another bonus; if you generated function transfromAtoB(a: A) above, then you only have to do: /* do the reverse of function transfromAtoB, accept json structure B as input and return structure A */ And it'll come up with the reverse. It's not hard to write yourself, but it's boring and error prone (some of these structures are huge). Now I press tab a bunch of times, and run the tests to see if it worked. I am also not that worried i'm infringing someone's open source code; this is all way to custom to look like anything else. That's where this shines; things where it verbatim copies something, you should've been using a library anyway. Statically typing and using typescript definitely works better than other combinations I have tried (C# was pretty bad last I tried it, JS is good but often subtly wrong because of type issues).

teratron27 4y ago

I'm sure this will work well with my Scottish accent... (or any non-US accent)

crucialfelix 4y ago

I spent half an hour today trying to convince the O2 voice agent to get me a real person. Conversational AI is a special kind of hell filled with unhappy paths. But for a glimpse of the future watch The Expanse or read William Gibson's Agency.

iillexial 4y ago

Those who say it's useless, what do you think about blind people using this, or those who couldn't type?

MauranKilom 4y ago

Related: https://www.youtube.com/watch?v=MzJ0CytAsec https://www.youtube.com/watch?v=MzJ0CytAsec It does look like we've made some progress in the 15 years since. I do wonder how this would work in an office setting though - so much noise, so much distraction, and so much crosstalk between programmers...

avian 4y ago
> I do wonder how this would work in an office setting Everyone gets a throat mic and the cubicle farm is full of unintelligible whispering instead of clacking of keyboards? Can't wait for the future. /s
Hortinstein 4y ago
Hahahah thank you for posting this, I was about to go look for this because I remember being in tears laughing when I saw it this first time and immediately thinonof this whenever I see voice controlled things

dimazhlobo 4y ago

Why does the oauth scope requires to “operate on your behalf” but the app is “not owned or operated by GitHub”. :/

tweetle_beetle 4y ago

From memory, there was a time (end of millenium?) when using voice recognition to write documents was the next big thing. There was a pricey bit of software for Windows that was popular with power users and they would spend hours training it to their voice. Then it seemed to just die off. I don't think it was bad technology, because I don't think novelty value was enough to account for its popularity - you had to put hours in to get it to work well, it wasn't a casual toy. What's changed since then in terms of technology? Unless it's very significant, I suspect it will go the same way. Apart from an assistive technology viewpoint, my gut instinct is that it's not that satisfying or rewarding talking to a computer all day.

msaharia 4y ago
Dragon speech? I used it quite a bit 10 years ago!
kreddor 4y ago
It might have been Dragon NaturallySpeaking. I remember toying around with it 20 years ago or so. Apparently it has just been bought by Microsoft.
- tjixxu 4y ago
  I have good memories spending 4+ hours training Dragon to end up with what seemed like 30% accuracy.
bagels 4y ago
Many people dictate messages on their phones. Doctors use it extensively.
- towawy 4y ago
  Not a doctor, but I also started to using the dictation feature on my iPhone more and more recently. It's often more convenient than typing when I'm walking (which I do a lot) and the pickup of voice messaging, talking to your phone like it's a mic, made me more comfortable doing it in public.

anshumankmr 4y ago

No. Thanks.

deleted 4y ago

[deleted]

susrev 4y ago

All i could think of while looking at this was having to tell Siri where every comma and period should go while texting with it. "insert curly brace", "insert semicolon", "insert insertion", etc. does not sound to fun.

glenjamin 4y ago

There was an excellent talk at Strange Loop a few years ago by Emily Shea about how she'd learnt to code vim using her voice to combat RSI. https://www.youtube.com/watch?v=YKuRkGkf5HU https://www.youtube.com/watch?v=YKuRkGkf5HU The demos are in Ruby, but I could imagine that languages with strong type-aware auto-completion could be easier to do.

dustedcodes 4y ago

When I talk to my Google Home then 50% of my brain power is engaged in predicting and working out how to best phrase something so that the "AI" understands what I mean and the other 50% is used to actually think about what I want to accomplish in the first place. This is just about okay for things like switching lights on/off or requesting a nice song I want to listen to, but I could never be productive programming like this. When I'm in the zone I don't want to have to waste any mental capacity on supplementing an imperfect AI, I want to be thinking 100% about what I want to code and just let my fingers do the work. For that reason I think this will be less appealing to developers than GitHub may think, otherwise I think it's a cool idea.

wizardofmysore 4y ago
It's really useful for those who have challenges typing (arthritis, disabilities etc..), perhaps not best for general audience as typing with auto complete is faster.
danielbln 4y ago
The assistants (Google, Alexa, Siri) are not great at NLP. Compare how you speak to them vs speaking to a LLM like gpt-3, there is a world of difference. The latter feels like speaking to a human, the former more like your trying to get your voice commands into a state machine.
raylad 4y ago
Around 1998 I broke my collarbone and had to use Dragon Dictate. I found that for general subjects it was quite difficult to use because of the fairly poor recognition rate. But when I talked about computers, it got almost everything right. I assumed it must have been trained by the developers, who talked about computers mostly. This is another special purpose vocabulary, so it seems as if it would have a good chance of a high recognition rate.
- eurasiantiger 4y ago
  It’s most likely just Cortana bolted on to Copilot.

onion2k 4y ago

I've tried writing documentation and fiction using text-to-speech and, for me, it doesn't work because the apparently the of my part brain I use to think about what I'm going to say is the same part I use to actually say it, so I can't do both things at once. I end up writing far more slowly than I can type.

polishdude20 4y ago

Thank god we're remote. An open office space with this would suck.

deleted 4y ago

[deleted]

gopheryourshelf 4y ago

Imagine an office where everyone is sitting screaming at their computer.

singularity2001 4y ago

In case anyone else stopped after watching the video, if you scroll down a bit further you see the list of FEATURES Write/edit code Just state your intent in natural language and let Hey, GitHub! do the heavy lifting of suggesting a code snippet. And if you don't like what was generated, ask for a change in plain English. Go to the next method Code navigation No more using mouse and arrow keys. Ask Hey, GitHub! to... go to line 34 go to method X go to next block Control the IDE "Toggle zen mode", “run the program”, or use any other VisualStudio Code command. Code Summarization Don’t know what a piece of code does? No problem! Ask Hey, GitHub! to explain lines 3-10 and get a summary of what the code does. Explain lines 3 - 10

evnix 4y ago

Eye strain is one reason I have been waiting for something like this. If I could close my eyes and just navigate the codebase through a mental modal and some voice commands, I really wouldn't mind paying! I have looked at some tools for the blind, but you need just way too much dedication for it to work for you and since you have working eyes it is usually easier to just open your eyes.

tabasselejambon 4y ago

Let's try to picture the noise in an openspace full of people using that ... focusing is going to be difficult, well at least for people like me who are easily distracted by background noise/conversations.

ddevault 4y ago

Someone emailed me the other day to share their FOSS voice control system. I was really impressed. It seems to map syllables onto actions in a modal sense ala vim. If I were to build a voice control system, it would look much like this. https://numen.johngebbie.com/index.html https://numen.johngebbie.com/index.html It's free software, it's local to your machine, you don't have to sign up for it, and it works today.

mtkhaos 4y ago

Nice attempt and interesting workflow using a prompt based transformer. I would prefer being able to spawn a command palette and skip over the voice, alongside having the choice between different variations.

okasaki 4y ago

Great for accessibility, but I don't see this would work well in an open office, or even at home if other people are around. Seems really annoying.

pmontra 4y ago

My reactions to the demo (when all is good there is no reaction, so here are only the problematic ones, sorry) 1) import matplotlib.pyplot as plt Why "as plt"?! Let the import alone. But this is a matter of style. 2) Get titanic csv data from the web [...] Surprise, it turns out that "the web" is an URL on raw.githubusercontent.com Hopefully I'll be able to spell an URL of my choice 3) clean records from titanic data where age is null Somehow I already know that there is an Age field and somehow it knows that it must capitalize age into Age 4) fill null values of column Fare with average column values The generated code looks great but somehow I managed to spell a capitalized Fare this time :-) (this is probably a typo in the demo) 5) Hey,Github! New line Inserting a new line can't take so many words. We're going to do without new lines or rely on a formatter or something equivalent. 6) plot line graph of age vs fare column This is where it becomes evident that there was no need to import as plt because I'm not pressing those keys anyway. But this is style and it's going to be uniform across all the users of these tools. 7) Hey, Github! Run program Good. Considerations: A) Why do commands (new line, run) need "Hey, Github!" which is pretty long and terrible to repeat all the day long (just imagine having to say Hey Joe every time we have to say a sentence to Joe, withing a long conversation with Joe) and text-to-code doesn't? B) We got a graph at the end. Now what should I do to edit the code in those 99% of cases where I got the graph wrong? An acceptable answer could be mouse and keyboard. It's a little underwhelming but voice to code already gave me the structure of the code. C) Does that mean that Microsoft and GitHub are going to know all the closed source code we'll write for our customers (there might be contractual implications) or is this something that will be self hosted in our machines?

rahulpandita 4y ago
GitHubNext here! Here is a little writeup that explains a bit more about the project https://github.com/githubnext/githubnext/tree/main/HeyGitHub https://github.com/githubnext/githubnext/tree/main/HeyGitHub Hope this is helpful :)

kajaktum 4y ago

This feels like Github expanding because it can't find anything else to do...It being a for profit organization means that it's unable to say "you know what we pretty much have everything we wanted so we're just going into a maintenance/optimization mode". This happens all the time in open source project where they simply tell their users to move elsewhere for the better alternatives but will never happen to a for profit organization.

lakomen 4y ago

Imagine sitting there, talking to your computer, and trying to get the notations right. If err unequal nil opening bracket, no no don't open the racket opening bracket... BRACKET, do you know what a bracket is No don't do a do while, delete delete. Don't delete everything... sigh Well something like that, I imagine it being a very painful experience.

3D30497420 4y ago
Or debugging. Goodness. I can only imagine.
dorkwood 4y ago
Maybe you could use little clicks and pops with your mouth to signify different characters. The computer could learn which one is which. That way instead of typing "if (int i = 0; i < count; i++)" you could say something like "if beep int i click zero boop i bop count boop i pop pop zing" This would achieve the same thing, but much faster and with less effort than typing.
- xigoi 4y ago
  If saying that is faster than typing it, you're really slow at typing.

nxpnsv 4y ago

GitHub is doing a whole lot. I think I prefer to edit my code in an editor, not on the website where it's hosted. And I think I don't want fancy AI driven code editor features using my code either. But I guess it is nice they are considering solutions for vision impaired users.

hbarka 4y ago

How does it do with SQL?

prima-facie 4y ago

Hey Github what did the previous developer actually _mean_ with this piece of legacy code?

pcj-github 4y ago

I could see it being useful for things like "goto line 42" or "rename this file as...", or very simple things like that, otherwise, I don't want the cognitive overhead of having to translate coding intent through a voice interpreter.

Cort3z 4y ago

However weird and seemingly useless this might appear to the normal programmer on here, I see this as a huge accomplishment and an incredibly important tool. Why? Accessibility. Let’s hope that I never get in a serious accident or get an disabling disease, but if I do I am not planning on giving up coding. What would you do if you lost your hands, or became permanently paralyzed. This is the tool we need to combat that. Hats off to github on this one.

bamboozled 4y ago
People who cannot see or use a keyboard already use tools like this to code. Been doing this for a long time.

eurasiantiger 4y ago

I do not want this.

deleted 4y ago

[deleted]

karmasimida 4y ago

Actually would be useful. If this is reliable I would pay to use it to some capacity, like add an argument.

birriel 4y ago

In the meantime, Talon is pretty good. You can use Vim motions and commands as you normally would, except using your voice (this applies to any editor, really): https://talonvoice.com/ https://talonvoice.com/

willjp 4y ago
Talon is exceptional, I only wish it was more natural to drive cli commands, I find I need to spell them out which I’m still quite slow at.

hdjjhhvvhga 4y ago

I'd like to see how they do with my creative variable and function names.

raidicy 4y ago

I really hope this is very easy to use. I have severe RSI and can barely surf the web. I tried using other voice to code stuff and it just hurt my voice so I'm hoping I can speak very naturally. I'm really looking forward to seeing if this can help me code again.

karmasimida 4y ago

One concern is in office space, saying things aloud is ... awkward to say the least.

univue 4y ago

sgdf

univue 4y ago

Addd some comments

Sheeny96 4y ago

I feel like this is being misunderstood - the long term view of this wouldn't be for code scribing, it'd be for non technical people to be able to instantly create things. Imagine being able to say outloud to your phone "hey, create me a view of all the weather data from the past year correlated against x and y in z view format". The code is the means, not the product.

urthor 4y ago
Exactly. The crazy thing is... this probably will work. In 20 years. But, it probably will work. There is absolutely no reason you cannot use a neural network to transcribe appropriately phrased requirements into an AST.
- BiteCode_dev 4y ago
  Before copilot, we were far from it. But now, given how magical this thing is, it opens many doors to what's possible with no code. I never really believe in anything no code before apart from Excel and RAD. But basic tasks are going to get accessible to a lot of people sooner than I expected
  - urthor 4y ago
    I'm almost certain a trained professional is still going to do a far better job than an amateur. That part is unlikely to change much, if it all. Trained professionals will always outperform the latest fellow dragged off the street.
    - BiteCode_dev 4y ago
      That doesn't matter much. You free part of the industry from needing professionals for some simple tasks, which is enough to empower users with a lot of possibilities, and focus pros on where they are really needed. "Hey phone, next time mum send me a text about voting, you can send back a 'ok boomer'?" or "hey phone, can you setup a webpage that list my tiktok videos up to last year?" Most people are not going to hire a pro for that, but we could end up with a AI general enough to be able to do that for users. It's ok if the result is not extensible, maintainable or modular.

pfd1986 4y ago

I think commenters here are -- as usual -- missing the point. This is the training ground (literally) for better models able to respond to commands like "take the CSV from me desktop, plot columns A and D and check if the KL divergence os close to zero". And from that to more complex tasks. You always need the first step and this is it. I'm bullish.

BiteCode_dev 4y ago
Exactly. Copilot is getting better everyday, because it's learning from the way we are using it.
- rahulpandita 4y ago
  GitHubNext here! We appreciate your support.
rahulpandita 4y ago
GitHubNext here! We appreciate your support.

deleted 4y ago

[deleted]

akuji1993 4y ago

export const ButtonComponent; FunctionComponent no Github no semicolon i meant colon Github backspace 5 times no backspace delete delete Github Arrrgh goddammit

cdrini 4y ago
What you're describing is more like dictation. What you'd probably say is "export the button component", and it would determine the syntax.
- akuji1993 4y ago
  Which will probably, outside of small, perfectly planned experiments, work similarly well
  - cdrini 4y ago
    You're free to speculate, but I would respectfully disagree. Extrapolating from how effective GPT3/Copilot are at interpreting text and generating code, I think it has more of a chance of working well than any other tech has ever had. I definitely hope it succeeds, anyhow!

philmander 4y ago

This is effectively a new higher level programming language without a fixed syntax. Describing more the "what", not "how", and being much closer to natural language over computer language. The voice part seems like an (albeit important) accessibility add on. I'm sure it won't be perfect but an amazing step forward in the evolution of programming languages

meowface 4y ago
I could be wrong, but I think (minus editor commands) much of this can be emulated in existing Copilot by writing a comment symbol followed by natural language. I wouldn't even be surprised if under the hood "Hey, GitHub!" is basically doing exactly that with the voice input.

WormholeCreator 4y ago

it is not practical if we have to describe each and every line. Also, imagine you are sitting in an office with other team mates - what happens if all of them talk together but are working on different projects. It will disturb others in terms of noise pollution. but it will definitely be a fun project and might work perfectly when you are working alone from home.

jenscow 4y ago

bool success equals user dot no i mean ah fuck stop stop quit

danjc 4y ago

And you thought open plan offices were bad already!

mindvirus 4y ago

This is awesome. I could see using this to write code on my phone even.

tempodox 4y ago

Imagine using this in a setting where you're not alone in the room. Imagine using this surrounded by other developers who do the same.

squarefoot 4y ago

If translation is semantic and not literally identical, chances are that the user asks for a piece of code and it outputs something that is 100% identical to code that is copyrighted elsewhere. Big "blame the AI" legal loophole waiting to happen?

falcor84 4y ago

I think this, or a future version of this, would have real potential. I'm thinking about this in terms of the navigator-pilot pair programming approach, and believe that as a senior, if it's even half-as-good as working with a fresh out of uni hire, then it could have real value. When there's a piece of code that I would like written, when I have good test cases in mind, but would prefer to offload it on someone, I could perhaps write the test cases and function signatures (maybe with the bot's help), get the bot to fill in the blanks until it passes the tests, and then give it direct feedback on how to refactor the code. I've signed up for the waiting list and am excited to try this out.

knutzui 4y ago
What you are describing is more akin to what GitHub Copilot already does. It is really good at taking a description and a function signature and producing a solution. Paired with a solid test suite it can definitely speed up development in my experience.

deleted 4y ago

[deleted]

lleontop 4y ago

We have come a long way. I remember when announcements like this one were done by companies on April 1st!

kdmytro 4y ago

This is not going to play well with open-space offices.

amarant 4y ago

Oh cool, my brother used to wish out loud something like this existed a few years back when his wrists were really killing him. He's wrists were so far gone he couldn't even type on a ergonomical keyboard for any greater duration of time, so he used to wish he could just talk instead. For me, I got a ergonomical keyboard before my wrists went bad, and so far they seem to be holding up! Moral of the story: get a good keyboard early, or you might need a tool like this one someday!

darepublic 4y ago

Execution is everything with this. I've wanted something like this so I could actually code while performing other activities or in various states of intoxication. Don't code and drive. Don't drink and code

FloatArtifact 4y ago

The challenges that remain in speech coding is not generating code as much as it is navigating through existing code or an application. There's only two ways to do this effectively and unfortunately no one has taken the true path to accessibility. The more common way is plugins/extensions to grab a information from the editor. Accessibility is more than just one editor. It's the OS and all the applications. Microsoft needs to take the hard route to make an accessibility UI automation server to grab that information and only make up the difference through plugins as needed. It's all about grabbing information from the application and generating on the fly commands, not just parsing free dictation in order to get the best accuracy. It takes a lot of expertise to make any sort of UI automation, fast and efficient for navigating and selecting text or out of focus menu items. I've fussed around and managed to get tree sitter to navigate across code. For example generic commands are like 'next function'. Code simply isn't pronounceable when it's written by others. Therefore, navigating across generic tokens is really the best method. Then other methods can be used for fine navigation if needed. My hope is that they develop a grammar system that is open source and integrates with accessibility frameworks focused on performance. I wish I could have a phone call with the development team.

skydhash 4y ago
I think an accessibility a la vim or with something like tree sitter, would help immensely like: “Top of file Down 5 lines Modify import source to … inside first class Down 5 methods Insert new method after Inside arg list Append arg named … of type … … …” And add a way to indentify types and parameters with special pronunciation.
- FloatArtifact 4y ago
  I recognize that all applications are not accessible through accessibility APIs. However, there is no high level access to accessibility APIs. There are quite a few for automated testing UI. However, none of them are performant enough for speech to code or screen readers. Testing automation frameworks don't really require high performance. Accessibility accessing the content of the application and the context is what's important. It's more important than the speech recognition backend. Speech recognition shines work best with a narrow context. (when those commands are available) The type of performance we need as a speech recognition community and screen reader community is quite high. By the beginning of speech and just before decode time information needs to be available to be parsed for navigation/editing. That way these tokens can be weighted as commands for recognition. Commands could be modeled after vim functionality though. Outside of tree sitter it would be interesting to hook into hooking into as a client a language protocol server. However, I think they only expect one client. In addition, I still see that as a lesser approach without dedicated support for high performance UI automation server for speech recognition engine to leverage.

troelsSteegin 4y ago

What if the code in question is a DSL? Something say that is syntactically python, but with a namespace defined through a narrow set of imports. This would be interesting to explore for end-user scripting.

deleted 4y ago

[deleted]

boredumb 4y ago

As someone who works remotely from home, the last thing I need is to start babbling to myself in code for 8 hours a day. I imagine that's a one way ticket to developing some sort of disorder.

silverlake 4y ago

I’m working on something similar. The target market is the 99% of people who want to program ad-hoc domain-specific problems. For example, generating charts w/o having to dig through all the data sources (Wolfram Alpha does a simple version of this). Building a financial risk model for a client’s specific request (you have to be a whiz at Excel, python or some internal ide). Even for home automation, my mom can’t use Alexa’s awful app to customize routines. I don’t think the voice part is necessary. It’s easy enough to slap ASR on the front. But going from natural language -> full problem spec -> code is hard in the general case, but doable in well-understood domains. Why can’t Scotty talk to a computer? (https://youtube.com/watch?v=hShY6xZWVGE&feature=share https://youtube.com/watch?v=hShY6xZWVGE&feature=share)

kevmo314 4y ago

Why are all the comments here so negative? Maybe typing is a hard sell, but some of the navigation stuff seems quite useful. Even being able to invoke VS Code's command palette would be really cool with this. Something like "Open Dockerfile" would be useful and maybe faster than typing.

kashanjunaid 4y ago

very intresting!

manesioz 4y ago

Interesting. I would find this annoying because its so different from what I'm used to, but the potential it has for people with disabilities is huge.

qntmfred 4y ago

see also https://serenade.ai/ https://serenade.ai/ https://news.ycombinator.com/item?id=22404264 https://news.ycombinator.com/item?id=22404264

geewee 4y ago

Having programmed and navigated my PC via voice exclusively for about 6 months, done a ton of research and written several articles about it and what options are out there [0][1], I think might be pretty ground-breaking stuff. Inputting code with voice is generally difficult, often due to variable names, casing, punctuation etc being hard to get right in voice-to-text. I think this might help quite a lot with that. _However_, some of the hardest things in voice coding isn't necessarily just the input. Navigating large codebases is hard, and particularly editing existing code can be extremely difficult, probably much more difficult than just inputting new code. I have my doubt that with the demonstration shown here, that it's able to make complex editing tasks simple, but if it does - I cannot overstate how huge of a leap forward it is. [0]: https://www.gustavwengel.dk/state-of-voice-coding-2017/ https://www.gustavwengel.dk/state-of-voice-coding-2017/ [1]: https://www.gustavwengel.dk/state-of-voice-coding-2019/ https://www.gustavwengel.dk/state-of-voice-coding-2019/

jovial_cavalier 4y ago
>Having programmed and navigated my PC via voice exclusively for about 6 months... I'm curious, why have you done this?
- FloatArtifact 4y ago
  I can't speak for his use case. However, people with medical conditions like RSI, stroke or anything that limits their action between keyboard and mouse. However, the average developer doesn't need those fine-grained navigation controls but can still benefit from enhanced input through voice. Some have mental disabilities who interface differently. Others are simply supplement their input as an average developer by voice as a preventative measure for repetitive strain RSI. The day the hope is develop something that every developer could see the value and leverage. In a way accessibility is for everyone. In general I see accessibility as a hierarchy that could benefit everyone. Accessibility APIs, close to real time OCR, Eye tracking, alternative inputs (eg pedal, touch pad, stylus) allowing for the broadest possible input and APIs to extract information from applications. Extraction of information from applications and input to applications allows the user to specialize for their use case. My experience as people will become experts in voice their command vernacular shortens as they carve out their niche use case. It goes beyond singular shortcuts too series of actions to get stuff done. However, what really means to happen is voice systems need access to the OS and to the application to really shine. That would empower not only navigation for those that are disabled but context-specific commands that are intuitive and abstracted like next function or parameter.

nicolas_lorenzi 4y ago

I imagine happiness in the open space

danwee 4y ago

Curious: In "Clean records from titanic data where age is null", how does it know that the age field is exactly `Age` and not just `age`? You cannot know this without examing the data set (the headers), so is the software inspecting the loaded CSV "on the fly" before us telling it to actually execute the code?

lkrubner 4y ago

My worst prediction ever was at the end of my book, when I struck a positive note about voice interfaces. The startup I was at in 2015 had the pitch "Let your sales people talk directly to Salesforce" and we pushed the limits of what we could do with NLP. That particular startup had spectacularly bad management and so it flamed out in a series of screaming, raging fights, which I documented here: https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/0998997617/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1668096943&sr=8-1 https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/09... But at the end of the book I struck an upbeat note, about how the technology was advancing quickly and within 3 or 4 years someone would achieve something much greater than our own limited successes. But I was wrong. 7 years later I'm surprised at how little progress there has been. I don't see any startup that's done much better than what we did in 2015. Voice interfaces remain limited in accuracy and use.

polyterative 4y ago

I have rsi, github please make it work well

dang 4y ago

This is the third such thread in the last 24 hours which consists of nothing but an elaborate waiting list signup. I've changed the titles to make that clear. GitHub Blocks – waiting list signup - https://news.ycombinator.com/item?id=33537706 https://news.ycombinator.com/item?id=33537706 - Nov 2022 (41 comments) GitHub code search – waiting list signup - https://news.ycombinator.com/item?id=33537614 https://news.ycombinator.com/item?id=33537614 - Nov 2022 (48 comments) A good HN discussion needs more than a waiting list signup. A good time to have a thread would be when something is actually available.

hintymad 4y ago

So this is a frontend of Copilot. The example of "import pandas" getting translated into "import pandas as pd" is pretty convincing, as the tool helps developers to state their intentions. On the other hand, "hey, github, a new line" kills me.

v3ss0n 4y ago

So software development houses will become call centers.