9 ms·

Hey I'm one of the Console devs who's been working on this feature for a while now. I'll be hanging around in the comments for a little while to try and answer

by zadjii 8y ago

Hey I'm one of the Console devs who's been working on this feature for a while now. I'll be hanging around in the comments for a little while to try and answer any questions that people might have. TL;DR of this announcement: We've added a new pseudoconsole feature to the Windows Console that will the people create "Terminal" applications on Windows very similarly to how they work on *nix. Terminals will be able to interact with the conpty using only a stream of characters, while commandline applications will be able to keep using the entire console API surface as they always have.

jamesgeck0 8y ago
This is awesome. Thank you for all ya'll's hard work!
llampx 8y ago
Will this replace Command Prompt or Powershell? Or is this more the back-end for these apps? I believe that we are heading for console overload (in a good way?!) with Command Prompt and Powershell installed on every computer and Debian/Ubuntu available on the MS store.
- nailer 8y ago
  The back end. Basically people like ConEmu and Hyper and Terminus have been having to use various unreliable hacks for ages because there was no real console API. Now there is one.
- zadjii 8y ago
  First off: command prompt (cmd.exe) and powershell are commandline client applications. They are shells just the same as bash is. All commandline clients run attached to a console server, and that server is conhost.exe. Conhost is responsible not only for being the console server, but drawing the actual terminal window these apps run in. So when you alunch cmd or powershell, what you're seeing is conhost.exe "hosting" these console applications. What we're exposing here is the "master side" of conhost, which will the other applications act as Terminals, like how there is gnome-terminal, xterm, terminator, etc on linux.
  - paulie_a 8y ago
    Is there any chance cmd and powershell will improve from a user interface perspective? And perhaps become usable? Cmd has been garbage since it's inception.
    - zadjii 8y ago
      cmd is parked for pretty much everything except for major issues. It's a scary codebase that has a LOT of code that's dependent on it, and we can't really add any new features there without the possibility of breaking someone. This feature is mostly focused on the other end of the communication, on being able to create new Terminal windows to run shells inside of them.
      - paulie_a 8y ago
        Is there a chance this will be connected to a functional shell interface. I get your point that cmd cannot be upgraded because of legacy issues and that is understandable and unfortunate, but windows needs a proper shell. This is obviously a great start for one side of the equation. But until there is a decent terminal app, windows will continue to be a nonstarter.
        zadjii 8y ago
        It sounds like you're asking for two different things here: cmd.exe is a shell, and that's the guy that's parked. conhost.exe is a terminal, and that's under active development, though it's slower than something like VsCode, because we can't just go adding features as we see fit, we have a LOT of back compat we still need to support. Fortunately, conpty will allow for the creation of new terminal applications on Windows. If you're looking for a better shell experience on windows, I can point you to powershell or even [yori](http://www.malsmith.net/yori/ http://www.malsmith.net/yori/), which looks pretty cool
        cryptonector 8y ago
        I think u/paulie_a is just asking for a better shell. If cmd can't be made better, then make a new one.
nailer 8y ago
Zadji I love your work. Which build is this going to land in? Do you know if any of the third party console apps using the new API yet?
- zadjii 8y ago
  It's already available in current insider's builds, and will be landing officially in the next available Windows release some time later this year. We're still working with ConEmu, VsCode, and OpenSSH to get them all over to the new API, with varying levels of adoption in the next few months likely. Currently, WSL is also using the same functionality, if you open a WSL distro and run any Windows executables (eg `cmd.exe`), they'll run attached to a conpty. I use this as my daily driver.
  - cryptonector 8y ago
    Out of curiosity, are you backporting onto Windows 10, or is Windows 10 the only release vehicle for everything in master? If the latter, how are you releasing piecemeal?
    - zadjii 8y ago
      Pretty much any new features we release are only available on new Windows 10 releases. Unless there's a gigantic demand for a feature, or business impact, we're not really capable on our team of backporting anything.
      - cryptonector 8y ago
        What I meant is: are you building a Windows 11, or is Windows 10 the only release vehicle? I'm trying to imagine you pushing to a master branch to integrate into what becomes the insider builds, and then... either those become a future release of Windows 10, or you backport onto a Windows 10 branch. Neither option would be very appealing to me if it was up to me.
        8y ago
zokier 8y ago
While it is nice that MS is focusing on console and command line now, it seems to me that you are mostly working on improving compatibility with legacy UNIXy stuff. Do you have some vision or plans to go well beyond the classic UNIXy style of console and command line? I'm thinking in the lines of projects like DomTerm http://domterm.org/ http://domterm.org/ which could have nice interactions with e.g. PowerShell.
- cryptonector 8y ago
  The ConPTY is not just about compatibility with nix. It's about proper remoting of consoles. Unix got that right / Windows got that wrong, and now Windows will finally get it right too -- that it helps nix compat seems like a happy accident (though obviously they want that too, so not so accidental.
  - quotemstr 8y ago
    Right. For a long time, the MS remoting philosophy was that applications should be remoted, not text streams. The stance goes all the way back to DCOM. That's why PowerShell remoting looks more like a local PowerShell executing commands on a remote machine than it looks like you just connecting to a PowerShell running elsewhere. The difference is important, since in the traditional MS model, each program that wants to do the remote thing needs to essentially implement its own client-server setup, albeit with a massive amount of help from various runtimes. Named pipes and central authentication made this approach not quite as horrible as it sounds. This new API is a departure from this model. It will make it possible to just remote via text streams. Perhaps that's uglier --- everyone knows in-band signaling is fragile. But long experience shows they just remoting the damn text streams is easily the more pragmatic option.
    - cryptonector 8y ago
      I mean, it could be a packet stream where some packets are control packets, and then the Console API could be used and remoted, but you'd need clients that understand it. The fragility of in-band signaling in the TTY world is not *nix's fault or anything. VMS had that too, since VMS too had to deal with TTYs. The fact is that a) the system evolved from real, hardware TTYs of the 1970s, b) using a text stream with some in-band signaling doesn't even half suck -- mostly it rocks, and all you have to do for it to rock is get the right $TERM value and not output binary files to the tty.
rogerbinns 8y ago
Have you given thought on how to solve the "unwanted console" problem? For example if you run a .py file (Python) under Windows then you get a console. That is fine for command line stuff, but beyond annoying if the file displays as a gui. So there are now two Python binaries - python.exe and pythonw.exe. The only difference is the latter ensures no console appears. Also good luck if the script printed a help message since often the console disappears before you even know that happened. I presume many tools deal with this issue, and do it in different ways. Perhaps it is as simple as making the console itself only appear once there is any output, or a blocking read of input.
- zadjii 8y ago
  Technically, any executable that's compiled as a commandline application is going to get a console allocated for it, no matter what on Windows. I don't believe that's something we can fix retroactively unfortunately, that's just a part of how things have to be. Now, I believe that python could have python.exe compiled as a win32 application, then call AllocateConsole as soon as the script called print() or something. If the app was already running in a console, I believe (don't quote me) that AllocateConsole won't allocate a new console for it, but if it doesn't yet have a console it'll spawn one.
  - cryptonector 8y ago
    Why... can't you have I/O redirection to the null device and have it understand (and throw away) Console API messages? You might as well also add some pseudo-device to convert Console API messages into Unix-style text streams (with or without metadata converted to terminal control sequences), so that one could redirect console programs' output to files / pipes. When the user's (programmer's) intent is to run a program with no console window, then that's what they should get: no console window.
  - rossy 8y ago
    I think the behaviour of cmd.exe is part of the problem here. When an interactive cmd.exe launches a console-subsystem app, it waits for the process to finish before showing the prompt again, but when it launches a GUI-subsystem app, cmd.exe writes the prompt again immediately, so even if the new process calls AttachConsole(ATTACH_PARENT_PROCESS) before it tries to write to the console, it will write over cmd.exe's prompt, which makes a poor user experience. So, if someone wants to make a "dual-mode" app that works as a win32-subsystem app when launched from Explorer and a console-subsystem app when launched from a console, they have to choose between two bad options. They can make their app a console-subsystem app, which means a console will always briefly appear on screen when the app is started (no matter how quickly the app calls FreeConsole(),) or they can make their app a GUI-subsystem app (that opportunistically calls AttachConsole(),) which behaves sub-optimally in cmd.exe. Maybe the solution is to add a flag (in the .manifest file?) that makes the console initially hidden for a console-subsystem app. That would prevent the brief appearance of a console window when launching a console-subsystem app from Explorer. Then there would be no need for pythonw.exe and python.exe could show the console window only after a message is printed.
berbec 8y ago
Will there be the ability to disown, background, nohup processes and close the console, leaving the commands running?
- zadjii 8y ago
  Those sound like they'll be more like the responsibility of the terminal emulator, unfortunately. Windows console applications aren't really able to live without being attached to a console. Now, a terminal might be able to implement those features... actually now you've got me thinking. I'll play around with that idea. Definitely non-committal, but it might be possible in the future.
  - cryptonector 8y ago
    I mean, the same is true on Unix. If an app is in the background and wants to write to (or read from) the terminal, it gets SIGTTOU (SIGTTIN) sent to it immediately. It might have to be impossible on Windows to ignore SIGTTOU/SIGTTIN/SIGTSTOP, but I think that's just fine. Mind you, ptys + tmux/similar is certainly very good, and if that's all we'll get that's still way way better than the current state of affairs, but if that's all that will be possible it should at least be possible to pause the console's output (and flow-control the console application).
  - JdeBP 8y ago
    The sad thing is that this was all already implemented and done in Microsoft's second POSIX subsystem for Windows NT. It provided signals and process groups support for job control shells. It had a full control sequence interpreter for output and control sequence generator for extended keys. There were termcap/terminfo database records that people had added to other operating systems. It had a line discipline with "canonical" and "raw" modes. It had pseudo-terminals, with both BSD and System 5 access semantics. * https://technet.microsoft.com/en-gb/library/bb497016.aspx https://technet.microsoft.com/en-gb/library/bb497016.aspx * https://technet.microsoft.com/en-gb/library/bb463219.aspx https://technet.microsoft.com/en-gb/library/bb463219.aspx * https://news.ycombinator.com/item?id=12866843 https://news.ycombinator.com/item?id=12866843 * http://jdebp.info./FGA/interix-terminal-type.html http://jdebp.info./FGA/interix-terminal-type.html And Microsoft owns it.
quotemstr 8y ago
Can conhost still do anything that users of the new API can't?
- zadjii 8y ago
  Excellent question! There are a few limitations that we have to place on the ConPty to make it work quite right. Primarily, client apps running attached to the conpty will not be able to have separate viewport and buffer sizes. On *nix, the entire "console buffer" is just the size of the window, but on Windows, technically, the buffer is much larger than the window. (as an example, when you open up a command prompt, there's a giant empty space at the bottom if you scroll down). Fortunately, we haven't came across any apps that _need_ the buffer to be a different size than the viewport, and it's a technically valid console configuration, so apps should have been able to support it before. Input is also tricky - VT doesn't let you express input with as much fidelity as a console app might be expecting, though this we're working on a solution for :)
  - quotemstr 8y ago
    Now Windows just needs to ship with a decent pager. :-) Programmatic access to scroll back is useful for a few things. For example, back when I was on Windows Phone, I wrote a compiler wrapper that would scroll back to the first error message. It'd be nice for the POSIX terminal world to standardize on similar scrollback access. I know the zsh people would love it.
  - caf 8y ago
    *nix consoles typically have two buffers - many full-screen temrinal applications switch to the "alternate screen" on start and back to the principal screen on exit. That's why when you exit vim(1), you see the terminal state back as it was before you started it. Will ConPty support this?
    - cryptonector 8y ago
      I hate the alternate screen with a passion.
theclaw 8y ago
I'm unclear as to whether I'll be able to pipe binary data between a classic console application and a ConPTY application due to the VT translation and rendering components in ConHost. So, for example if I was to pipe into 7z.exe, a classic console app, using something like "type mybinaryfile.bin | 7z.exe a -si c:\temp\myarchive.7z" from a ConPTY console, would the VT translation affect the piped stream?
- zadjii 8y ago
  Nope! We're only rendering the effect of any attached processes to the VT on the conpty side of things. On the client side (where cmd, type, 7z.exe are all running), they're going to keep working just the same as they always have. They're all running on the "slave" side of conpty, while the emitted VT is coming from the "master" side of the conpty.

by zadjii 8y ago

jamesgeck0 8y ago

This is awesome. Thank you for all ya'll's hard work!

llampx 8y ago

Will this replace Command Prompt or Powershell? Or is this more the back-end for these apps? I believe that we are heading for console overload (in a good way?!) with Command Prompt and Powershell installed on every computer and Debian/Ubuntu available on the MS store.

nailer 8y ago
The back end. Basically people like ConEmu and Hyper and Terminus have been having to use various unreliable hacks for ages because there was no real console API. Now there is one.
zadjii 8y ago
First off: command prompt (cmd.exe) and powershell are commandline client applications. They are shells just the same as bash is. All commandline clients run attached to a console server, and that server is conhost.exe. Conhost is responsible not only for being the console server, but drawing the actual terminal window these apps run in. So when you alunch cmd or powershell, what you're seeing is conhost.exe "hosting" these console applications. What we're exposing here is the "master side" of conhost, which will the other applications act as Terminals, like how there is gnome-terminal, xterm, terminator, etc on linux.
- paulie_a 8y ago
  Is there any chance cmd and powershell will improve from a user interface perspective? And perhaps become usable? Cmd has been garbage since it's inception.
  - zadjii 8y ago
    cmd is parked for pretty much everything except for major issues. It's a scary codebase that has a LOT of code that's dependent on it, and we can't really add any new features there without the possibility of breaking someone. This feature is mostly focused on the other end of the communication, on being able to create new Terminal windows to run shells inside of them.
    - paulie_a 8y ago
      Is there a chance this will be connected to a functional shell interface. I get your point that cmd cannot be upgraded because of legacy issues and that is understandable and unfortunate, but windows needs a proper shell. This is obviously a great start for one side of the equation. But until there is a decent terminal app, windows will continue to be a nonstarter.
      - zadjii 8y ago
        It sounds like you're asking for two different things here: cmd.exe is a shell, and that's the guy that's parked. conhost.exe is a terminal, and that's under active development, though it's slower than something like VsCode, because we can't just go adding features as we see fit, we have a LOT of back compat we still need to support. Fortunately, conpty will allow for the creation of new terminal applications on Windows. If you're looking for a better shell experience on windows, I can point you to powershell or even [yori](http://www.malsmith.net/yori/ http://www.malsmith.net/yori/), which looks pretty cool
        cryptonector 8y ago
        I think u/paulie_a is just asking for a better shell. If cmd can't be made better, then make a new one.

nailer 8y ago

Zadji I love your work. Which build is this going to land in? Do you know if any of the third party console apps using the new API yet?

zadjii 8y ago
It's already available in current insider's builds, and will be landing officially in the next available Windows release some time later this year. We're still working with ConEmu, VsCode, and OpenSSH to get them all over to the new API, with varying levels of adoption in the next few months likely. Currently, WSL is also using the same functionality, if you open a WSL distro and run any Windows executables (eg `cmd.exe`), they'll run attached to a conpty. I use this as my daily driver.
- cryptonector 8y ago
  Out of curiosity, are you backporting onto Windows 10, or is Windows 10 the only release vehicle for everything in master? If the latter, how are you releasing piecemeal?
  - zadjii 8y ago
    Pretty much any new features we release are only available on new Windows 10 releases. Unless there's a gigantic demand for a feature, or business impact, we're not really capable on our team of backporting anything.
    - cryptonector 8y ago
      What I meant is: are you building a Windows 11, or is Windows 10 the only release vehicle? I'm trying to imagine you pushing to a master branch to integrate into what becomes the insider builds, and then... either those become a future release of Windows 10, or you backport onto a Windows 10 branch. Neither option would be very appealing to me if it was up to me.
      - 8y ago

zokier 8y ago

While it is nice that MS is focusing on console and command line now, it seems to me that you are mostly working on improving compatibility with legacy UNIXy stuff. Do you have some vision or plans to go well beyond the classic UNIXy style of console and command line? I'm thinking in the lines of projects like DomTerm http://domterm.org/ http://domterm.org/ which could have nice interactions with e.g. PowerShell.

cryptonector 8y ago
The ConPTY is not just about compatibility with nix. It's about proper remoting of consoles. Unix got that right / Windows got that wrong, and now Windows will finally get it right too -- that it helps nix compat seems like a happy accident (though obviously they want that too, so not so accidental.
- quotemstr 8y ago
  Right. For a long time, the MS remoting philosophy was that applications should be remoted, not text streams. The stance goes all the way back to DCOM. That's why PowerShell remoting looks more like a local PowerShell executing commands on a remote machine than it looks like you just connecting to a PowerShell running elsewhere. The difference is important, since in the traditional MS model, each program that wants to do the remote thing needs to essentially implement its own client-server setup, albeit with a massive amount of help from various runtimes. Named pipes and central authentication made this approach not quite as horrible as it sounds. This new API is a departure from this model. It will make it possible to just remote via text streams. Perhaps that's uglier --- everyone knows in-band signaling is fragile. But long experience shows they just remoting the damn text streams is easily the more pragmatic option.
  - cryptonector 8y ago
    I mean, it could be a packet stream where some packets are control packets, and then the Console API could be used and remoted, but you'd need clients that understand it. The fragility of in-band signaling in the TTY world is not *nix's fault or anything. VMS had that too, since VMS too had to deal with TTYs. The fact is that a) the system evolved from real, hardware TTYs of the 1970s, b) using a text stream with some in-band signaling doesn't even half suck -- mostly it rocks, and all you have to do for it to rock is get the right $TERM value and not output binary files to the tty.

rogerbinns 8y ago

Have you given thought on how to solve the "unwanted console" problem? For example if you run a .py file (Python) under Windows then you get a console. That is fine for command line stuff, but beyond annoying if the file displays as a gui. So there are now two Python binaries - python.exe and pythonw.exe. The only difference is the latter ensures no console appears. Also good luck if the script printed a help message since often the console disappears before you even know that happened. I presume many tools deal with this issue, and do it in different ways. Perhaps it is as simple as making the console itself only appear once there is any output, or a blocking read of input.

zadjii 8y ago
Technically, any executable that's compiled as a commandline application is going to get a console allocated for it, no matter what on Windows. I don't believe that's something we can fix retroactively unfortunately, that's just a part of how things have to be. Now, I believe that python could have python.exe compiled as a win32 application, then call AllocateConsole as soon as the script called print() or something. If the app was already running in a console, I believe (don't quote me) that AllocateConsole won't allocate a new console for it, but if it doesn't yet have a console it'll spawn one.
- cryptonector 8y ago
  Why... can't you have I/O redirection to the null device and have it understand (and throw away) Console API messages? You might as well also add some pseudo-device to convert Console API messages into Unix-style text streams (with or without metadata converted to terminal control sequences), so that one could redirect console programs' output to files / pipes. When the user's (programmer's) intent is to run a program with no console window, then that's what they should get: no console window.
- rossy 8y ago
  I think the behaviour of cmd.exe is part of the problem here. When an interactive cmd.exe launches a console-subsystem app, it waits for the process to finish before showing the prompt again, but when it launches a GUI-subsystem app, cmd.exe writes the prompt again immediately, so even if the new process calls AttachConsole(ATTACH_PARENT_PROCESS) before it tries to write to the console, it will write over cmd.exe's prompt, which makes a poor user experience. So, if someone wants to make a "dual-mode" app that works as a win32-subsystem app when launched from Explorer and a console-subsystem app when launched from a console, they have to choose between two bad options. They can make their app a console-subsystem app, which means a console will always briefly appear on screen when the app is started (no matter how quickly the app calls FreeConsole(),) or they can make their app a GUI-subsystem app (that opportunistically calls AttachConsole(),) which behaves sub-optimally in cmd.exe. Maybe the solution is to add a flag (in the .manifest file?) that makes the console initially hidden for a console-subsystem app. That would prevent the brief appearance of a console window when launching a console-subsystem app from Explorer. Then there would be no need for pythonw.exe and python.exe could show the console window only after a message is printed.

berbec 8y ago

Will there be the ability to disown, background, nohup processes and close the console, leaving the commands running?

zadjii 8y ago
Those sound like they'll be more like the responsibility of the terminal emulator, unfortunately. Windows console applications aren't really able to live without being attached to a console. Now, a terminal might be able to implement those features... actually now you've got me thinking. I'll play around with that idea. Definitely non-committal, but it might be possible in the future.
- cryptonector 8y ago
  I mean, the same is true on Unix. If an app is in the background and wants to write to (or read from) the terminal, it gets SIGTTOU (SIGTTIN) sent to it immediately. It might have to be impossible on Windows to ignore SIGTTOU/SIGTTIN/SIGTSTOP, but I think that's just fine. Mind you, ptys + tmux/similar is certainly very good, and if that's all we'll get that's still way way better than the current state of affairs, but if that's all that will be possible it should at least be possible to pause the console's output (and flow-control the console application).
- JdeBP 8y ago
  The sad thing is that this was all already implemented and done in Microsoft's second POSIX subsystem for Windows NT. It provided signals and process groups support for job control shells. It had a full control sequence interpreter for output and control sequence generator for extended keys. There were termcap/terminfo database records that people had added to other operating systems. It had a line discipline with "canonical" and "raw" modes. It had pseudo-terminals, with both BSD and System 5 access semantics. * https://technet.microsoft.com/en-gb/library/bb497016.aspx https://technet.microsoft.com/en-gb/library/bb497016.aspx * https://technet.microsoft.com/en-gb/library/bb463219.aspx https://technet.microsoft.com/en-gb/library/bb463219.aspx * https://news.ycombinator.com/item?id=12866843 https://news.ycombinator.com/item?id=12866843 * http://jdebp.info./FGA/interix-terminal-type.html http://jdebp.info./FGA/interix-terminal-type.html And Microsoft owns it.

quotemstr 8y ago

Can conhost still do anything that users of the new API can't?

zadjii 8y ago
Excellent question! There are a few limitations that we have to place on the ConPty to make it work quite right. Primarily, client apps running attached to the conpty will not be able to have separate viewport and buffer sizes. On *nix, the entire "console buffer" is just the size of the window, but on Windows, technically, the buffer is much larger than the window. (as an example, when you open up a command prompt, there's a giant empty space at the bottom if you scroll down). Fortunately, we haven't came across any apps that _need_ the buffer to be a different size than the viewport, and it's a technically valid console configuration, so apps should have been able to support it before. Input is also tricky - VT doesn't let you express input with as much fidelity as a console app might be expecting, though this we're working on a solution for :)
- quotemstr 8y ago
  Now Windows just needs to ship with a decent pager. :-) Programmatic access to scroll back is useful for a few things. For example, back when I was on Windows Phone, I wrote a compiler wrapper that would scroll back to the first error message. It'd be nice for the POSIX terminal world to standardize on similar scrollback access. I know the zsh people would love it.
- caf 8y ago
  *nix consoles typically have two buffers - many full-screen temrinal applications switch to the "alternate screen" on start and back to the principal screen on exit. That's why when you exit vim(1), you see the terminal state back as it was before you started it. Will ConPty support this?
  - cryptonector 8y ago
    I hate the alternate screen with a passion.

theclaw 8y ago

I'm unclear as to whether I'll be able to pipe binary data between a classic console application and a ConPTY application due to the VT translation and rendering components in ConHost. So, for example if I was to pipe into 7z.exe, a classic console app, using something like "type mybinaryfile.bin | 7z.exe a -si c:\temp\myarchive.7z" from a ConPTY console, would the VT translation affect the piped stream?

zadjii 8y ago
Nope! We're only rendering the effect of any attached processes to the VT on the conpty side of things. On the client side (where cmd, type, 7z.exe are all running), they're going to keep working just the same as they always have. They're all running on the "slave" side of conpty, while the emitted VT is coming from the "master" side of the conpty.