7 ms·

Windows really needs a public psuedoconsole API. For the uninitiated: cmd.exe is the shell, like bash. conhost.exe is the terminal emulator, like xterm. bash o

Windows really needs a public psuedoconsole API. For the uninitiated: cmd.exe is the shell, like bash. conhost.exe is the terminal emulator, like xterm. bash or cmd (the shell) talks to the terminal via its standard input, output, and error file handles. Terminal handles are like full-duplex pipes, but richer: they transmit all the information needed for interaction with the user. Each terminal handle has two ends: master* and slave. The slave is the terminal to which cmd or bash talks; it basically emulates a teletype. The master handle allows for reading both in-band and out-of-band information from the slave terminal handle and supports special APIs that allow anyone with access to the master handle to fully emulate a terminal. POSIX systems allow anyone to create master and slave terminal handles. When you start xterm, xterm calls openpty, gets master and slave handles, starts your shell with the slave handle, and uses the master terminal to get the information needed to draw characters. There's no equivalent for Windows. The only Windows program that can call the Windows equivalent of openpty is conhost, which ships with the operating system. It's not possible for a different program to make a master-side terminal handle. There are two workarounds, which both suck: 1) use pipes, not console handles, and 2) make a hidden conhost.exe and scrape it. The former causes programs to think they're being run non-interactively, with all sorts of negative side effects (like excessive buffering). The latter loses screen updates, is inefficient, and is generally awful. Microsoft could easily make this situation much better by opening the protocol that conhost uses to communicate with the kernel. If any program can create a console handle, it's much easier to implement things like ssh daemons, Cygwin, and terminal emulators that suck less than conhost. ... * As an aside: when bash talks to xterm, it uses in-band signaling (mostly: see termios): it sends control code sequences, which xterm understands as "move cursor to 3,3" and "set color to green". By contrast, when cmd communicates to conhost, it uses out-of-band signaling: it uses special API functions to move the cursor and set the text color.

bpye 10y ago
Is there any historical reason that conhost is the only executable able to open a master handle?
- quotemstr 10y ago
  Well, originally, console handles weren't real handles. They were userspace hacks, which is why DuplicateHandle[1] has special cases for console handles. (Imagine using negative file descriptor values that libc understood.) Originally, the terminal emulator ran as SYSTEM and was spawned from (IIRC) csrss (which is special, because reasons). In Vista, with integrity levels introduced, the terminal emulator had to run in a different security context, and so wasn't themed. (To prevent shatter attacks.) In Windows 7 and 8, Microsoft developed a real, honest-to-god pseudoconsole API, with a real protocol. In this implementation, console handles are real handles and conhost is just a normal program that speaks a protocol. IIRC, the only thing at this point stopping Microsoft creating a pseudoconsole API based on real, honest-to-god kernel HANDLEs is a willingness to externalize, document, and support what has been until now a Windows-internal API. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms7...
  - dman 10y ago
    How do you know all this? :)
    - quotemstr 10y ago
      Three years on Windows and Windows Phone. I sometimes miss those days. :-)
  - dfox 10y ago
    On Vista, conhost.also ran in user context and console windows were themed, on XP they was not. I believe that conhost.exe was introduced in Vista together with new desktop separation mechanism and on earlier version all the console window logic was internal to csrss (which in original NT design was essentially display server and window manager and placing console emulation in there makes some kind of sense given the NT's model of that).
    - quotemstr 10y ago
majkinetor 10y ago
Conemu doesnt suck and it uses method 2. It has few quirks but nothing serious and is almost on pair with usability as tmux and in some aspects even better.
- _ix 10y ago
  I use conemu at work. It's not as awful as some alternatives. I would recommend it, but I can't say that it's a good experience. Maybe I'm doing something wrong here... do you run vim in conemu? For me, it's slow when redrawing lines of text while paging and sometimes renders artifacts in the pane, especially when I vertically split vim. For a command line interface, I find it to be fairly unstable, too.
  - vishbar 10y ago
    I use vim in cmder occasionally. The only way I've been able to use it properly is to disable conemu hooks before I execute it. Basically, I've defined a function my profile to: * Save the ConEmuHooks environment variable value * Set it to "Off" * Execute vim * Reset it to the previous value I alias this command to "vim". It's not pretty at all, and it caused me to use Sublime or VSCode most of the time, but it worked for me. More info here: http://conemu.github.io/en/ConEmuEnvironment.html http://conemu.github.io/en/ConEmuEnvironment.html
    - majkinetor 10y ago
      Why would you do that when gvim is available and is ways better experience then vim ? For the sake of being shell purist ? On remote shell ?
    - _ix 10y ago
      I'll give this a shot, but I'm unclear how the hooks affect vim or other processes-does this address all of my vim+conemu problems? Thank you, in any event.
  - quotemstr 10y ago
    Have you tried Cygwin vim in mintty? Also, Emacs has a mode where it'll run as a Cygwin process, but use native win32 widgets. I've found using it to be the most pleasant way of interacting with Windows systems. When I was at Microsoft, I joked that I wasn't so much running "Windows" as GNU/NT. :-)
partycoder 10y ago
Forking a process on Windows is really slow, making most opensource toolchains sluggish on Windows. The UNIX philosophy is that everything is a file, so you can read and write from files and pass data around with convenient tools... that mindset is also missing in Windows. Command lines were not foreseen as a preferred way of interacting with the system in Windows until recently. PowerShell tried to fill some gaps, and many tools integrate with it... but it's not the same. It has a long way to go.
- nimish 10y ago
  Fork doesnt make a lot of sense on win32 processes (how do you handle a forked message loop?) so the functionality isn't exposed, though nt native processes can do it. The best you can do is a spawn equivalent with manual copying of structures so you lose the CoW fork semantics that allow for speed on Unix. For Linux subsystem processes fork works pretty well, but they are different animals
  - quotemstr 10y ago
    > Fork doesnt make a lot of sense on win32 processes (how do you handle a forked message loop? Message loops are per-thread and per-HWND (which are thread-affine) anyway. One way of making fork "make sense" in win32 processes is to say "Okay, you can fork, but your new process begins execution in a new thread, and none of the old threads survive." New thread means new message loop. You could even do things like automatically tear down single-threaded COM apartments in the child process.
- nimish 10y ago
  Also, on Unix everything is a fd until it isn't, like timers, signals, semaphores , pids...(and sockets have their own peculiar API). Pids not being fds leads to a really annoying issue of basically not being able to easily implement a robust way of handling child exits NT is more Unix than Unix in a lot of ways, where everything is an object (more or less) Plan 9 though really ran with the whole file thing
  - 10y ago
anonymfus 10y ago
There is a third way: you can create PowerShell Host application with the same .Net API that PowerShell ISE uses.
- quotemstr 10y ago
  I tried that --- specifically, I made a readline-powered psh host for my own use. There are a few problems in practice: 1) readline hardcodes the backslash in a bunch of places, especially in history expansion. Psh needs '^', not '\'. 2) the psh hosting APIs aren't ready: powershell.exe uses a few internal APIs to which other hosts don't have access, and without these APIs, psh does helpful things like wrap lines at 80 characters and paginates output. I also recall there being some problem with psh's completion. Still, it's amazing that embedding the CLR and posh inside a Cygwin process works at all.

bpye 10y ago

Is there any historical reason that conhost is the only executable able to open a master handle?

quotemstr 10y ago
Well, originally, console handles weren't real handles. They were userspace hacks, which is why DuplicateHandle[1] has special cases for console handles. (Imagine using negative file descriptor values that libc understood.) Originally, the terminal emulator ran as SYSTEM and was spawned from (IIRC) csrss (which is special, because reasons). In Vista, with integrity levels introduced, the terminal emulator had to run in a different security context, and so wasn't themed. (To prevent shatter attacks.) In Windows 7 and 8, Microsoft developed a real, honest-to-god pseudoconsole API, with a real protocol. In this implementation, console handles are real handles and conhost is just a normal program that speaks a protocol. IIRC, the only thing at this point stopping Microsoft creating a pseudoconsole API based on real, honest-to-god kernel HANDLEs is a willingness to externalize, document, and support what has been until now a Windows-internal API. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms7...
- dman 10y ago
  How do you know all this? :)
  - quotemstr 10y ago
    Three years on Windows and Windows Phone. I sometimes miss those days. :-)
- dfox 10y ago
  On Vista, conhost.also ran in user context and console windows were themed, on XP they was not. I believe that conhost.exe was introduced in Vista together with new desktop separation mechanism and on earlier version all the console window logic was internal to csrss (which in original NT design was essentially display server and window manager and placing console emulation in there makes some kind of sense given the NT's model of that).
  - quotemstr 10y ago

majkinetor 10y ago

Conemu doesnt suck and it uses method 2. It has few quirks but nothing serious and is almost on pair with usability as tmux and in some aspects even better.

_ix 10y ago
I use conemu at work. It's not as awful as some alternatives. I would recommend it, but I can't say that it's a good experience. Maybe I'm doing something wrong here... do you run vim in conemu? For me, it's slow when redrawing lines of text while paging and sometimes renders artifacts in the pane, especially when I vertically split vim. For a command line interface, I find it to be fairly unstable, too.
- vishbar 10y ago
  I use vim in cmder occasionally. The only way I've been able to use it properly is to disable conemu hooks before I execute it. Basically, I've defined a function my profile to: * Save the ConEmuHooks environment variable value * Set it to "Off" * Execute vim * Reset it to the previous value I alias this command to "vim". It's not pretty at all, and it caused me to use Sublime or VSCode most of the time, but it worked for me. More info here: http://conemu.github.io/en/ConEmuEnvironment.html http://conemu.github.io/en/ConEmuEnvironment.html
  - majkinetor 10y ago
    Why would you do that when gvim is available and is ways better experience then vim ? For the sake of being shell purist ? On remote shell ?
  - _ix 10y ago
    I'll give this a shot, but I'm unclear how the hooks affect vim or other processes-does this address all of my vim+conemu problems? Thank you, in any event.
- quotemstr 10y ago
  Have you tried Cygwin vim in mintty? Also, Emacs has a mode where it'll run as a Cygwin process, but use native win32 widgets. I've found using it to be the most pleasant way of interacting with Windows systems. When I was at Microsoft, I joked that I wasn't so much running "Windows" as GNU/NT. :-)

partycoder 10y ago

Forking a process on Windows is really slow, making most opensource toolchains sluggish on Windows. The UNIX philosophy is that everything is a file, so you can read and write from files and pass data around with convenient tools... that mindset is also missing in Windows. Command lines were not foreseen as a preferred way of interacting with the system in Windows until recently. PowerShell tried to fill some gaps, and many tools integrate with it... but it's not the same. It has a long way to go.

nimish 10y ago
Fork doesnt make a lot of sense on win32 processes (how do you handle a forked message loop?) so the functionality isn't exposed, though nt native processes can do it. The best you can do is a spawn equivalent with manual copying of structures so you lose the CoW fork semantics that allow for speed on Unix. For Linux subsystem processes fork works pretty well, but they are different animals
- quotemstr 10y ago
  > Fork doesnt make a lot of sense on win32 processes (how do you handle a forked message loop? Message loops are per-thread and per-HWND (which are thread-affine) anyway. One way of making fork "make sense" in win32 processes is to say "Okay, you can fork, but your new process begins execution in a new thread, and none of the old threads survive." New thread means new message loop. You could even do things like automatically tear down single-threaded COM apartments in the child process.
nimish 10y ago
Also, on Unix everything is a fd until it isn't, like timers, signals, semaphores , pids...(and sockets have their own peculiar API). Pids not being fds leads to a really annoying issue of basically not being able to easily implement a robust way of handling child exits NT is more Unix than Unix in a lot of ways, where everything is an object (more or less) Plan 9 though really ran with the whole file thing
- 10y ago

anonymfus 10y ago

There is a third way: you can create PowerShell Host application with the same .Net API that PowerShell ISE uses.

quotemstr 10y ago
I tried that --- specifically, I made a readline-powered psh host for my own use. There are a few problems in practice: 1) readline hardcodes the backslash in a bunch of places, especially in history expansion. Psh needs '^', not '\'. 2) the psh hosting APIs aren't ready: powershell.exe uses a few internal APIs to which other hosts don't have access, and without these APIs, psh does helpful things like wrap lines at 80 characters and paginates output. I also recall there being some problem with psh's completion. Still, it's amazing that embedding the CLR and posh inside a Cygwin process works at all.