5 ms·

I hope this issue is the motivator Microsoft needs to finally do something about their awful printing stack. Aside from taking it out of SYSTEM scope(!), the wh

I hope this issue is the motivator Microsoft needs to finally do something about their awful printing stack. Aside from taking it out of SYSTEM scope(!), the whole thing needs and has needed substantial modernization for a long time now (and hiding parts of it using the new Settings app is a very weak abstraction). Microsoft like a lot of big techs needs to start rewarding people for maintaining/updating legacy stuff, rather than promoting/rewarding people for new shinies while the technical debt grows out of control. The printer stack which has barely received an update since Windows XP is a perfect example.

zinekeller 5y ago
I mean, they seriously done many things in Windows 10 (and analogous servers) to improve security without too much breakage, like moving font processing from kernel-level to user-level (n.b. in the process, they have thrown Adobe code). Print spooler is tricky though: even if the official design of the API theoretically allows it to run user-level, ask users of HP products (including business ones) how many times their computer crash due to weirdness in HP drivers. (Why does the HP printing drivers run in kernel-level mode? I don't know!) I'm pretty sure that nothing will happen when the spooler is moved to user-side processing, right? (In all seriousness, the drivers will break and probably crash the system. These products are officially out-of-support, so short of Microsoft paying HP to revamp the drivers nothing will happen.) In all probability, probably even Microsoft is contemplating such move but breaking a lot of stuff is reputationally damaging to them, even if it's not really their fault in this case. Removing SafeROM is easier because it's just grumbling gamers who can probably patch up Windows 7, but imagine the support calls when the print spooler, a critical component, is moved. Even when a rational mind would put the blame on HP (et al., it's not just HP designing "weird" drivers), the problem will appear due to a Microsoft change and therefore Microsoft gets the blame.
- freeone3000 5y ago
  It's getting to the point where a sane mitigation would be to spin up a Windows 10 instance in Hyper-V, running the print spooler and driver in that, and using USB/ethernet passthrough!
  - zinekeller 5y ago
    Some might react and say "just move to Linux (even only the servers!)" and while I'll say NeXT/Apple's design of CUPS is great, it's harder to get a good Linux printer driver that's designed for business use (HPLIP is... painful, and I'm pretty sure that's out of most support contracts).
    - gerdesj 5y ago
      De-install HPLIP and use "IPP Everywhere" if you can. Since I've done that I've had reliable printing that doesn't break on package updates. Given that I use Arch, that was an occasional surprise finale to an update session. It usually started with my SO having an urgent print job to do. Also, put your preferred paper size in /etc/papersize if you are not letter size inclined.
      - merb 5y ago
        xerox color machines with fiery controllers aren't that well working with ipp, the last time I worked with them (3 years ago) it crashed pretty often, which meant a full reboot was needed which takes like 5+ minutes on these which was not possible. lpd is still super simple and works super well on these machines, even when using non standard ports
        tonyedgecombe 5y ago
        The trouble with LPD is there is no support for authentication, authorisation or encryption.
        gerdesj 5y ago
        We all have anecdotes with regards printers 8) I think that nmap still refuses to pound away at the JetDirect port ... ... OK I've just tested that against my home HP MFP and 9100/tcp shows in the results. If lpd does it for you on Xerox printers, then fine: use that. I was discussing HPLIP ie getting rid of a Hewlett Packard thing!
  - 5y ago
- tonyedgecombe 5y ago
  >Why does the HP printing drivers run in kernel-mode? I don't know! Printer drivers haven't run in the kernel since NT 4.0.
  - zinekeller 5y ago
    > Printer drivers haven't run in the kernel since NT 4.0. I wish that's true, and the APIs are there (and should be used!), but except for incompetence I don't know how some drivers still have that kernel-side code (unless DRM? And if it is, why?) I even thought that the 64-bit and Vista driver reset have banished kernel-level drivers, but in the past 6 months I still see printer drivers with kernel-level components. I just scratch my head why? (Some of them are for PCI (yes, not a typo) connector, which is reasonable but weird for a printer. Others are running to printer port and USB but the kernel-level components are still there, active according to Device Manager).
    - cogman10 5y ago
      Oh I can tell you exactly why, particularly because I've worked with a bunch of ex HP printer devs. "Hey we should upgrade to the latest standards!" "Why?" "Because they are safer!" "Yeah, but it might break things" "But it's safer!" "Yeah, but the current shit is working. Don't mess with a working thing". It's practically a religion there. I interned there years ago, it was a momentous task getting internal people to move from IE 7 to firefox, chrome, or IE8. What I'd done was improve one of their internal tools with an updated javascript library. However, it ran like ass on IE 7 and perfectly fine on pretty much everything else.
- tinus_hn 5y ago
  Interesting how crashes caused by printer drivers in the kernel are an argument against moving them to user space where they can’t cause crashes
temac 5y ago
> Aside from taking it out of SYSTEM scope(!) Windows does all kind of random things at SYSTEM scope and maybe for most of it, it is a terrible idea; yet they think that somehow VBS will save the world, and do not give a shit about the obvious existing shortcomings of their stack... At one point a virus could exploit a bug in the MS Defender AV engine and ironically execute under SYSTEM by just being scanned (!!) and they reacted by introducing MP_FORCE_USE_SANDBOX, however it seems that instead of quickly extending the usage of that sandbox to every user they... now just spawn the worker process (if the env var is there) but it does not do anything anymore :/ I get that engineers are attracted to working on new shinny things, but at one point not maintaining the old stuff (or even just properly removing!) under the excuse that the new unfinished one is more important is just lack of vaguely competent project management, and overall lack of professionalism. Even maybe of individual contributors: if you do not push strongly to get the shit cleaned-up in priority, you are as guilty as the people directing you to work on other (likely soon to be deprecated and unmaintained too) things.
- hulitu 5y ago
  Not wanting to defend MS employees here, but maybe they are paid based on the new shiny things they produce, not based on fixing old stuff. KPI forever
  - jcpham2 5y ago
    I found this funny because a large portion of my job has morphed into designing reports for different departments to measure performance. It boils down to a lot of binary yes/no formulas in excel and measuring those yes/no as a performance indicator. KPI forever indeed. I never expected as a sysadmin I’d be creating excel reports to measure the performance of others.
- jeffbee 5y ago
  And when you say "at one point" you mean as recently as today. https://twitter.com/taviso/status/1412809764363411457 https://twitter.com/taviso/status/1412809764363411457
londons_explore 5y ago
I think they might be hoping that printing, and print spooling, might be going away. In the future, applications might be able to communicate directly with a network printer via TCP/HTTP. The printer itself does spooling, queueing, management, etc. The OS doesn't need to be involved at all. "Legacy Printer" support can be disabled by default. If you need to change ink cartridges you either do it on the screen of the printer, or install the manufacturers tool which also just communicates over TCP/HTTP and doesn't use any special interfaces.
- tonyedgecombe 5y ago
  More likely they will try and find a way to insert Azure into the process, like this: https://www.microsoft.com/en-ww/microsoft-365/windows/universal-print https://www.microsoft.com/en-ww/microsoft-365/windows/univer...
swiley 5y ago
On other OSes the people maintaining/updating legacy stuff are paid consultants. On Windows only Microsoft is allowed to do this so it isn't done.
tonyedgecombe 5y ago
>The printer stack which has barely received an update since Windows XP is a perfect example. The big change was the shift to XPS based drivers in Vista (which was back ported to XP). The trouble is the printer manufacturers have largely ignored this and continued to produce GDI based drivers. I sometimes wonder if Windows has too much backwards compatibility.
- WorldMaker 5y ago
  Allegedly Vista was going to 100% force XPS drivers, but because they dropped backward compatibility in the network and sound stacks they got a lot of push back that they were crossing too many backwards compatible lines and the printer companies were the ones with the most skin in the game (and cash warchests from ink cartridge sales; and yet the least incentives to update old drivers with better software) so they backed off. Appearances are they tried to break printer driver compatibility again in Windows 8 (which had some smart ideas on how to modernize the print stack but never got off the ground with them because no drivers adapted to the new stack; and then Windows 10 dropped most of what made the new stack special including not just moving everything into user space but moving everything into app sandboxes; no more annoying printer "background app" notifications and ads would have been amazing) and got worse pushback from the entrenched interests.
  - tonyedgecombe 5y ago
    You would think Microsoft would have enough might to push back on that. Which printer manufacturer is going to sell a printer without drivers for the dominant operating system.
    - WorldMaker 5y ago
      Unfortunately, it would be the consumers that suffer if Microsoft did push back. Since somewhere in the 90s consumer printer drivers have been as much about DRM for ink cartridges as anything else. Microsoft isn't allowed the keys to the DRM so they can't make generic drivers for old hardware. The printer companies would love an excuse to force everyone off old hardware to new hardware with fancier DRM. Also, in terms of naming and shaming specific manufacturers, HP/HPE alone has already been known to hold entire industries hostage to old (unsupported) versions of Windows or the various whims of their AIX/HP-UX Unix-based OS efforts rather than support up-to-date drivers on Windows for printers and plotters (and medical equipment and other electronics). It really is easy to imagine them telling the same bad infosec advice to consumers if Microsoft were to break compatibility in their printer drivers. No one wants the chaos of supporting their family members trying to stick on an ancient unsupported Windows XP version simply because their reliable old inkjet printer told them to. Sure everyone thinks they want the Year of Linux on the Desktop, but no one actually wants to help a friend dual boot into HP-UX, and debug cups dump output for them, just to print something twice a year.
PhantomGremlin 5y ago
I hope this issue is the motivator Microsoft needs to finally do something Yes, finally, this will be the defining event that finally gets Microsoft to give a flying fuck about security or about code quality. /s Anyone remember this stuff circa January 2002? Gates Finally Discovers Security https://www.wired.com/2002/01/gates-finally-discovers-security/ https://www.wired.com/2002/01/gates-finally-discovers-securi... To quote from that: Bill Gates' directive that Microsoft emphasize security features in its product is a major strategy shift, but some experts wonder if there's anything behind the words. Now, 19 1/2 years later, Bill's directive will finally be taken seriously? Maybe once we hit 1,000,000 "critical" security flaws in Microsoft products? How many more before we hit that number? Does anyone even bother to count?

zinekeller 5y ago

I mean, they seriously done many things in Windows 10 (and analogous servers) to improve security without too much breakage, like moving font processing from kernel-level to user-level (n.b. in the process, they have thrown Adobe code). Print spooler is tricky though: even if the official design of the API theoretically allows it to run user-level, ask users of HP products (including business ones) how many times their computer crash due to weirdness in HP drivers. (Why does the HP printing drivers run in kernel-level mode? I don't know!) I'm pretty sure that nothing will happen when the spooler is moved to user-side processing, right? (In all seriousness, the drivers will break and probably crash the system. These products are officially out-of-support, so short of Microsoft paying HP to revamp the drivers nothing will happen.) In all probability, probably even Microsoft is contemplating such move but breaking a lot of stuff is reputationally damaging to them, even if it's not really their fault in this case. Removing SafeROM is easier because it's just grumbling gamers who can probably patch up Windows 7, but imagine the support calls when the print spooler, a critical component, is moved. Even when a rational mind would put the blame on HP (et al., it's not just HP designing "weird" drivers), the problem will appear due to a Microsoft change and therefore Microsoft gets the blame.

freeone3000 5y ago
It's getting to the point where a sane mitigation would be to spin up a Windows 10 instance in Hyper-V, running the print spooler and driver in that, and using USB/ethernet passthrough!
- zinekeller 5y ago
  Some might react and say "just move to Linux (even only the servers!)" and while I'll say NeXT/Apple's design of CUPS is great, it's harder to get a good Linux printer driver that's designed for business use (HPLIP is... painful, and I'm pretty sure that's out of most support contracts).
  - gerdesj 5y ago
    De-install HPLIP and use "IPP Everywhere" if you can. Since I've done that I've had reliable printing that doesn't break on package updates. Given that I use Arch, that was an occasional surprise finale to an update session. It usually started with my SO having an urgent print job to do. Also, put your preferred paper size in /etc/papersize if you are not letter size inclined.
    - merb 5y ago
      xerox color machines with fiery controllers aren't that well working with ipp, the last time I worked with them (3 years ago) it crashed pretty often, which meant a full reboot was needed which takes like 5+ minutes on these which was not possible. lpd is still super simple and works super well on these machines, even when using non standard ports
      - tonyedgecombe 5y ago
        The trouble with LPD is there is no support for authentication, authorisation or encryption.
      - gerdesj 5y ago
        We all have anecdotes with regards printers 8) I think that nmap still refuses to pound away at the JetDirect port ... ... OK I've just tested that against my home HP MFP and 9100/tcp shows in the results. If lpd does it for you on Xerox printers, then fine: use that. I was discussing HPLIP ie getting rid of a Hewlett Packard thing!
- 5y ago
tonyedgecombe 5y ago
>Why does the HP printing drivers run in kernel-mode? I don't know! Printer drivers haven't run in the kernel since NT 4.0.
- zinekeller 5y ago
  > Printer drivers haven't run in the kernel since NT 4.0. I wish that's true, and the APIs are there (and should be used!), but except for incompetence I don't know how some drivers still have that kernel-side code (unless DRM? And if it is, why?) I even thought that the 64-bit and Vista driver reset have banished kernel-level drivers, but in the past 6 months I still see printer drivers with kernel-level components. I just scratch my head why? (Some of them are for PCI (yes, not a typo) connector, which is reasonable but weird for a printer. Others are running to printer port and USB but the kernel-level components are still there, active according to Device Manager).
  - cogman10 5y ago
    Oh I can tell you exactly why, particularly because I've worked with a bunch of ex HP printer devs. "Hey we should upgrade to the latest standards!" "Why?" "Because they are safer!" "Yeah, but it might break things" "But it's safer!" "Yeah, but the current shit is working. Don't mess with a working thing". It's practically a religion there. I interned there years ago, it was a momentous task getting internal people to move from IE 7 to firefox, chrome, or IE8. What I'd done was improve one of their internal tools with an updated javascript library. However, it ran like ass on IE 7 and perfectly fine on pretty much everything else.
tinus_hn 5y ago
Interesting how crashes caused by printer drivers in the kernel are an argument against moving them to user space where they can’t cause crashes

temac 5y ago

> Aside from taking it out of SYSTEM scope(!) Windows does all kind of random things at SYSTEM scope and maybe for most of it, it is a terrible idea; yet they think that somehow VBS will save the world, and do not give a shit about the obvious existing shortcomings of their stack... At one point a virus could exploit a bug in the MS Defender AV engine and ironically execute under SYSTEM by just being scanned (!!) and they reacted by introducing MP_FORCE_USE_SANDBOX, however it seems that instead of quickly extending the usage of that sandbox to every user they... now just spawn the worker process (if the env var is there) but it does not do anything anymore :/ I get that engineers are attracted to working on new shinny things, but at one point not maintaining the old stuff (or even just properly removing!) under the excuse that the new unfinished one is more important is just lack of vaguely competent project management, and overall lack of professionalism. Even maybe of individual contributors: if you do not push strongly to get the shit cleaned-up in priority, you are as guilty as the people directing you to work on other (likely soon to be deprecated and unmaintained too) things.

hulitu 5y ago
Not wanting to defend MS employees here, but maybe they are paid based on the new shiny things they produce, not based on fixing old stuff. KPI forever
- jcpham2 5y ago
  I found this funny because a large portion of my job has morphed into designing reports for different departments to measure performance. It boils down to a lot of binary yes/no formulas in excel and measuring those yes/no as a performance indicator. KPI forever indeed. I never expected as a sysadmin I’d be creating excel reports to measure the performance of others.
jeffbee 5y ago
And when you say "at one point" you mean as recently as today. https://twitter.com/taviso/status/1412809764363411457 https://twitter.com/taviso/status/1412809764363411457

londons_explore 5y ago

I think they might be hoping that printing, and print spooling, might be going away. In the future, applications might be able to communicate directly with a network printer via TCP/HTTP. The printer itself does spooling, queueing, management, etc. The OS doesn't need to be involved at all. "Legacy Printer" support can be disabled by default. If you need to change ink cartridges you either do it on the screen of the printer, or install the manufacturers tool which also just communicates over TCP/HTTP and doesn't use any special interfaces.

tonyedgecombe 5y ago
More likely they will try and find a way to insert Azure into the process, like this: https://www.microsoft.com/en-ww/microsoft-365/windows/universal-print https://www.microsoft.com/en-ww/microsoft-365/windows/univer...

swiley 5y ago

On other OSes the people maintaining/updating legacy stuff are paid consultants. On Windows only Microsoft is allowed to do this so it isn't done.

tonyedgecombe 5y ago

>The printer stack which has barely received an update since Windows XP is a perfect example. The big change was the shift to XPS based drivers in Vista (which was back ported to XP). The trouble is the printer manufacturers have largely ignored this and continued to produce GDI based drivers. I sometimes wonder if Windows has too much backwards compatibility.

WorldMaker 5y ago
Allegedly Vista was going to 100% force XPS drivers, but because they dropped backward compatibility in the network and sound stacks they got a lot of push back that they were crossing too many backwards compatible lines and the printer companies were the ones with the most skin in the game (and cash warchests from ink cartridge sales; and yet the least incentives to update old drivers with better software) so they backed off. Appearances are they tried to break printer driver compatibility again in Windows 8 (which had some smart ideas on how to modernize the print stack but never got off the ground with them because no drivers adapted to the new stack; and then Windows 10 dropped most of what made the new stack special including not just moving everything into user space but moving everything into app sandboxes; no more annoying printer "background app" notifications and ads would have been amazing) and got worse pushback from the entrenched interests.
- tonyedgecombe 5y ago
  You would think Microsoft would have enough might to push back on that. Which printer manufacturer is going to sell a printer without drivers for the dominant operating system.
  - WorldMaker 5y ago
    Unfortunately, it would be the consumers that suffer if Microsoft did push back. Since somewhere in the 90s consumer printer drivers have been as much about DRM for ink cartridges as anything else. Microsoft isn't allowed the keys to the DRM so they can't make generic drivers for old hardware. The printer companies would love an excuse to force everyone off old hardware to new hardware with fancier DRM. Also, in terms of naming and shaming specific manufacturers, HP/HPE alone has already been known to hold entire industries hostage to old (unsupported) versions of Windows or the various whims of their AIX/HP-UX Unix-based OS efforts rather than support up-to-date drivers on Windows for printers and plotters (and medical equipment and other electronics). It really is easy to imagine them telling the same bad infosec advice to consumers if Microsoft were to break compatibility in their printer drivers. No one wants the chaos of supporting their family members trying to stick on an ancient unsupported Windows XP version simply because their reliable old inkjet printer told them to. Sure everyone thinks they want the Year of Linux on the Desktop, but no one actually wants to help a friend dual boot into HP-UX, and debug cups dump output for them, just to print something twice a year.

PhantomGremlin 5y ago

I hope this issue is the motivator Microsoft needs to finally do something Yes, finally, this will be the defining event that finally gets Microsoft to give a flying fuck about security or about code quality. /s Anyone remember this stuff circa January 2002? Gates Finally Discovers Security https://www.wired.com/2002/01/gates-finally-discovers-security/ https://www.wired.com/2002/01/gates-finally-discovers-securi... To quote from that: Bill Gates' directive that Microsoft emphasize security features in its product is a major strategy shift, but some experts wonder if there's anything behind the words. Now, 19 1/2 years later, Bill's directive will finally be taken seriously? Maybe once we hit 1,000,000 "critical" security flaws in Microsoft products? How many more before we hit that number? Does anyone even bother to count?