7 ms·

As a sysadmin on a Windows network of ~100 computers, this story makes me want to cry, although maybe for the wrong reason: I see weird problems of the sort "I

by krylon 10y ago

As a sysadmin on a Windows network of ~100 computers, this story makes me want to cry, although maybe for the wrong reason: I see weird problems of the sort "It did work before I went on my lunch break" on a fairly regular basis. How often would I like to go down the rabbit hole and explore these problems in such depth, but if I did that, I would hardly get any work done. The frequency at which our users run into these problems is just too high. It is so frustrating so restart a computer or maybe re-install it and see the problem disappear, because now all hope to understand what caused the problem in the first place is gone. And problems that disappear for no good reason have a tendency to return for no good reason, most often on a Friday afternoon, just when you're about to call it a day. ;-/

616c 10y ago
I am in a similar boat. With DSC, I am increasingly excited with the idea end user computers can also shift to immutable, or approaching what we call immutable, infrastructure that has logically valid weight in the communities represented here. The problem? Culture. So many people do not understand when I say the following things: - Do not install with the GUI, please use the deployment system to document unattended installation. - You should not have to log into a computer and manually install and customize things, especially if you forget to document. - You should not be doing things, install, configuration change, with following this AND updating the team in your notes, so when I ask you, you in fact remember. These things are the product of less fires, as I see it, and why I too have to re-image everything (although is largely out of concern for any malware, especially with the majority of our userbase).
- tikhonj 10y ago
  Honestly, this is what got me most excited about NixOS. I mean, the underlying technology is spiffy, of course, but the real key is that it lets you configure everything declaratively while keeping track of different versions and actively pushes you in that direction. Doing things the "right" way is also the path of least resistance. Sometimes this is a bit annoying when you just want something to work but can't just hack it because the system files are hidden all over the place and papered over with symlinks, but in the long run it means your system configuration is way easier to maintain.
  - 616c 10y ago
    I am very excited for NixOS, for me. In the professional IT sphere, outside of DevOps, this is so far away it is depressing. I spend my weekends with QubesOS, NixOS, and maybe in the future GuixSD and SubgraphOS as my therapy. I have started to look into a career transition into DevOps (please do not laugh, feel free to downvote) not because of how cool it looks or the increasing culture around it, but I need a break from the mainstream of throw everything and anything on the wall until it sticks without rigor, or your time is cheap automation is a very, very low priority since we pay you to do the tasks we dislike as more serious engineers (who also build some systems with checklists).
- krylon 10y ago
  I read about DSC a while ago, and it sounds very appealing. So far, though, I have not managed to actually get to know it personally, so to speak. We have over the past two years tried to move as much of the configuration as possible to GPOs, although they bring their own share of problems. On Unix-like systems, one can use log files to track down problems most of the time, on Windows it seems like logging is kind of an afterthought. I especially hate it when gpresult says it did apply a certain setting when upon inspecting the system the setting clearly has not been applied (or overriden by something else? Who knows?). It is frustrating because GPOs seem like such a great idea in theory.
  - 616c 10y ago
    I have just read about it and the WMF (WMI Management Foundation) docs, whitepapers, and framework. I have not played much. GPO, and GPP (the preferences), are a nightmare. - The gpresult utility and rsop.msc tell you changes, but that does not mean much, because - GPO is async and requires sometimes a shutdown, not just a reboot (fom 5+ years of experience), so good luck dropping this crap on a dime; God help you - The Registry.pol files are not easily auditable or usable outside gpresult and rsop.msc - If you hit slow links, processing will be disabled; this is not slow all the time, but when 2% of the computers booting at 6:54AM on Tuesday it does not get applied - A whole bunch of other stuff I forget in this rant But I totally agree with you, such a pain. I have not had a lot of time for Salt, Puppet, and Chef. Whether it is in parallel or thanks to DSC (I saw some Powershell in one repo ... Ansible?) those tools are also a reality. I use an expensive SCCM alternative, but I am seriously considering proposing moving to one of the Chef/Puppet/Anisble/Salt stacks with SSH becoming a reality on Windows. All I can say is thank God.
vacri 10y ago
One of the biggest cultural changes I had when I jumped the fence from Windows to Linux was the reboot. On Windows, the culture was strongly "reboot first, if it still happens, then it's a problem". On linux, it was a measure of last resort, because then you can't fix the problem (as you say). Obviously the latter is the best way, but it's interesting that the culture of the two systems is so different, no doubt borne from Windows' legacy era.
- paulddraper 10y ago
  Well, the other difference is that reboot isn't often needed to solve the issue, unless you're messing around with kernel modules.
  - digi_owl 10y ago
    This may well be changing though, as more and more "desktop-isms" claw their way deeper into the stack.
- XorNot 10y ago
  Except you have to triage. I'll happily reboot any box which fails unusually because my overall system should be HA enough to survive that. It's only interesting if the failure count is high, and I'm running reasonably up to date code. Otherwise there's too many things in a day to get through.
  - wpietri 10y ago
    I'm with Vacri; I'm very reluctant to do that. Small problems can be harbingers of big ones. Even when they aren't, small problems often confound the ability to solve big ones. Most concerning for me, though, is what safety experts call "normalization of deviance". It's the process by which people become accustomed to small failures, which creates opportunities for big failures to happen. A big example is the Challenger disaster. [1] I see shops with low bug rates, where people think a lot about quality. And I see shops that, thanks to high bug rates, are too busy fighting fires to ever spend much time on quality. I never see any place in between. And I think normalization of deviance is why. [1] http://mikemullane.com/stopping-normalization-of-deviance/ http://mikemullane.com/stopping-normalization-of-deviance/
chris_wot 10y ago
This is where a good incident/problem management tracking system comes in handy. Sure, you can't chase down all the oddness happening on Windows, but there is nothing from stopping you from having the incidents logged. What you do is use the incident you clone the incident to an open problem record (regardless of whether you have a workaround or not) with all the details of what you saw and everything you did. Then keep it open till you determine the root cause of the problem. When other incidents are logged, if you have defined the problem well enough then you search for a problem record that matches the incident symptoms and link it to the problem record. The problem record also holds the workaround that use used to get the end user up and running so you can use this if the issue is super critical. If you find that you've linked a certain number of incidents to the problem, then you know you it's actually worthwhile doing root cause analysis and spend the time figuring out what is causing the error, so you go down the rabbit hole - and you can justify the time to do so. When you figure out the root cause, if it's a simple resolution that doesn't require a major change to the environment then you may not have to do much to prevent the issue in future - sort of depends on the complexity/needs of your environment and organization. But regardless you raise a known error record and link the problem to this. In the known error record you document the problem and as many symptoms as possible (some people list workarounds here, others list the workarounds in the problem record, other prefer to keep workaround info strictly in incident records), the root cause and how you resolved the issue fully. Regardless, mainly from the known issue record if you find you need to make a scheduled change that may impact environments then you lodge a change request through the CAB processes you have in place. Normally I find for server and network infrastructure the change just requires coordination with teams who use the infrastructure, which if you've setup your CMDB properly you can work out by backtracking the infrastructure configuration items to linked services. I've found that if you have defined your service catalog properly then you will have defined your operational services and linked these to business services that are mostly customer facing. This helps impact analysis and finding the correct window in which to make the change. For things like fixing application bugs, I have found that it's still worthwhile raising a change request, then have that change moved into the development fix process with all appropriate testing, etc - normally this then links into a wider release management process which may actually require a new overarching change management request as other fixes are part of the change - sometimes you need to review the impact of how deploying the release might impact the environment in unexpected ways. I that's basically a big chunk of ITIL, and I found that if it's done correctly and busy-work is reduced (mainly by asking for too much info), when an appropriate setup of the service layer is made and the CMDB has been mapped well, then it actually can help medium to large organizations pretty effectively. The key is to define a catalog of services across the business, without this it's hard to know the impact of incidents, bugs and any changes you may want to make in your environment.
MertsA 10y ago
This is what I hate about closed source software in general. Every sysadmin out there has complaints about how terrible support is from XYZ vendor, but it's not just that the vendor can't provide support, it's that by going closed source, the vendor is the only one who can really track down these sort of bugs. With an open source stack, no matter what breaks in any software component, I'm not left high and dry hoping that some vendor who charges buckets of cash per incident is competent. As a sysadmin, I'm putting my neck on the line anytime something breaks. I need to be able to fix things when the vendor fails.
- loteck 10y ago
  As a sysadmin, I'm putting my neck on the line anytime something breaks. In many years of doing this type of work, I've noticed this fear of your neck being on the line is a myth. Heads rarely roll merely because something goes wrong. I've also noticed almost every sysadmin believes that myth, and it can make us very difficult to relate to or even work with. Ironically, that's a problem that may actually cause you to lose your job.
  - SystemOut 10y ago
    Back when I used to lead the engineering group for a company I was asked to also lead the IT and Ops teams. I was amazed at how many of the sys admins lived in fear that anything going wrong or taking a risk on something new and it not working as expected would get them fired. Virtually no one wanted to take any risks because of this fear. It took a long time of working with the team and demonstrating that people did not get fired when things went wrong. It was a real eye opener for me.
- tonyedgecombe 10y ago
  A good vendor will work with you to resolve the problem, not all closed source software is bad.
  - wpietri 10y ago
    I'm familiar with that theory, but from my personal experience I'd put the number of good-vendor software packages around 1%. And in retrospect, I'm not sure those cases were so much "good vendor" as "vendor over whom we had significant financial leverage".

by krylon 10y ago

616c 10y ago

I am in a similar boat. With DSC, I am increasingly excited with the idea end user computers can also shift to immutable, or approaching what we call immutable, infrastructure that has logically valid weight in the communities represented here. The problem? Culture. So many people do not understand when I say the following things: - Do not install with the GUI, please use the deployment system to document unattended installation. - You should not have to log into a computer and manually install and customize things, especially if you forget to document. - You should not be doing things, install, configuration change, with following this AND updating the team in your notes, so when I ask you, you in fact remember. These things are the product of less fires, as I see it, and why I too have to re-image everything (although is largely out of concern for any malware, especially with the majority of our userbase).

tikhonj 10y ago
Honestly, this is what got me most excited about NixOS. I mean, the underlying technology is spiffy, of course, but the real key is that it lets you configure everything declaratively while keeping track of different versions and actively pushes you in that direction. Doing things the "right" way is also the path of least resistance. Sometimes this is a bit annoying when you just want something to work but can't just hack it because the system files are hidden all over the place and papered over with symlinks, but in the long run it means your system configuration is way easier to maintain.
- 616c 10y ago
  I am very excited for NixOS, for me. In the professional IT sphere, outside of DevOps, this is so far away it is depressing. I spend my weekends with QubesOS, NixOS, and maybe in the future GuixSD and SubgraphOS as my therapy. I have started to look into a career transition into DevOps (please do not laugh, feel free to downvote) not because of how cool it looks or the increasing culture around it, but I need a break from the mainstream of throw everything and anything on the wall until it sticks without rigor, or your time is cheap automation is a very, very low priority since we pay you to do the tasks we dislike as more serious engineers (who also build some systems with checklists).
krylon 10y ago
I read about DSC a while ago, and it sounds very appealing. So far, though, I have not managed to actually get to know it personally, so to speak. We have over the past two years tried to move as much of the configuration as possible to GPOs, although they bring their own share of problems. On Unix-like systems, one can use log files to track down problems most of the time, on Windows it seems like logging is kind of an afterthought. I especially hate it when gpresult says it did apply a certain setting when upon inspecting the system the setting clearly has not been applied (or overriden by something else? Who knows?). It is frustrating because GPOs seem like such a great idea in theory.
- 616c 10y ago
  I have just read about it and the WMF (WMI Management Foundation) docs, whitepapers, and framework. I have not played much. GPO, and GPP (the preferences), are a nightmare. - The gpresult utility and rsop.msc tell you changes, but that does not mean much, because - GPO is async and requires sometimes a shutdown, not just a reboot (fom 5+ years of experience), so good luck dropping this crap on a dime; God help you - The Registry.pol files are not easily auditable or usable outside gpresult and rsop.msc - If you hit slow links, processing will be disabled; this is not slow all the time, but when 2% of the computers booting at 6:54AM on Tuesday it does not get applied - A whole bunch of other stuff I forget in this rant But I totally agree with you, such a pain. I have not had a lot of time for Salt, Puppet, and Chef. Whether it is in parallel or thanks to DSC (I saw some Powershell in one repo ... Ansible?) those tools are also a reality. I use an expensive SCCM alternative, but I am seriously considering proposing moving to one of the Chef/Puppet/Anisble/Salt stacks with SSH becoming a reality on Windows. All I can say is thank God.

vacri 10y ago

One of the biggest cultural changes I had when I jumped the fence from Windows to Linux was the reboot. On Windows, the culture was strongly "reboot first, if it still happens, then it's a problem". On linux, it was a measure of last resort, because then you can't fix the problem (as you say). Obviously the latter is the best way, but it's interesting that the culture of the two systems is so different, no doubt borne from Windows' legacy era.

paulddraper 10y ago
Well, the other difference is that reboot isn't often needed to solve the issue, unless you're messing around with kernel modules.
- digi_owl 10y ago
  This may well be changing though, as more and more "desktop-isms" claw their way deeper into the stack.
XorNot 10y ago
Except you have to triage. I'll happily reboot any box which fails unusually because my overall system should be HA enough to survive that. It's only interesting if the failure count is high, and I'm running reasonably up to date code. Otherwise there's too many things in a day to get through.
- wpietri 10y ago
  I'm with Vacri; I'm very reluctant to do that. Small problems can be harbingers of big ones. Even when they aren't, small problems often confound the ability to solve big ones. Most concerning for me, though, is what safety experts call "normalization of deviance". It's the process by which people become accustomed to small failures, which creates opportunities for big failures to happen. A big example is the Challenger disaster. [1] I see shops with low bug rates, where people think a lot about quality. And I see shops that, thanks to high bug rates, are too busy fighting fires to ever spend much time on quality. I never see any place in between. And I think normalization of deviance is why. [1] http://mikemullane.com/stopping-normalization-of-deviance/ http://mikemullane.com/stopping-normalization-of-deviance/

chris_wot 10y ago

This is where a good incident/problem management tracking system comes in handy. Sure, you can't chase down all the oddness happening on Windows, but there is nothing from stopping you from having the incidents logged. What you do is use the incident you clone the incident to an open problem record (regardless of whether you have a workaround or not) with all the details of what you saw and everything you did. Then keep it open till you determine the root cause of the problem. When other incidents are logged, if you have defined the problem well enough then you search for a problem record that matches the incident symptoms and link it to the problem record. The problem record also holds the workaround that use used to get the end user up and running so you can use this if the issue is super critical. If you find that you've linked a certain number of incidents to the problem, then you know you it's actually worthwhile doing root cause analysis and spend the time figuring out what is causing the error, so you go down the rabbit hole - and you can justify the time to do so. When you figure out the root cause, if it's a simple resolution that doesn't require a major change to the environment then you may not have to do much to prevent the issue in future - sort of depends on the complexity/needs of your environment and organization. But regardless you raise a known error record and link the problem to this. In the known error record you document the problem and as many symptoms as possible (some people list workarounds here, others list the workarounds in the problem record, other prefer to keep workaround info strictly in incident records), the root cause and how you resolved the issue fully. Regardless, mainly from the known issue record if you find you need to make a scheduled change that may impact environments then you lodge a change request through the CAB processes you have in place. Normally I find for server and network infrastructure the change just requires coordination with teams who use the infrastructure, which if you've setup your CMDB properly you can work out by backtracking the infrastructure configuration items to linked services. I've found that if you have defined your service catalog properly then you will have defined your operational services and linked these to business services that are mostly customer facing. This helps impact analysis and finding the correct window in which to make the change. For things like fixing application bugs, I have found that it's still worthwhile raising a change request, then have that change moved into the development fix process with all appropriate testing, etc - normally this then links into a wider release management process which may actually require a new overarching change management request as other fixes are part of the change - sometimes you need to review the impact of how deploying the release might impact the environment in unexpected ways. I that's basically a big chunk of ITIL, and I found that if it's done correctly and busy-work is reduced (mainly by asking for too much info), when an appropriate setup of the service layer is made and the CMDB has been mapped well, then it actually can help medium to large organizations pretty effectively. The key is to define a catalog of services across the business, without this it's hard to know the impact of incidents, bugs and any changes you may want to make in your environment.

MertsA 10y ago

This is what I hate about closed source software in general. Every sysadmin out there has complaints about how terrible support is from XYZ vendor, but it's not just that the vendor can't provide support, it's that by going closed source, the vendor is the only one who can really track down these sort of bugs. With an open source stack, no matter what breaks in any software component, I'm not left high and dry hoping that some vendor who charges buckets of cash per incident is competent. As a sysadmin, I'm putting my neck on the line anytime something breaks. I need to be able to fix things when the vendor fails.

loteck 10y ago
As a sysadmin, I'm putting my neck on the line anytime something breaks. In many years of doing this type of work, I've noticed this fear of your neck being on the line is a myth. Heads rarely roll merely because something goes wrong. I've also noticed almost every sysadmin believes that myth, and it can make us very difficult to relate to or even work with. Ironically, that's a problem that may actually cause you to lose your job.
- SystemOut 10y ago
  Back when I used to lead the engineering group for a company I was asked to also lead the IT and Ops teams. I was amazed at how many of the sys admins lived in fear that anything going wrong or taking a risk on something new and it not working as expected would get them fired. Virtually no one wanted to take any risks because of this fear. It took a long time of working with the team and demonstrating that people did not get fired when things went wrong. It was a real eye opener for me.
tonyedgecombe 10y ago
A good vendor will work with you to resolve the problem, not all closed source software is bad.
- wpietri 10y ago
  I'm familiar with that theory, but from my personal experience I'd put the number of good-vendor software packages around 1%. And in retrospect, I'm not sure those cases were so much "good vendor" as "vendor over whom we had significant financial leverage".