5 ms·

The comic strip was tough to swallow for me, having spent the last few months automating configuration of my Windows workstation setup I use for 3D/VR. I was do

by gt_ 9y ago

The comic strip was tough to swallow for me, having spent the last few months automating configuration of my Windows workstation setup I use for 3D/VR. I was doing freelance jobs and had my workstation collapse a few times during aggressive deadline sprints so I figured I would automate the rebuild so that reinstalling the OS becomes more of a Plan A than a Plan B. Why am I posting this here? I am honestly on the fence about whether this has been a good use of my time. It took longer than expected and I had to use Powershell for some of it, which was just dreadful. It was ‘for business’ but I am still not sure it was a great call. Has anyone else out there found themselves automating something like this? Was it worth it?

contingencies 9y ago
Set yourself a goal never to do that again. Learn about modern filesystems with snapshots like ZFS, and PXE. If you don't need native performance, use a VM in preference to PXE. This is how many cluster/cloud data centers operate. https://en.wikipedia.org/wiki/Snapshot_(computer_storage) https://en.wikipedia.org/wiki/Snapshot_(computer_storage) https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/Preboot_Execution_Environment https://en.wikipedia.org/wiki/Preboot_Execution_Environment
- gt_ 9y ago
  Thanks for the response. I will certainly research these items. Unfortunately, I do need native performance. My workstation runs a 5960X, 256 GB RAM, x4 GTX 980 Tis, a stack of SSDs and I use all of it regularly. I’m very curious about this PXE solution. I am googling now. If anyone has pointers on methods that fit my needs, I would be gracious. Thanks! UPDATE: Reading the Wikipedia on PXE suggests to me I would be using one of the Microsoft deployment solutions. I messed with these a littleand it felt like cutting bread with a chainsaw. It sort of worked but, on top of many known bugs, it it was designed for big deployments, prioritizing hundreds of features and admin relationships that I have no need for. Maybe I’m looking at the wrong thing, but it’s pretty hard to tell. My general goal in my scripting solution was to automate out the Windows annoyances like UAC, bloatware, registry configs, and software installation. I learned how to do that with scripting and it seems predictable even after OS updates, and I can modify it very easily. It has a few run-all directories for quick mods, and some .jsons with config data. I made an install USP that puts a script on my desktop which installs git, pulls my setup repo down and runs it, downloading files needed from my google drive... just to give a picture of what worked. It takes about the length of a lunch break to run and I’m back in action, with exception of a few licensed softwares that I chose not to automate. I’m still interested in learning better solutions.
  - devereaux 9y ago
    With a large storage space, once your installation is done, from a linux: cat /dev/sde3 > /storage/windows-vr-installation.bin Then when the install dies for some reason, restore it with: cat /storage/windows-vr-installation.bin > /dev/sde3 You will say it is inefficient. Maybe. On an optane 900, the burst write speed will be decent enough. Keeping .bin images that are a few Tb is not really a problem with modern setup (zfs, raid, etc.) I know there are commercial solutions doing that, but I wouldn't bother much with that. A simple free software setup will end up being more efficient if you know some basics. If I were you, I really wouldn't bother more than that with automation for reinstalls. You are not deploying to N identical devices (in which case the above trick may still work by having sufficiently similar workstations, and a special licensing scheme - I often buy a few similar laptops for the same reason). Regarding native performance, since being able to reconfigure windows quickly seems very important, if I were you I would do a GPU passthrough. KVM on computers with VT-d etc offers near native performance. That's what gamers do, and I trust them for their insane attention to fps performance :-) It may take some time for the initial tweaking (recompiling kernels, selecting some hardware etc) but it very possible, and also very stable. And it is more and more common. (disclamer: I sell such solutions, and do various weird things for my clients -- including supporting Wine when their need for a very stable solution is greater than the costs of the time I will bill them + the specific hardware I will have them buy to guarantee the speed of the results)
    - gt_ 9y ago
      Thanks the response! It makes sense. It sounds like a similar result operationally to using something like Macrium for a master image backup. This is something I wanted to prevent because modifying that master image backup on a regular basis is such a task. With my scripting solution, it's just editing a few json files. My working environment is constantly changing, especially these days. My preferences also change as I optimize my workflows. My thoughts were to use the approach of For each change in implementation, make a change in code. The other important thing, to me, is that I am free to install any wacky software I may need to try without worrying what it's doing to my system. Something like intermittent backups will lead to accumulation of garbage files when doing this. And if I choose to adopt the software, I add it to my setup package. How do I modify the VM on a regular basis? I trust the input but maybe my understanding is limited by lack of context. I wish I could find a decent training source on this.
AstralStorm 9y ago
Yes, took a week rolling out a new VM system that has no complication of Windows. This way a single use fleet of build nodes. Main issues were related to RAM amount. It is rock solid since, remaining issues are caused by ancient software running on it which cannot be bumped without reinstalling the server. The management and ease of deployment made it totally worth it and the system is much simpler than running a "real cluster" like Kubrenetes or especially a Docker setup. Original was a chroot that got messed up on OS updates.
- gt_ 9y ago
  RAM is a critical necessity for offline 3D rendering. I have 256 GB of it in my workstation. I very rarely need that, but it’s not just a speed issue. Rendering particle physics sims requires the whole sim to be in memory. So, it’s not an option when it is needed. For someone who knows little about cloud computing or VMs but is handy with Python, where would I start? When trying to learn this stuff, I have always come to the conclusion that everybody who uses these technologies is using them for drastically different use cases than what I would use them for. So, I quickly conclude these solutions will involve forcing them to do something they were not designed for.
barrkel 9y ago
Automating something usually involves a risk from predicting the future: the cost of automation is amortized over the number of repetitions, but you don't always repeat everything. It's generally cheaper to automate something if you can perform it on a command line. With command history, it's easy enough to copy the commands and turn them into a script with appropriate parameterization. It doesn't help you a great deal with Windows if you've been doing everything with GUIs. But Powershell itself is an investment that can pay off. If you can automate more everyday tasks, the pain of learning is amortized away. I generally stick with the Unix-like shell that Cygwin gives me, and use the same commands and idioms on as I do on Linux and on Mac, on the very few occasions I use a Mac.
- gt_ 9y ago
  My setup script starts by making a backup of the registry, then configuring Python. I use Python for most of the rest and Powershell only where necessary. Contrary to what my comment may suggest, I am actually a software minimalist. A large part of my setup involves removing clutter from the desktop environment. I have avoided cygwin partially just so I'm not scared of native windows command line, which I find not quite as bad as Powershell to work with, but more limited in interacting with the OS of course. I am familiar with basic linux Bash already and that's what I'm comparing all of this to. Maybe now is a good time to adopt cygwin because, well, better late than never.
  - barrkel 9y ago
    Getting Cygwin set up right takes a little bit of work - it's much easier if you're familiar with configuring a Linux shell and terminal to taste. Out of the box, it's a bit rough; kind of similar to how Emacs is a bit rough. I am scared of the Windows command line - not because it's scary, but because it's so limited. I feel almost helpless, all my tools are missing, it's just so awful. In practice, the reason I use Windows is mostly games, but it's rare I fire them up these days - I have to upgrade and/or re-verify my Steam account almost every time I start the client. Outside of that the browser and the terminal are almost all I use, and I can have them just as easily elsewhere. My point being my experience may not translate to your environment quite as well. Registry in Cygwin is mounted under /proc/registry: ~$ ls -l /proc/registry/HKEY_CURRENT_USER/Software/Microsoft/Windows total 0 dr-xr-x--- 72 SYSTEM SYSTEM 0 Dec 16 23:57 CurrentVersion dr-xr-x--- 2 SYSTEM SYSTEM 0 Dec 13 01:58 DWM dr-xr-x--- 4 SYSTEM SYSTEM 0 Dec 13 01:48 PrivacySettingsBeforeCreatorsUpdate dr-xr-x--- 6 SYSTEM SYSTEM 0 Dec 13 01:52 Shell dr-xr-x--- 4 SYSTEM SYSTEM 0 Dec 13 01:47 TabletPC dr-xr-x--- 3 SYSTEM SYSTEM 0 Dec 13 01:47 'Windows Error Reporting' dr-xr----- 3 barrkel None 0 Dec 13 01:58 Winlogon It's read-only however. Cygwin `regtool` lets you poke values in.
deleted 9y ago
[deleted]

by gt_ 9y ago

contingencies 9y ago

Set yourself a goal never to do that again. Learn about modern filesystems with snapshots like ZFS, and PXE. If you don't need native performance, use a VM in preference to PXE. This is how many cluster/cloud data centers operate. https://en.wikipedia.org/wiki/Snapshot_(computer_storage) https://en.wikipedia.org/wiki/Snapshot_(computer_storage) https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/Preboot_Execution_Environment https://en.wikipedia.org/wiki/Preboot_Execution_Environment

gt_ 9y ago
Thanks for the response. I will certainly research these items. Unfortunately, I do need native performance. My workstation runs a 5960X, 256 GB RAM, x4 GTX 980 Tis, a stack of SSDs and I use all of it regularly. I’m very curious about this PXE solution. I am googling now. If anyone has pointers on methods that fit my needs, I would be gracious. Thanks! UPDATE: Reading the Wikipedia on PXE suggests to me I would be using one of the Microsoft deployment solutions. I messed with these a littleand it felt like cutting bread with a chainsaw. It sort of worked but, on top of many known bugs, it it was designed for big deployments, prioritizing hundreds of features and admin relationships that I have no need for. Maybe I’m looking at the wrong thing, but it’s pretty hard to tell. My general goal in my scripting solution was to automate out the Windows annoyances like UAC, bloatware, registry configs, and software installation. I learned how to do that with scripting and it seems predictable even after OS updates, and I can modify it very easily. It has a few run-all directories for quick mods, and some .jsons with config data. I made an install USP that puts a script on my desktop which installs git, pulls my setup repo down and runs it, downloading files needed from my google drive... just to give a picture of what worked. It takes about the length of a lunch break to run and I’m back in action, with exception of a few licensed softwares that I chose not to automate. I’m still interested in learning better solutions.
- devereaux 9y ago
  With a large storage space, once your installation is done, from a linux: cat /dev/sde3 > /storage/windows-vr-installation.bin Then when the install dies for some reason, restore it with: cat /storage/windows-vr-installation.bin > /dev/sde3 You will say it is inefficient. Maybe. On an optane 900, the burst write speed will be decent enough. Keeping .bin images that are a few Tb is not really a problem with modern setup (zfs, raid, etc.) I know there are commercial solutions doing that, but I wouldn't bother much with that. A simple free software setup will end up being more efficient if you know some basics. If I were you, I really wouldn't bother more than that with automation for reinstalls. You are not deploying to N identical devices (in which case the above trick may still work by having sufficiently similar workstations, and a special licensing scheme - I often buy a few similar laptops for the same reason). Regarding native performance, since being able to reconfigure windows quickly seems very important, if I were you I would do a GPU passthrough. KVM on computers with VT-d etc offers near native performance. That's what gamers do, and I trust them for their insane attention to fps performance :-) It may take some time for the initial tweaking (recompiling kernels, selecting some hardware etc) but it very possible, and also very stable. And it is more and more common. (disclamer: I sell such solutions, and do various weird things for my clients -- including supporting Wine when their need for a very stable solution is greater than the costs of the time I will bill them + the specific hardware I will have them buy to guarantee the speed of the results)
  - gt_ 9y ago
    Thanks the response! It makes sense. It sounds like a similar result operationally to using something like Macrium for a master image backup. This is something I wanted to prevent because modifying that master image backup on a regular basis is such a task. With my scripting solution, it's just editing a few json files. My working environment is constantly changing, especially these days. My preferences also change as I optimize my workflows. My thoughts were to use the approach of For each change in implementation, make a change in code. The other important thing, to me, is that I am free to install any wacky software I may need to try without worrying what it's doing to my system. Something like intermittent backups will lead to accumulation of garbage files when doing this. And if I choose to adopt the software, I add it to my setup package. How do I modify the VM on a regular basis? I trust the input but maybe my understanding is limited by lack of context. I wish I could find a decent training source on this.

AstralStorm 9y ago

Yes, took a week rolling out a new VM system that has no complication of Windows. This way a single use fleet of build nodes. Main issues were related to RAM amount. It is rock solid since, remaining issues are caused by ancient software running on it which cannot be bumped without reinstalling the server. The management and ease of deployment made it totally worth it and the system is much simpler than running a "real cluster" like Kubrenetes or especially a Docker setup. Original was a chroot that got messed up on OS updates.

gt_ 9y ago
RAM is a critical necessity for offline 3D rendering. I have 256 GB of it in my workstation. I very rarely need that, but it’s not just a speed issue. Rendering particle physics sims requires the whole sim to be in memory. So, it’s not an option when it is needed. For someone who knows little about cloud computing or VMs but is handy with Python, where would I start? When trying to learn this stuff, I have always come to the conclusion that everybody who uses these technologies is using them for drastically different use cases than what I would use them for. So, I quickly conclude these solutions will involve forcing them to do something they were not designed for.

barrkel 9y ago

Automating something usually involves a risk from predicting the future: the cost of automation is amortized over the number of repetitions, but you don't always repeat everything. It's generally cheaper to automate something if you can perform it on a command line. With command history, it's easy enough to copy the commands and turn them into a script with appropriate parameterization. It doesn't help you a great deal with Windows if you've been doing everything with GUIs. But Powershell itself is an investment that can pay off. If you can automate more everyday tasks, the pain of learning is amortized away. I generally stick with the Unix-like shell that Cygwin gives me, and use the same commands and idioms on as I do on Linux and on Mac, on the very few occasions I use a Mac.

gt_ 9y ago
My setup script starts by making a backup of the registry, then configuring Python. I use Python for most of the rest and Powershell only where necessary. Contrary to what my comment may suggest, I am actually a software minimalist. A large part of my setup involves removing clutter from the desktop environment. I have avoided cygwin partially just so I'm not scared of native windows command line, which I find not quite as bad as Powershell to work with, but more limited in interacting with the OS of course. I am familiar with basic linux Bash already and that's what I'm comparing all of this to. Maybe now is a good time to adopt cygwin because, well, better late than never.
- barrkel 9y ago
  Getting Cygwin set up right takes a little bit of work - it's much easier if you're familiar with configuring a Linux shell and terminal to taste. Out of the box, it's a bit rough; kind of similar to how Emacs is a bit rough. I am scared of the Windows command line - not because it's scary, but because it's so limited. I feel almost helpless, all my tools are missing, it's just so awful. In practice, the reason I use Windows is mostly games, but it's rare I fire them up these days - I have to upgrade and/or re-verify my Steam account almost every time I start the client. Outside of that the browser and the terminal are almost all I use, and I can have them just as easily elsewhere. My point being my experience may not translate to your environment quite as well. Registry in Cygwin is mounted under /proc/registry: ~$ ls -l /proc/registry/HKEY_CURRENT_USER/Software/Microsoft/Windows total 0 dr-xr-x--- 72 SYSTEM SYSTEM 0 Dec 16 23:57 CurrentVersion dr-xr-x--- 2 SYSTEM SYSTEM 0 Dec 13 01:58 DWM dr-xr-x--- 4 SYSTEM SYSTEM 0 Dec 13 01:48 PrivacySettingsBeforeCreatorsUpdate dr-xr-x--- 6 SYSTEM SYSTEM 0 Dec 13 01:52 Shell dr-xr-x--- 4 SYSTEM SYSTEM 0 Dec 13 01:47 TabletPC dr-xr-x--- 3 SYSTEM SYSTEM 0 Dec 13 01:47 'Windows Error Reporting' dr-xr----- 3 barrkel None 0 Dec 13 01:58 Winlogon It's read-only however. Cygwin `regtool` lets you poke values in.

deleted 9y ago

[deleted]