5 ms·

In my profressional life I have cause to work with Powershell a great deal, and I love the object oriented pipelines that it has. Being able to filter, sort, o

In my profressional life I have cause to work with Powershell a great deal, and I love the object oriented pipelines that it has. Being able to filter, sort, or transform properties of the objects passed over makes it much more straightforward to work with than bash et al.

brazzledazzle 3y ago
I’m with you here. Once you get the hang of it you even miss the most awkward pipelines once they’re gone and you’re just passing around single dimensional strings. The memory overhead with sets of objects and their properties vs vanilla strings is significant though and easy to bump up against when working with large data sets. Best to try to keep all of the processing for that dataset in a single pipeline if you can. But that’s tradeoffs.
- frameset 3y ago
  Yeah the memory usage of the pipeline is real. Often using foreach instead to save memory is needed. Or good old fashioned array chunking.
  - brazzledazzle 3y ago
    Exactly. If you really have to you can take advantage of its ability to use .net libraries directly and force immediate cleanup/gc of variables but it's super hacky and always found pipelining to work better once you get the hang of processing data in flight.
- shortrounddev2 3y ago
  Personally I only use shell scripting for simple tasks, not anything that would run into a memory limit. If it requires more heavy lifting, I'll write it in a programming language like C#. Nothing we do in production relies on shell scripts, they exist only as shortcuts for our workflow
  - brazzledazzle 3y ago
    I think that's a good rule of thumb but not universally applicable. Powershell is used a lot with Microsoft sysadmin/identity/mail software and other vendor software used to manage or integrate with said MS software. If you're processing tens of thousands of AD objects you can easily hit memory issues. You could have your staff learn a few best practices and continue using the wealth of existing tooling/knowledge that exists for Powershell or write a bunch of custom .net code for every simple integration or job they want to run to process the objects in their directory.
    - shortrounddev2 3y ago
      Personally we use Linux boxes for everything remote, and I just develop on Windows locally
pmarreck 3y ago
There's a lot of people trying to copy this to the Linux space. NuShell comes to mind: https://www.nushell.sh/ https://www.nushell.sh/ I had an idea where I'd make wrappers for all the popular commands that would accept JSON structured data instead of raw binary data and "do the right thing". And that way you could take advantage of it without having to change your shell tooling. And they'd all accept an extra argument called either --structured-out or --unstructured-out which would either emit JSON or render the output back to a "flat" string, as appropriate.

brazzledazzle 3y ago

I’m with you here. Once you get the hang of it you even miss the most awkward pipelines once they’re gone and you’re just passing around single dimensional strings. The memory overhead with sets of objects and their properties vs vanilla strings is significant though and easy to bump up against when working with large data sets. Best to try to keep all of the processing for that dataset in a single pipeline if you can. But that’s tradeoffs.

frameset 3y ago
Yeah the memory usage of the pipeline is real. Often using foreach instead to save memory is needed. Or good old fashioned array chunking.
- brazzledazzle 3y ago
  Exactly. If you really have to you can take advantage of its ability to use .net libraries directly and force immediate cleanup/gc of variables but it's super hacky and always found pipelining to work better once you get the hang of processing data in flight.
shortrounddev2 3y ago
Personally I only use shell scripting for simple tasks, not anything that would run into a memory limit. If it requires more heavy lifting, I'll write it in a programming language like C#. Nothing we do in production relies on shell scripts, they exist only as shortcuts for our workflow
- brazzledazzle 3y ago
  I think that's a good rule of thumb but not universally applicable. Powershell is used a lot with Microsoft sysadmin/identity/mail software and other vendor software used to manage or integrate with said MS software. If you're processing tens of thousands of AD objects you can easily hit memory issues. You could have your staff learn a few best practices and continue using the wealth of existing tooling/knowledge that exists for Powershell or write a bunch of custom .net code for every simple integration or job they want to run to process the objects in their directory.
  - shortrounddev2 3y ago
    Personally we use Linux boxes for everything remote, and I just develop on Windows locally

pmarreck 3y ago

There's a lot of people trying to copy this to the Linux space. NuShell comes to mind: https://www.nushell.sh/ https://www.nushell.sh/ I had an idea where I'd make wrappers for all the popular commands that would accept JSON structured data instead of raw binary data and "do the right thing". And that way you could take advantage of it without having to change your shell tooling. And they'd all accept an extra argument called either --structured-out or --unstructured-out which would either emit JSON or render the output back to a "flat" string, as appropriate.