4 ms·

Is this based on experience..? I would disagree.. The pipeline is powerful, but it's slow. There's a meaningful performance gain I've seen with foreach(), and

Is this based on experience..? I would disagree.. The pipeline is powerful, but it's slow. There's a meaningful performance gain I've seen with foreach(), and even more so when loops need to be nested. I believe the speed increase is because the runtime/clr/jit can infer the types contained within the collection before it actually starts executing the statement. yes, using generic collections with foreach helps as well, because (i think) powershell can completely avoid redetermining what every type of object is at execution time (as the collection is generic underneath). The downside is readability and memory usage as you pointed out, but i could argue that piping to foreach-object can also be painful to read, and many times the loop could be refactored into a proper function and given a meaningful name....

da_chicken 9y ago
Yes, based on my experience it really isn't worth bothering with. Compare: Measure-Command { # ~4,000 files $Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse foreach ($file in $Files) { $file.LastWriteTime.Date; } } | Select-Object -Property TotalMilliseconds Measure-Command { Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object { $_.LastWriteTime.Date; } } | Select-Object -Property TotalMilliseconds On my system and run hot (i.e., after running both multiple times), the foreach statement takes 790-855 ms. The ForEach-Object takes 860-890 ms. That's a 10% cost at most, and this particular operation is trivial. Is that "meaningful"? In some senses, yes, because 10% is a lot, but realistically, no because the scripts both run in less than a second. I'm not writing an application here. However, let's take something decidedly non-trivial: Measure-Command { # ~4,000 files $Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse $Algorithm = @('MD5','SHA1','SHA256') foreach ($file in $Files) { Get-FileHash -Path $file.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]); } } | Select-Object -Property TotalMilliseconds Measure-Command { $Algorithm = @('MD5','SHA1','SHA256') Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object { Get-FileHash -Path $_.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]); } } | Select-Object -Property TotalMilliseconds Now, run hot, foreach takes an average of 20.3 seconds over three runs. ForEach-Object takes an average of 20.8 seconds over three runs. Run cold (i.e., after a reboot), they both take about 63-65 seconds. This is on a slightly aging laptop with a spinning metal disk. Most of my workloads involve calling commands that involve significant time like Get-FileHash does. I'm doing things like splitting PDFs based on content and inserting them into an SQL database, or fetching 8,000 records from Active Directory and verifying file share permissions. I have found that for nested loops that I tend to use a ForEach-Object in the pipeline and then use foreach statements for operations on arrayed properties in the object and that works well, but for the outer loop, no, it's not usually worth the time to refactor or eliminate the pipeline.
- juststeve 9y ago
  ok, you've partially convinced me :) i mean, if the code is 90% I/O bound i agree: micro-optimisations can be a waste of time. But tight, nested loops that are only memory bound i think is worth the effort.. so for me, if i need need to do a lot of field comparisons for 300k accounts or groups from AD, i will download the ad data into memory and use foreach(), hashtables and generic collections wherever possible.. i actually think were making the same point anyway. I now reach for C# before powershell for situations like above where performance matters, but powershell does have advantages for smashing out some quick work. But i honestly cringe when i hear people talk about powershell scripts that take minutes to execute for tasks that should be reasonably quick.
  - da_chicken 9y ago
    No, I'm right there with you. Almost all my tasks are I/O bound by local disks, network disks, or some form of network data store. The more I get into PowerShell and begin using complex tasks, the more I find I'm really just writing C# code. And, yes, generally the easiest path to faster performance is just loading everything into memory and working with it there. In my PDF script I need to validate that the ID numbers are being read are valid, so I pull all 20,000 of them and put them into a HashSet<String>. That's extremely fast to validate with compared to an array and performs a bit better than a HashTable. There are times when I need to use an SqlDataReader or StreamReader and read row by row or line by line (4 GB+ text files suck), but that's only when memory has become a problem. I've been meaning to really dig into the High Performance PowerShell with LINQ article (https://www.red-gate.com/simple-talk/dotnet/net-framework/high-performance-powershell-linq/ https://www.red-gate.com/simple-talk/dotnet/net-framework/hi...), but I just don't have a need for it at present unless I refactor an existing script and they all work great.
    - juststeve 9y ago
      Yep, it really depends on what you're trying to do hey.. I mean, fancy powershell can't out perform good SQL either if the data is already sitting in a database... Thanks for the link though, those LINQ queries are super fast. i would like to incorporate this into some code at work, but dont quite have the need for it right at this moment.

da_chicken 9y ago

Yes, based on my experience it really isn't worth bothering with. Compare: Measure-Command { # ~4,000 files $Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse foreach ($file in $Files) { $file.LastWriteTime.Date; } } | Select-Object -Property TotalMilliseconds Measure-Command { Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object { $_.LastWriteTime.Date; } } | Select-Object -Property TotalMilliseconds On my system and run hot (i.e., after running both multiple times), the foreach statement takes 790-855 ms. The ForEach-Object takes 860-890 ms. That's a 10% cost at most, and this particular operation is trivial. Is that "meaningful"? In some senses, yes, because 10% is a lot, but realistically, no because the scripts both run in less than a second. I'm not writing an application here. However, let's take something decidedly non-trivial: Measure-Command { # ~4,000 files $Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse $Algorithm = @('MD5','SHA1','SHA256') foreach ($file in $Files) { Get-FileHash -Path $file.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]); } } | Select-Object -Property TotalMilliseconds Measure-Command { $Algorithm = @('MD5','SHA1','SHA256') Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object { Get-FileHash -Path $_.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]); } } | Select-Object -Property TotalMilliseconds Now, run hot, foreach takes an average of 20.3 seconds over three runs. ForEach-Object takes an average of 20.8 seconds over three runs. Run cold (i.e., after a reboot), they both take about 63-65 seconds. This is on a slightly aging laptop with a spinning metal disk. Most of my workloads involve calling commands that involve significant time like Get-FileHash does. I'm doing things like splitting PDFs based on content and inserting them into an SQL database, or fetching 8,000 records from Active Directory and verifying file share permissions. I have found that for nested loops that I tend to use a ForEach-Object in the pipeline and then use foreach statements for operations on arrayed properties in the object and that works well, but for the outer loop, no, it's not usually worth the time to refactor or eliminate the pipeline.

juststeve 9y ago
ok, you've partially convinced me :) i mean, if the code is 90% I/O bound i agree: micro-optimisations can be a waste of time. But tight, nested loops that are only memory bound i think is worth the effort.. so for me, if i need need to do a lot of field comparisons for 300k accounts or groups from AD, i will download the ad data into memory and use foreach(), hashtables and generic collections wherever possible.. i actually think were making the same point anyway. I now reach for C# before powershell for situations like above where performance matters, but powershell does have advantages for smashing out some quick work. But i honestly cringe when i hear people talk about powershell scripts that take minutes to execute for tasks that should be reasonably quick.
- da_chicken 9y ago
  No, I'm right there with you. Almost all my tasks are I/O bound by local disks, network disks, or some form of network data store. The more I get into PowerShell and begin using complex tasks, the more I find I'm really just writing C# code. And, yes, generally the easiest path to faster performance is just loading everything into memory and working with it there. In my PDF script I need to validate that the ID numbers are being read are valid, so I pull all 20,000 of them and put them into a HashSet<String>. That's extremely fast to validate with compared to an array and performs a bit better than a HashTable. There are times when I need to use an SqlDataReader or StreamReader and read row by row or line by line (4 GB+ text files suck), but that's only when memory has become a problem. I've been meaning to really dig into the High Performance PowerShell with LINQ article (https://www.red-gate.com/simple-talk/dotnet/net-framework/high-performance-powershell-linq/ https://www.red-gate.com/simple-talk/dotnet/net-framework/hi...), but I just don't have a need for it at present unless I refactor an existing script and they all work great.
  - juststeve 9y ago
    Yep, it really depends on what you're trying to do hey.. I mean, fancy powershell can't out perform good SQL either if the data is already sitting in a database... Thanks for the link though, those LINQ queries are super fast. i would like to incorporate this into some code at work, but dont quite have the need for it right at this moment.