4 ms·

piping to foreach-object is super slow as well, it's much faster to use foreach($x in $y) {}

da_chicken 9y ago
Here, yes, I agree that the foreach statements would be better, possibly even significantly, but in a general case the difference is usually not worth bothering with. While ForEach-Object is like an order of magnitude slower, we're talking like 100 ms vs 10 ms to iterate through 10,000 items. (https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/08/getting-to-know-foreach-and-foreach-object/ https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/...) In other words, you have to have a good number of objects to make foreach meaningfully faster that when you do, you often find that either a) lose time to powershell.exe allocating more system memory to store the collection, or b) the loop itself is two or three orders of magnitude slower already so it's ultimately a trivial optimization. Also, the foreach statement waits to allocate the entire set to memory ($y in your example) while ForEach-Object begins processing as soon as the first object comes through the pipeline. If you're using the output of a command for $y that returns output with some lag, you might find that ForEach-Object actually turns out to be faster because the first objects start being processed immediately. If you're returning data over a network or dealing with particularly large objects, ForEach-Object can be better. Finally, the foreach statement itself doesn't work with pipelines either as input or output, so it's not appropriate for a lot of scenarios.
- juststeve 9y ago
  Is this based on experience..? I would disagree.. The pipeline is powerful, but it's slow. There's a meaningful performance gain I've seen with foreach(), and even more so when loops need to be nested. I believe the speed increase is because the runtime/clr/jit can infer the types contained within the collection before it actually starts executing the statement. yes, using generic collections with foreach helps as well, because (i think) powershell can completely avoid redetermining what every type of object is at execution time (as the collection is generic underneath). The downside is readability and memory usage as you pointed out, but i could argue that piping to foreach-object can also be painful to read, and many times the loop could be refactored into a proper function and given a meaningful name....
  - da_chicken 9y ago
    Yes, based on my experience it really isn't worth bothering with. Compare: Measure-Command { # ~4,000 files $Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse foreach ($file in $Files) { $file.LastWriteTime.Date; } } | Select-Object -Property TotalMilliseconds Measure-Command { Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object { $_.LastWriteTime.Date; } } | Select-Object -Property TotalMilliseconds On my system and run hot (i.e., after running both multiple times), the foreach statement takes 790-855 ms. The ForEach-Object takes 860-890 ms. That's a 10% cost at most, and this particular operation is trivial. Is that "meaningful"? In some senses, yes, because 10% is a lot, but realistically, no because the scripts both run in less than a second. I'm not writing an application here. However, let's take something decidedly non-trivial: Measure-Command { # ~4,000 files $Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse $Algorithm = @('MD5','SHA1','SHA256') foreach ($file in $Files) { Get-FileHash -Path $file.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]); } } | Select-Object -Property TotalMilliseconds Measure-Command { $Algorithm = @('MD5','SHA1','SHA256') Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object { Get-FileHash -Path $_.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]); } } | Select-Object -Property TotalMilliseconds Now, run hot, foreach takes an average of 20.3 seconds over three runs. ForEach-Object takes an average of 20.8 seconds over three runs. Run cold (i.e., after a reboot), they both take about 63-65 seconds. This is on a slightly aging laptop with a spinning metal disk. Most of my workloads involve calling commands that involve significant time like Get-FileHash does. I'm doing things like splitting PDFs based on content and inserting them into an SQL database, or fetching 8,000 records from Active Directory and verifying file share permissions. I have found that for nested loops that I tend to use a ForEach-Object in the pipeline and then use foreach statements for operations on arrayed properties in the object and that works well, but for the outer loop, no, it's not usually worth the time to refactor or eliminate the pipeline.
    - juststeve 9y ago
      ok, you've partially convinced me :) i mean, if the code is 90% I/O bound i agree: micro-optimisations can be a waste of time. But tight, nested loops that are only memory bound i think is worth the effort.. so for me, if i need need to do a lot of field comparisons for 300k accounts or groups from AD, i will download the ad data into memory and use foreach(), hashtables and generic collections wherever possible.. i actually think were making the same point anyway. I now reach for C# before powershell for situations like above where performance matters, but powershell does have advantages for smashing out some quick work. But i honestly cringe when i hear people talk about powershell scripts that take minutes to execute for tasks that should be reasonably quick.

Here, yes, I agree that the foreach statements would be better, possibly even significantly, but in a general case the difference is usually not worth bothering with. While ForEach-Object is like an order of magnitude slower, we're talking like 100 ms vs 10 ms to iterate through 10,000 items. (https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/08/getting-to-know-foreach-and-foreach-object/ https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/...) In other words, you have to have a good number of objects to make foreach meaningfully faster that when you do, you often find that either a) lose time to powershell.exe allocating more system memory to store the collection, or b) the loop itself is two or three orders of magnitude slower already so it's ultimately a trivial optimization. Also, the foreach statement waits to allocate the entire set to memory ($y in your example) while ForEach-Object begins processing as soon as the first object comes through the pipeline. If you're using the output of a command for $y that returns output with some lag, you might find that ForEach-Object actually turns out to be faster because the first objects start being processed immediately. If you're returning data over a network or dealing with particularly large objects, ForEach-Object can be better. Finally, the foreach statement itself doesn't work with pipelines either as input or output, so it's not appropriate for a lot of scenarios.

juststeve 9y ago
Is this based on experience..? I would disagree.. The pipeline is powerful, but it's slow. There's a meaningful performance gain I've seen with foreach(), and even more so when loops need to be nested. I believe the speed increase is because the runtime/clr/jit can infer the types contained within the collection before it actually starts executing the statement. yes, using generic collections with foreach helps as well, because (i think) powershell can completely avoid redetermining what every type of object is at execution time (as the collection is generic underneath). The downside is readability and memory usage as you pointed out, but i could argue that piping to foreach-object can also be painful to read, and many times the loop could be refactored into a proper function and given a meaningful name....
- da_chicken 9y ago
  Yes, based on my experience it really isn't worth bothering with. Compare: Measure-Command { # ~4,000 files $Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse foreach ($file in $Files) { $file.LastWriteTime.Date; } } | Select-Object -Property TotalMilliseconds Measure-Command { Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object { $_.LastWriteTime.Date; } } | Select-Object -Property TotalMilliseconds On my system and run hot (i.e., after running both multiple times), the foreach statement takes 790-855 ms. The ForEach-Object takes 860-890 ms. That's a 10% cost at most, and this particular operation is trivial. Is that "meaningful"? In some senses, yes, because 10% is a lot, but realistically, no because the scripts both run in less than a second. I'm not writing an application here. However, let's take something decidedly non-trivial: Measure-Command { # ~4,000 files $Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse $Algorithm = @('MD5','SHA1','SHA256') foreach ($file in $Files) { Get-FileHash -Path $file.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]); } } | Select-Object -Property TotalMilliseconds Measure-Command { $Algorithm = @('MD5','SHA1','SHA256') Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object { Get-FileHash -Path $_.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]); } } | Select-Object -Property TotalMilliseconds Now, run hot, foreach takes an average of 20.3 seconds over three runs. ForEach-Object takes an average of 20.8 seconds over three runs. Run cold (i.e., after a reboot), they both take about 63-65 seconds. This is on a slightly aging laptop with a spinning metal disk. Most of my workloads involve calling commands that involve significant time like Get-FileHash does. I'm doing things like splitting PDFs based on content and inserting them into an SQL database, or fetching 8,000 records from Active Directory and verifying file share permissions. I have found that for nested loops that I tend to use a ForEach-Object in the pipeline and then use foreach statements for operations on arrayed properties in the object and that works well, but for the outer loop, no, it's not usually worth the time to refactor or eliminate the pipeline.
  - juststeve 9y ago
    ok, you've partially convinced me :) i mean, if the code is 90% I/O bound i agree: micro-optimisations can be a waste of time. But tight, nested loops that are only memory bound i think is worth the effort.. so for me, if i need need to do a lot of field comparisons for 300k accounts or groups from AD, i will download the ad data into memory and use foreach(), hashtables and generic collections wherever possible.. i actually think were making the same point anyway. I now reach for C# before powershell for situations like above where performance matters, but powershell does have advantages for smashing out some quick work. But i honestly cringe when i hear people talk about powershell scripts that take minutes to execute for tasks that should be reasonably quick.