5 ms·

PowerShell is slow, but writing to files the way he is doing is just a bad pattern. He uses Add-Content not just in loops, but in nested loops. The problem wi

PowerShell is slow, but writing to files the way he is doing is just a bad pattern. He uses Add-Content not just in loops, but in nested loops. The problem with that is that each time Add-Content is called the system opens the file, adds the content, flushes to disk, and then closes the file. That's a lot of overhead. In his wine glass script he also uses the PowerShell patten of creating an array ($x = @()) and then appending to it (foreach ($i in $set) { $x += $value * $i }). The problem here is that PowerShell arrays are fixed in size. To append a value to an array, the system creates a new array, copies all the values over with the new one, and then disposes of the old array. It works fine up to about 100 items, but it gets noticeably slow after that. Since it's done with a loop it makes it somewhere between O(n log n) and O(n^2). It's better to just output all the values as a single array ($x = $set | ForEach { $value * $_ }) or to create an ArrayList or List<String> or some C# collection that supports an O(1) append. He also assigns a lot of variables in his loops and then uses them only once to format a string. He could eliminate those variables and just embed the expressions in the strings. I got a ~25% performance improvement (210 ms to under 140 ms) with just the script embedded in the article when I switched to StringBuilder instead of Add-Content, and that script doesn't have the poor array pattern. StreamWriter would work, too, with less memory pressure than a StringBuilder. I suspect that with better code you could easily get this down below 3 seconds.

lburton 9y ago
I got a lot more than a 25% improvement.. you seem to have a much quicker machine or perhaps you were running one of the earlier scripts? For the wine glass script it went from >2 minutes to 1.5 seconds when output buffered. https://github.com/cosmosdarwin/obj-in-powershell/pull/1 https://github.com/cosmosdarwin/obj-in-powershell/pull/1 Also for those that haven't come across it https://github.com/dlwyatt/PowershellProfiler https://github.com/dlwyatt/PowershellProfiler is pretty useful.. despite add-content being the obvious offender here..
- da_chicken 9y ago
  I have a 10 year old PC. The 210ms to 140 ms time was with the single cylinder script, which is much shorter, and the < 3 second prediction was for the wine glass script. I just guessed by looking that it wouldn't be hard to improve by an order of magnitude. Sorry if I wasn't clear!
- juststeve 9y ago
  get-content is very slow as well
throwaway7645 9y ago
Thanks for the analysis. This is one of my main PS issues in that all the obvious ways are in fact wrong. I tried 3-4 different file write methods before and all were unbearably slow.
- ygra 9y ago
  I wouldn't say any obvious way is wrong. Dropping down to the .NET mechanisms should be more of a last resort if performance is really abysmal for some reason, but apart from that, common sense applies. As they said, opening, writing to, and closing a file repeatedly in a loop is a stupid idea. I wouldn't say that's an obvious way to do file I/O in PowerShell. On that note, what are the obvious ways you've tried?
  - throwaway7645 9y ago
    Agreed that having to drop to .NET is a fail. So usually when trying to learn to do file I/O in a new language you google it. The first few examples I came across were slow and when I looked into it, it was doing as this example did. It's been awhile, but it wasn't obvious how to do it with a stream without resorting to a C# esque version which at that point, you might as well use C#. I'm not sure why there isn't a PS oneliner option like Python's with statement. I really like the idea of PS, but some simple things are more trouble than they're worth.
    - juststeve 9y ago
      > Agreed that having to drop to .NET is a fail. So many times i run into performance walls with powershell, that calling the .net library directly is basically required.
    - ygra 9y ago
      Well, the most natural way would be to use the pipeline. Instead of writing to the file in the innermost ForEach-Object, just pipe the whole pipeline into Out-File. I'd say it's the most straightforward and natural way, considering that PowerShell is a shell.
juststeve 9y ago
piping to foreach-object is super slow as well, it's much faster to use foreach($x in $y) {}
- da_chicken 9y ago
  Here, yes, I agree that the foreach statements would be better, possibly even significantly, but in a general case the difference is usually not worth bothering with. While ForEach-Object is like an order of magnitude slower, we're talking like 100 ms vs 10 ms to iterate through 10,000 items. (https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/08/getting-to-know-foreach-and-foreach-object/ https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/...) In other words, you have to have a good number of objects to make foreach meaningfully faster that when you do, you often find that either a) lose time to powershell.exe allocating more system memory to store the collection, or b) the loop itself is two or three orders of magnitude slower already so it's ultimately a trivial optimization. Also, the foreach statement waits to allocate the entire set to memory ($y in your example) while ForEach-Object begins processing as soon as the first object comes through the pipeline. If you're using the output of a command for $y that returns output with some lag, you might find that ForEach-Object actually turns out to be faster because the first objects start being processed immediately. If you're returning data over a network or dealing with particularly large objects, ForEach-Object can be better. Finally, the foreach statement itself doesn't work with pipelines either as input or output, so it's not appropriate for a lot of scenarios.
  - juststeve 9y ago
    Is this based on experience..? I would disagree.. The pipeline is powerful, but it's slow. There's a meaningful performance gain I've seen with foreach(), and even more so when loops need to be nested. I believe the speed increase is because the runtime/clr/jit can infer the types contained within the collection before it actually starts executing the statement. yes, using generic collections with foreach helps as well, because (i think) powershell can completely avoid redetermining what every type of object is at execution time (as the collection is generic underneath). The downside is readability and memory usage as you pointed out, but i could argue that piping to foreach-object can also be painful to read, and many times the loop could be refactored into a proper function and given a meaningful name....

lburton 9y ago

I got a lot more than a 25% improvement.. you seem to have a much quicker machine or perhaps you were running one of the earlier scripts? For the wine glass script it went from >2 minutes to 1.5 seconds when output buffered. https://github.com/cosmosdarwin/obj-in-powershell/pull/1 https://github.com/cosmosdarwin/obj-in-powershell/pull/1 Also for those that haven't come across it https://github.com/dlwyatt/PowershellProfiler https://github.com/dlwyatt/PowershellProfiler is pretty useful.. despite add-content being the obvious offender here..

da_chicken 9y ago
I have a 10 year old PC. The 210ms to 140 ms time was with the single cylinder script, which is much shorter, and the < 3 second prediction was for the wine glass script. I just guessed by looking that it wouldn't be hard to improve by an order of magnitude. Sorry if I wasn't clear!
juststeve 9y ago
get-content is very slow as well

throwaway7645 9y ago

Thanks for the analysis. This is one of my main PS issues in that all the obvious ways are in fact wrong. I tried 3-4 different file write methods before and all were unbearably slow.

ygra 9y ago
I wouldn't say any obvious way is wrong. Dropping down to the .NET mechanisms should be more of a last resort if performance is really abysmal for some reason, but apart from that, common sense applies. As they said, opening, writing to, and closing a file repeatedly in a loop is a stupid idea. I wouldn't say that's an obvious way to do file I/O in PowerShell. On that note, what are the obvious ways you've tried?
- throwaway7645 9y ago
  Agreed that having to drop to .NET is a fail. So usually when trying to learn to do file I/O in a new language you google it. The first few examples I came across were slow and when I looked into it, it was doing as this example did. It's been awhile, but it wasn't obvious how to do it with a stream without resorting to a C# esque version which at that point, you might as well use C#. I'm not sure why there isn't a PS oneliner option like Python's with statement. I really like the idea of PS, but some simple things are more trouble than they're worth.
  - juststeve 9y ago
    > Agreed that having to drop to .NET is a fail. So many times i run into performance walls with powershell, that calling the .net library directly is basically required.
  - ygra 9y ago
    Well, the most natural way would be to use the pipeline. Instead of writing to the file in the innermost ForEach-Object, just pipe the whole pipeline into Out-File. I'd say it's the most straightforward and natural way, considering that PowerShell is a shell.

juststeve 9y ago

piping to foreach-object is super slow as well, it's much faster to use foreach($x in $y) {}

da_chicken 9y ago
Here, yes, I agree that the foreach statements would be better, possibly even significantly, but in a general case the difference is usually not worth bothering with. While ForEach-Object is like an order of magnitude slower, we're talking like 100 ms vs 10 ms to iterate through 10,000 items. (https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/08/getting-to-know-foreach-and-foreach-object/ https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/...) In other words, you have to have a good number of objects to make foreach meaningfully faster that when you do, you often find that either a) lose time to powershell.exe allocating more system memory to store the collection, or b) the loop itself is two or three orders of magnitude slower already so it's ultimately a trivial optimization. Also, the foreach statement waits to allocate the entire set to memory ($y in your example) while ForEach-Object begins processing as soon as the first object comes through the pipeline. If you're using the output of a command for $y that returns output with some lag, you might find that ForEach-Object actually turns out to be faster because the first objects start being processed immediately. If you're returning data over a network or dealing with particularly large objects, ForEach-Object can be better. Finally, the foreach statement itself doesn't work with pipelines either as input or output, so it's not appropriate for a lot of scenarios.
- juststeve 9y ago
  Is this based on experience..? I would disagree.. The pipeline is powerful, but it's slow. There's a meaningful performance gain I've seen with foreach(), and even more so when loops need to be nested. I believe the speed increase is because the runtime/clr/jit can infer the types contained within the collection before it actually starts executing the statement. yes, using generic collections with foreach helps as well, because (i think) powershell can completely avoid redetermining what every type of object is at execution time (as the collection is generic underneath). The downside is readability and memory usage as you pointed out, but i could argue that piping to foreach-object can also be painful to read, and many times the loop could be refactored into a proper function and given a meaningful name....