5 ms·

That's nice. Can you pipe binary data without powershell corrupting it, or is that now a feature? https://brianreiter.org/2010/01/29/powershells-object-pipelin

That's nice. Can you pipe binary data without powershell corrupting it, or is that now a feature? https://brianreiter.org/2010/01/29/powershells-object-pipeline-corrupts-piped-binary-data/ https://brianreiter.org/2010/01/29/powershells-object-pipeli...

gfody 10y ago
Related, running something like decrypt hugefile.dat | encode outfile.dat will usually result in powershell using up all your system ram and crashing.
- brazzledazzle 10y ago
  On a somewhat related note, one of PowerShell's biggest weaknesses is memory utilization. And since they make it so simple to pass around hefty object collections it tends to bite you in the ass early on. Particularly when you're using cmdlets. When doing real scripting that actually needs to be run as a job I am surprised when I can get away with cmdlets and don't need to manage .net APIs/objects directly. Especially third party cmdlets. One performance tip I picked up early on: If you're passing around large object sets you need to operate on try to keep them in a pipeline. Populating a variable and interacting with it is arguably more readable and maintainable, especially for junior people, but it has a heavy cost in terms of memory. That said every version of posh is better than the last so I may be wrong or overestimating the impact for the newest versions. YMMV, don't believe everything you read on the internet, etc.
  - youdontknowtho 10y ago
    That's great advice, man. Cheers.
  - JeffreySnover 10y ago
    That is a good tip. There are certain product teams that produced ENORMOUS objects and your suggestion helps when dealing with that. Jeffrey Snover [MSFT]
    - brazzledazzle 10y ago
      Upon reading that again maybe I should have said it's weakness is that it's easy for everyone involved to create objects that use a lot of memory. I'm not sure it's fair to say PowerShell itself is bad at utilizing or managing memory. It's more of a possible negative side effect of one of its great strengths.
- UK-AL 10y ago
  Most likely because it loads it all into memory, rather than say streaming it.
  - copperx 10y ago
    That's how I would expect MSFT to build it.
brainfire 10y ago
Haven't tried in the preview build but this is still the case in the recent Windows 10 build I have.
pjc50 10y ago
It's amazing, isn't it. They went to the effort of building a typed shell but didn't realise there are multiple different types of string. I bet it still truncates output of the ">" operator to the console width, too.
- new_hackers 10y ago
  this ^^^ This is the main reason I still use cmd.exe
- deleted 10y ago
  [deleted]
- jodrellblank 10y ago
  'troll' * 1000 > out.txt Open in NotePad++, no weird newlines in the wrong place, content not truncated. Were you using |format-table and |format-list (ft, fl) to explicitly format the text for console viewing, then redirecting that to a file?
  - pjc50 10y ago
    Took a while to reproduce the actual case, and it's Select-String that's at fault. And it must be in "powershell" not "powershell ISE". 'troll' * 1000 > troll.txt .. works fine as you say Select-String -Path .\troll.txt -Pattern "t" > t.txt .. wraps the output at the console width. Which means you can't use it as a grep replacement.
    - jodrellblank 10y ago
      You can't use it as grep in bash, because it isn't that. It outputs [MatchInfo] objects, not text, and Out-File (>) formats complex objects for console viewing (for ??? reasons), which is one reason > isn't a great PS habit. Select-String -Path .\troll.txt -Pattern "t" | Set-Content t.txt or sls 't' .\troll.txt | sc t.txt
ashmud 10y ago
This bit me badly when trying to pipe GPG output. I ended up building a command string and sending it to cmd. :|
- Tarean 10y ago
  That is odd, powershell shells out to cmd.exe if you try to pipe binary or text data. It goes back to powershell only if you use powershell commands in the pipeline. If you don't pass the flags zo handle it as binary in those cases stuff breaks.
melolife 10y ago
I found out yesterday that `Select-String`, PowerShell's "grep equivalent" expands tab characters to spaces.
jodrellblank 10y ago
Get-Content test.jpg -Encoding Byte -ReadCount 1000 | Set-Content test2.jpg -Encoding Byte Piped binary data, not corrupted, since a long time before that blog post was written. What you can't do is run native commands, and read their standard output as a byte stream without going through .Net and Start-Process.
1wd 10y ago
Was super fun to learn about this when restoring the "backup" dumped via pipe in a powershell script.
- nolok 10y ago
  While the first rule of backup is "do them", the second rule is "test them". If your backups are bad and you only notice it when it's time to restore, you only have yourself to blame. "The backups fail to restore" is such a common occurrence, it's really scary.
elchief 10y ago
Powershell drove me to Mac

gfody 10y ago

Related, running something like decrypt hugefile.dat | encode outfile.dat will usually result in powershell using up all your system ram and crashing.

brazzledazzle 10y ago
On a somewhat related note, one of PowerShell's biggest weaknesses is memory utilization. And since they make it so simple to pass around hefty object collections it tends to bite you in the ass early on. Particularly when you're using cmdlets. When doing real scripting that actually needs to be run as a job I am surprised when I can get away with cmdlets and don't need to manage .net APIs/objects directly. Especially third party cmdlets. One performance tip I picked up early on: If you're passing around large object sets you need to operate on try to keep them in a pipeline. Populating a variable and interacting with it is arguably more readable and maintainable, especially for junior people, but it has a heavy cost in terms of memory. That said every version of posh is better than the last so I may be wrong or overestimating the impact for the newest versions. YMMV, don't believe everything you read on the internet, etc.
- youdontknowtho 10y ago
  That's great advice, man. Cheers.
- JeffreySnover 10y ago
  That is a good tip. There are certain product teams that produced ENORMOUS objects and your suggestion helps when dealing with that. Jeffrey Snover [MSFT]
  - brazzledazzle 10y ago
    Upon reading that again maybe I should have said it's weakness is that it's easy for everyone involved to create objects that use a lot of memory. I'm not sure it's fair to say PowerShell itself is bad at utilizing or managing memory. It's more of a possible negative side effect of one of its great strengths.
UK-AL 10y ago
Most likely because it loads it all into memory, rather than say streaming it.
- copperx 10y ago
  That's how I would expect MSFT to build it.

brainfire 10y ago

Haven't tried in the preview build but this is still the case in the recent Windows 10 build I have.

pjc50 10y ago

It's amazing, isn't it. They went to the effort of building a typed shell but didn't realise there are multiple different types of string. I bet it still truncates output of the ">" operator to the console width, too.

new_hackers 10y ago
this ^^^ This is the main reason I still use cmd.exe
deleted 10y ago
[deleted]
jodrellblank 10y ago
'troll' * 1000 > out.txt Open in NotePad++, no weird newlines in the wrong place, content not truncated. Were you using |format-table and |format-list (ft, fl) to explicitly format the text for console viewing, then redirecting that to a file?
- pjc50 10y ago
  Took a while to reproduce the actual case, and it's Select-String that's at fault. And it must be in "powershell" not "powershell ISE". 'troll' * 1000 > troll.txt .. works fine as you say Select-String -Path .\troll.txt -Pattern "t" > t.txt .. wraps the output at the console width. Which means you can't use it as a grep replacement.
  - jodrellblank 10y ago
    You can't use it as grep in bash, because it isn't that. It outputs [MatchInfo] objects, not text, and Out-File (>) formats complex objects for console viewing (for ??? reasons), which is one reason > isn't a great PS habit. Select-String -Path .\troll.txt -Pattern "t" | Set-Content t.txt or sls 't' .\troll.txt | sc t.txt

ashmud 10y ago

This bit me badly when trying to pipe GPG output. I ended up building a command string and sending it to cmd. :|

Tarean 10y ago
That is odd, powershell shells out to cmd.exe if you try to pipe binary or text data. It goes back to powershell only if you use powershell commands in the pipeline. If you don't pass the flags zo handle it as binary in those cases stuff breaks.

melolife 10y ago

I found out yesterday that `Select-String`, PowerShell's "grep equivalent" expands tab characters to spaces.

jodrellblank 10y ago

Get-Content test.jpg -Encoding Byte -ReadCount 1000 | Set-Content test2.jpg -Encoding Byte Piped binary data, not corrupted, since a long time before that blog post was written. What you can't do is run native commands, and read their standard output as a byte stream without going through .Net and Start-Process.

1wd 10y ago

Was super fun to learn about this when restoring the "backup" dumped via pipe in a powershell script.

nolok 10y ago
While the first rule of backup is "do them", the second rule is "test them". If your backups are bad and you only notice it when it's time to restore, you only have yourself to blame. "The backups fail to restore" is such a common occurrence, it's really scary.

elchief 10y ago

Powershell drove me to Mac