3 ms·

a few years ago I had to write a data integrity check for tens-of-thousands of files. Just crawl the dir and compute filename, size, checksum, maybe date, can't

a few years ago I had to write a data integrity check for tens-of-thousands of files. Just crawl the dir and compute filename, size, checksum, maybe date, can't remember. I thought "ooh neat, I bet I can use some powershell trick to compute that tuple for each file", and no. Immediately ran into some obscure limitation with their "pipe" - I can't remember the details, it was like impossible to create many-things from a single-thing - and thought "how disappointing. the one time I give it an honest shake to do something entirely conceivable by the creator and it falls flat on its face.".

latkin 2y ago
Nothing in your description sounds difficult to do in powershell. You can certainly output "many-things" from a part of the pipeline that takes "single-things" as input. Crawling files is a single command, then you can do whatever you want with each one in the next part of the pipeline - "map" from file info object to something else (e.g. custom object with filename, size, checksum, etc props) 1-1, multiplex each file into N output objects, buffer file inputs until some heuristic is met then emit outputs, etc.
jodrellblank 2y ago
A lot of years ago, circa Exchange 2010, there was a limitation on pipelines-within-pipelines which had to be worked around. Files already include name, size, dates, so select all of those and add a calculated property of the file hash using Get-FileHash (MD5 and SHA options): gci | select *, @{L='Hash'; E={ ($_ | Get-FileHash).Hash }} Get-ChildItem | Select-Object -Property FullName, CreationTime, Length, @{Label='Hash'; Expression={ ($_ | get-filehash).Hash}} but even then you could do what you want with a traditional loop and no pipeline: $results = foreach ($file in get-childitem) { # a hashtable of things you want to be # in each object ('row') of the output: $data = @{ Name = $file.Name SizeGB = $file.Length / 1GB Hash = ($file | Get-FileHash).Hash } [pscustomobject]$data }
- gosub100 2y ago
  Thanks for taking my belly aching memory and being constructive. I just vividly remembered the sour grapes taste but I guess it was just due to my incompetence. I think what I didn't know was how to do the nested sub command for Hash. Cheers

latkin 2y ago

Nothing in your description sounds difficult to do in powershell. You can certainly output "many-things" from a part of the pipeline that takes "single-things" as input. Crawling files is a single command, then you can do whatever you want with each one in the next part of the pipeline - "map" from file info object to something else (e.g. custom object with filename, size, checksum, etc props) 1-1, multiplex each file into N output objects, buffer file inputs until some heuristic is met then emit outputs, etc.

jodrellblank 2y ago

A lot of years ago, circa Exchange 2010, there was a limitation on pipelines-within-pipelines which had to be worked around. Files already include name, size, dates, so select all of those and add a calculated property of the file hash using Get-FileHash (MD5 and SHA options): gci | select *, @{L='Hash'; E={ ($_ | Get-FileHash).Hash }} Get-ChildItem | Select-Object -Property FullName, CreationTime, Length, @{Label='Hash'; Expression={ ($_ | get-filehash).Hash}} but even then you could do what you want with a traditional loop and no pipeline: $results = foreach ($file in get-childitem) { # a hashtable of things you want to be # in each object ('row') of the output: $data = @{ Name = $file.Name SizeGB = $file.Length / 1GB Hash = ($file | Get-FileHash).Hash } [pscustomobject]$data }

gosub100 2y ago
Thanks for taking my belly aching memory and being constructive. I just vividly remembered the sour grapes taste but I guess it was just due to my incompetence. I think what I didn't know was how to do the nested sub command for Hash. Cheers