12 ms·

After a number of years using PowerShell, my conclusion is the opposite: text "scraping" is just better for most cases. Normal shell usage means doing a lot of

• all2well 10y ago
  I'm not entirely convinced that "plain text" is "simple." For one, utf-8 is a variable-length encoding, which can cause all sorts of subtle bugs and sometimes leads to security issues because somebody failed to parse a character correctly somewhere. On top of this, using plain text means that every program has to choose its own control characters, essentially an encoding within an encoding. This is great for readability, but it's not always easy to know how a certain program will treat certain corner cases, or even really just at all. I'm not sure what you mean by "you can just look at the text," because unlike something that encapsulates functionality alongside the data, text input into a program might do anything.
  • userbinator 10y ago
    For one, utf-8 is a variable-length encoding, which can cause all sorts of subtle bugs and sometimes leads to security issues because somebody failed to parse a character correctly somewhere. UTF-8 is nice in that extended characters, despite consisting of multiple bytes, will never contain a low-ASCII character amongst them. Unless you're dealing with byte offsets, splitting and scanning UTF-8 strings by delimiters works just like ASCII. because unlike something that encapsulates functionality alongside the data, text input into a program might do anything. On the other hand, that "encapsulated functionality" is even more hidden and non-obvious. Unlike a stream of bytes that you can easily inspect simply by dumping them into a file, PowerShell objects are far more opaque entities.
    • Pyxl101 10y ago
      Is there no trivial way to print a PowerShell object? As JSON or XML or an ascii table formatted into columns? That seems like a point of friction, then, I'd agree.
      • DHowett 10y ago
        By default, an object at the end of a pipeline will be printed to the screen in tabular format. (australis) ~\Desktop % Get-ItemProperty e Directory: C:\Users\Dustin\Desktop Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 8/12/2016 9:06 AM e I believe types can specify which columns to display by default; if you want more info, there's always `Format-List`: (australis) ~\Desktop % Get-ItemProperty e | format-list Directory: C:\Users\Dustin\Desktop Name : e CreationTime : 8/11/2016 10:58:56 PM LastWriteTime : 8/12/2016 9:06:13 AM LastAccessTime : 8/12/2016 9:06:13 AM Mode : d----- LinkType : Target : {} It's also possible to format any object via `ConvertTo-JSON`: (australis) ~\Desktop % Get-ItemProperty e | ConvertTo-JSON { "Name": "e", "Parent": { "Name": "Desktop", "Parent": { "Name": "Dustin", "Parent": "Users", "Exists": true, "Root": "C:\\", "FullName": "C:\\Users\\Dustin", "Extension": "", ...
        • useerup 10y ago
          > I believe types can specify which columns to display by default; Indeed, the default view (table/list/custom) can be specified, and for the view the default properties can be specified. Consider how 'ls' produces a sequence of FileSystemInfo objects (when run against a disk file system). The view specifies that the list be grouped by "parent". On the terminal this gives a nice list a la: Directory: C:\Dell\Drivers\T7MFF\PhysX\files\Engine\v2.8.0 Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 12-12-2014 07:42 362232 NxCooking.dll -a---- 12-12-2014 07:42 5520120 PhysXCore.dll Directory: C:\Dell\Drivers\T7MFF\PhysX\files\Engine\v2.8.1 Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 12-12-2014 07:42 363256 NxCooking.dll -a---- 12-12-2014 07:42 5823736 PhysXCore.dll Directory: C:\Dell\Drivers\T7MFF\PhysX\files\Engine\v2.8.3 Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 12-12-2014 07:42 339192 PhysXCooking.dll -a---- 12-12-2014 07:42 412408 PhysXCooking64.dll -a---- 12-12-2014 07:42 5952248 PhysXCore64.dll However, the underlying stream is still just a continous stream of FileSystemInfo's. It is the terminal display format definition that causes it formatting to break and write a directory heading whenever the next item has a different parent from the previous one.
• beefield 10y ago
  > And the over-verbose syntax doesn't help This is basically the reason I have not been willing to put any time even trying to learn ps. I mean, trying to translate unzip file.zip folder To ps was a nigthmare. Like: [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath) I don't _want_ to learn to type stuff like that for simple shell scripts.
  • kuma4 10y ago
    Use this. Expand-Archive file.zip
  • ygra 10y ago
    But that's picking an example where a. You can still use the native binary just as well. Especially in this case. b. There was no PowerShell cmdlet for it at the time. There also isn't one for executing Python code, you still have to call python. Or, if you're so inclined, load up IronPython's assembly, and use an overly verbose .NET method call ... c. There now is a cmdlet for doing so: Expand-ZipFile. I'm actually curious as to those complaints: Do you never write a function in bash to abstract away something? A function in other languages? Is everything just a series of crystal-clear one-liners? Shell scripts in Unix-likes tend to glue together a bunch of other programs and (usually) not do much programming in the shell's language. You can write PowerShell just the same, actually. It is a shell, after all.
    • beefield 10y ago
      That was picking an example from my previous try to work with ps. I was not the admin of the server so no installation of extra programs. Or upgrading ps to the newest version. The shell scripting I have needed has been typically very simple. Like schedule to copy this file from that server and run this command.(what was precisely all that I was trying to achieve the last time, only the file was compressed. Would have done with .bat, but that was even more difficult) As you say, using it as a glue and build the complexity to the programs behind the shell.
    • userbinator 10y ago
      If you're used to cmd or bash, the jump in verbosity is very noticeable and definitely gets in the way. It might be a cultural thing, because I'm no fan of C# and its typical style either; no surprise then, that I find PS syntax exceedingly verbose too. It just feels excessively bureaucratic and awkward to have to write so much. On the other hand, bash, awk, sed, and all the typical Unix commands and its associated ecosystem seem like they "get out of your way" far more effectively. c. There now is a cmdlet for doing so: Expand-ZipFile. That example already shows the verbosity increase clearly - why is it "Expand-ZipFile", and not "ZipFile-Expand", "Expand-Zip-Format-Archive", or something else? In contrast, "unzip" is short and easy to remember. The fact that a native binary might exist for a given task is irrelevant to the observation that the shell's language is itself more verbose. I'm actually curious as to those complaints: Do you never write a function in bash to abstract away something? A function in other languages? Is everything just a series of crystal-clear one-liners? Abstraction helps reduce code duplication but is not useful when each line of the script is quite different, and in that case PS remains more verbose. Ultimately, the overhead is still higher.
• IshKebab 10y ago
  > Normal shell usage means doing a lot of one-shot pipelines incrementally. Sure because no-one ever takes those "one-shot" pipelines full of `cut`s and `grep`s and distributes them in real products. /s As far as I can tell, Powershell is designed for scripts that are meant to be distributed, so they must be robust. Normal 'text scraping' Bash is shit at that.
  • d0mine 10y ago
    If we are not talking about interactive use then to distribute "robust" scripts, you could use nicer and richer Perl, Python eco-systems. For interactive use, the conciseness, the human-readability of input/output and the fault-tolerance to unrelated changes in the input/output formats provided by Unix shells are also preferable.
    • jjnoakes 10y ago
      The motivation for PowerShell was to have something good at both. Why switch contexts when moving from interactive use to coded automation if you don't have to?
      • qwertyuiop924 10y ago
        Because the requirements change depending on which one you're doing: quick shell automation isn't the same as the kind of automation you'd use perl/python for.
        • jjnoakes 10y ago
          And yet most of the time I see bash being used for the same kind of automation that you think is good for perl/python. And not because bash is good, but because it is what people know (since they use it interactively) and because it is installed everywhere. So if someone had a tool that was installed everywhere, and used interactively, that could also be used to create more robust automation tasks, that seems like a win to me.
• wcoenen 10y ago
  > I compare this to human languages. It would be tempting to create a language with minimum ambiguity and clear cut concepts, but it wouldn't be practical. Lojban[1] is usable in practice (though that may be because it still allows you to be ambiguous when desired). [1] http://lojban.org http://lojban.org
• gpvos 10y ago
  Passing CSV around can be a useful middle ground, if you're using utilities that all do proper escaping and allow newlines in strings etc., such as csvkit.
  • tonyedgecombe 10y ago
    What is proper escaping for CSV?
    • userbinator 10y ago
      RFC4180.
    • gpvos 10y ago
      The problem is that not all tools agree on this one, but: lines are separated by newlines, items are separated by commas. An item can be quoted (and this must be done if it contains a newline, comma, or double quote) by surrounding them with double quotes. Inside such quoted items, double quotes can be escaped by doubling them.
  • qwertyuiop924 10y ago
    DSV is the default format for many unix utils, and AWK and cut parse it easily.
    • gpvos 10y ago
      Yes, of course; that is one end of the "middle ground" I described. The problem is that in Unix, there is no common convention for how to escape the delimiter, so you cannot pass arbitrary strings. CSV can do this, with the proper tooling. Being able to pass around arbitrary strings is quite valuable, and although it is not quite the same as passing objects around, it has the advantage of still being human-readable text. That is why I called it a "useful middle ground" between Powershell and the delimiter-separated fields of Unix tools. However, you do need special tools such as csvkit, as Unix tools like awk, sort and cut cannot properly parse it.
• progman 10y ago
  > text "scraping" is just better for most cases. Unix pipes handle bytes, not just text. For instance, copy a hard disk partition to a remote hard disk via ssh: (dd if=/dev/hda1) | (ssh root@host dd of=/dev/sda1) The KISS principle ("Keep it simple, Stupid") is the best way in many cases. In Unix you can quickly enter a pipe without special coding which does also non-trivial stuff. For instance, find -name "*.xls" -exec echo -n '"{}" ' \; | xargs xls2csv | grep "Miller" | sort gets a sorted list of all entries in all Excel files which contain the name "Miller", no matter how deeply the files are located in the directories. Can you do this in Powershell quickly? I don't know, I am actually curious. Objects in pipes are convenient and powerful. However, for most applications of pipes they are probably overkill. If things get tough you can simply use files rather than objects. I would not be surprised if many Windows users will prefer bash pipes from the Ubuntu subsystem in Windows 10 rather than Powershell because it is much more handy for most pipe applications.
  • ctolkien 10y ago
    > Can you do this in Powershell quickly? I don't know, I am actually curious. You can basically do the same thing with Powershell. I don't know of a 'built-in' module to handle XLS->CSV conversion, so you need to bring one in: Install-Module ImportExcel Then: ls *.xlsx -r | %{ Import-excel $_ } | ? { $_.Name -eq "Miller" } | sort That's my naive, Powershell noob approach anyway. There's another option however, where you can leverage Excel itself (granted, this is likely to be a Windows only approach): $excel = New-Object -com excel.application And you can now open XLS/XLSX files and operate on them as an actual excel document (including iterating through workbooks/ worksheets, etc.). It's all just objects.
    • progman 10y ago
      Thanks for pointing this out. However, your example is not equivalent since "ls" searches only in the current directory. Nevertheless it is good to know that it's basically possible since it helps a lot to manage mixed Windows/Linux networks.
      • 10y ago
• yellowapple 10y ago
  This is all a big part of the reason why I personally advocate using a standardized text-based object serialization format (like YAML) for these sorts of things. In particular: * Still text based and human-readable, which helps for debugging/troubleshooting * Still text-based and machine-readable, so it's inherently cross-platform (assuming that all said platforms agree on their text encoding) * Still less troublesome than piping arbitrary text through `grep` or `sed` or `awk` or what have you * Still provides the "everything is an object" benefit that's lost with arbitrary text streams * Still orientable around streams of data, at least for YAML (by using the document delimiter)