4 ms·

Check out my pandemic project, marcel: https://marceltheshell.org https://marceltheshell.org. It hits that spot you described: still a shell, but more strongly

Check out my pandemic project, marcel: https://marceltheshell.org https://marceltheshell.org. It hits that spot you described: still a shell, but more strongly typed. Instead of bailing out and redoing everything in Python (when you reach the end of your pipeline), you can add a bit of Python code in your command. For example, find all the .py files under the current directory, find the ones that have changed in the last day, and then print the lines (with their filenames) that define classes: ls -fr \ | select (f: f.suffix == '.py' and now() - f.mtime < days(1)) \ | read -l \ | select (path, line: line.startswith('class ')) read -l reads and labels a file, i.e., filename -> stream of (filename, line in file) (Yes, this is doable with find and awk, but this is just an accessible example that gets across the idea of Python functions on the command line.)

geophile 6y ago
Oh, and when you do bail out and go to Python, marcel's operators are available in a module (marcel.api), so that you can take your pipeline and move it into Python easily. E.g. here's the same example in Python: from marcel.api import * for path, line in (ls(file=True, recursive=True) | select(lambda f: f.suffix == '.py' and now() - f.mtime < days(1)) | read(label=True) | select(lambda path, line: line.startswith('class '))): print(f'({path}: {line}) read(label=True)
enriquto 6y ago
> Yes, this is doable with find and awk You don't even need awk for that one: grep ^class `find . -mtime -1 | grep py$` Maybe you could try to use an example where it really makes a difference? Some natural operation that would be very cumbersome with plain shell but is easy and direct with marcel. Otherwise many people may fail to see the point.
- geophile 6y ago
  Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): read -c foo.csv | sort (*x: int(x[3]) + int(x[6])) I suppose we could argue about what is a "natural" operation. Marcel grew out of a set of tasks that were "natural" in the domain I was working in. An important part of that domain was operating on databases, and clusters of nodes, and databases on those nodes. So marcel has features for those capabilities. Partly, I was also scratching an itch: I prefer using Python to exploring the sublanguages of various shell commands. Much cleaner, in my opinion.
  - enriquto 6y ago
    > Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): Yes, that's a much better example. Here the "obvious" shell solution involves using awk to compute the sum of the two columns and putting it as the first field, sorting by the first field, and then removing that field. I guess a "rosetta stone" of such examples (e.g. on the frontpage of the site) would make a strong case for the interest of marcel.
    - em500 6y ago
      Concretely, awk -F, -v OFS=, {print $2+$3, $1, $2, $3}' foo.csv | sort -n | cut -d, -f2-
      - enriquto 6y ago
        I think this code does not fulfill the task: you need to sum columns 4 and 7, and keep the rest of the data intact in the output.
        em500 6y ago
        You're right, that should be awk -F, -v OFS=, '{print $4+$7, $0}' foo.csv | sort -n -k1 -t, | cut -f2- -d, Explanation: awk -F, -v OFS=, sets the input and output column separator to comma, '{print $4+$7, $0}' outputs the sum of column 4 and 7 before the rest of the line. sort -n -k1 -t, sorts the file numerically on column 1, with comma separator. cut -f2- -d, removes column 1, with comma separator. This is of course not robust for general CSV files, but I don't think OPs marcel is either. A robust solution requires a proper CSV parser. The biggest warts I see in the classic unix solution is that all the tools use different flags for the field separator. edit: if you know that the csv doesn't contain tabs, you can omit some flags for a more concise awk -F, -v OFS='\t' '{print $4+$7,$0}' foo.csv | sort -n -k1 | cut -f2- since sort and cut default to tabs/whitespace as separators. If you're unsure about the contents of the CSV, you really need a proper CSV parser.
        6y ago

geophile 6y ago

Oh, and when you do bail out and go to Python, marcel's operators are available in a module (marcel.api), so that you can take your pipeline and move it into Python easily. E.g. here's the same example in Python: from marcel.api import * for path, line in (ls(file=True, recursive=True) | select(lambda f: f.suffix == '.py' and now() - f.mtime < days(1)) | read(label=True) | select(lambda path, line: line.startswith('class '))): print(f'({path}: {line}) read(label=True)

enriquto 6y ago

> Yes, this is doable with find and awk You don't even need awk for that one: grep ^class `find . -mtime -1 | grep py$` Maybe you could try to use an example where it really makes a difference? Some natural operation that would be very cumbersome with plain shell but is easy and direct with marcel. Otherwise many people may fail to see the point.

geophile 6y ago
Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): read -c foo.csv | sort (*x: int(x[3]) + int(x[6])) I suppose we could argue about what is a "natural" operation. Marcel grew out of a set of tasks that were "natural" in the domain I was working in. An important part of that domain was operating on databases, and clusters of nodes, and databases on those nodes. So marcel has features for those capabilities. Partly, I was also scratching an itch: I prefer using Python to exploring the sublanguages of various shell commands. Much cleaner, in my opinion.
- enriquto 6y ago
  > Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): Yes, that's a much better example. Here the "obvious" shell solution involves using awk to compute the sum of the two columns and putting it as the first field, sorting by the first field, and then removing that field. I guess a "rosetta stone" of such examples (e.g. on the frontpage of the site) would make a strong case for the interest of marcel.
  - em500 6y ago
    Concretely, awk -F, -v OFS=, {print $2+$3, $1, $2, $3}' foo.csv | sort -n | cut -d, -f2-
    - enriquto 6y ago
      I think this code does not fulfill the task: you need to sum columns 4 and 7, and keep the rest of the data intact in the output.
      - em500 6y ago
        You're right, that should be awk -F, -v OFS=, '{print $4+$7, $0}' foo.csv | sort -n -k1 -t, | cut -f2- -d, Explanation: awk -F, -v OFS=, sets the input and output column separator to comma, '{print $4+$7, $0}' outputs the sum of column 4 and 7 before the rest of the line. sort -n -k1 -t, sorts the file numerically on column 1, with comma separator. cut -f2- -d, removes column 1, with comma separator. This is of course not robust for general CSV files, but I don't think OPs marcel is either. A robust solution requires a proper CSV parser. The biggest warts I see in the classic unix solution is that all the tools use different flags for the field separator. edit: if you know that the csv doesn't contain tabs, you can omit some flags for a more concise awk -F, -v OFS='\t' '{print $4+$7,$0}' foo.csv | sort -n -k1 | cut -f2- since sort and cut default to tabs/whitespace as separators. If you're unsure about the contents of the CSV, you really need a proper CSV parser.
        6y ago