4 ms·

I wonder if Python's rise in popularity has reduced shell scripting? Python has even displaced AWK in my tool use.

by OneOneOneOne 10y ago

Sukotto 10y ago
Can you give some brief examples of how you've done that? I wouldn't mind if I never had to use awk again...
- deleted 10y ago
  [deleted]
- gamegoblin 10y ago
  I admit I have never truly learned awk outside of the most dead simple stuff, but one of the most useful python utilities I have ever written is below. It allows you to use python lambdas on lines of stdin. Example usage is: lambda "x0 + x1 * x2" int int int Code: import sys # libs I use commonly... import random, itertools, re # parse the types of the columns types = map(eval, sys.argv[2:]) # craft a lambda with the args x0, x1, ... xN f = eval("lambda " + ','.join("x"+str(i) for i in xrange(len(types))) + ":" + sys.argv[1]) # apply lambda to stdin, don't print results of None for line in sys.stdin: args = [] for t, e in zip(types, line.strip().split()): args.append(t(e)) result = print f(*args) if result != None: print result Examples: Where data.txt is john doe 37 jane doe 35 jack bob 20 bill bob 40 And I do cat data.txt | lambda "x0 if x2 < 36 else x1" str str int It will output doe jane jack bob You can use this sort of tool for a million things. e.g. sample 1 out of every 1000 lines: lambda "x0 if random.random() <= .001 else None" str It's probably the same power and whatnot as awk, but I know and am much more familiar with python, so it's useful for me.
  - claystu 10y ago
    To accomplish the same thing in awk: $ awk '{if ($3 < 36) print $1; else print $2}' data.txt doe jane jack bob Python has a lot of strengths and is better than awk at a lot of things, but one-liner column based text processing on the command line is literally awk's bread and butter.
    - gamegoblin 10y ago
      Definitely. I don't think of it as a better awk, just a drop-in to minimize my cognitive overhead. It's basically Python-flavored awk.
    - visarga 10y ago
      I made a similar tool in Perl. It eval's a Perl command passed as string, on each line of stdin. In the middle I can go wild with regexes and hash/dictionaries and whatnot. It's one of my most used tools. Instead of writing scripts for each little task, I just write one-liners. When they become more than 2 lines long it becomes unwieldy and I switch it to a regular script.
      - iheartmemcache 10y ago
        I grew up data munging with awk/sed/tail/head manipulations, avoiding Perl 5 (not out of any antipathy, but I just worked off of what my fathers bookshelf had and there was no Camel book on there). Back in the 90s we'd publicly post our dotfiles to our Apache 1.3 servers amongst peers, but (perhaps this was just a component of the IRC community I was a part of) we didn't share much of scripts-sets we built up overtime. The furthest we'd go is "I want to do foo", someone with more knowledge than you would give you a series of invocation parameters and over time you'd osmotically acquire enough knowledge to be the one dispensing knowledge. From what I hear, Perl 5 is prime for the problem set you defined, but I've never seen any aggregated resource of people's Perl5 munging scripts. Do us all a favor and post a Github Gist of that tool (along with common invocations of you going wild with regexes and hashmaps). If you're feeling overly generous, post the source of the commonly used regular scripts as well.
      - claystu 10y ago
        You should check out Minimal Perl [https://amzn.com/1932394508 https://amzn.com/1932394508] if you haven't already done so.
    - doug1001 10y ago
      nice one. small, sharp tools in the unix toolbox. given that this is the type of processing one might routinely over a file with tens of millions of lines, what's your guess on the difference in performance between the python snippet above your your awk one-liner? (my guess is about 100x in favor of awk)
javier2 10y ago
Go has replaced all my 'scripting' needs. Way easier to deploy static binaries and I get static typing for my scripts. Maintaining a python install on multiple servers and OSes is a nightmare

I wonder if Python's rise in popularity has reduced shell scripting? Python has even displaced AWK in my tool use.

Sukotto 10y ago

Can you give some brief examples of how you've done that? I wouldn't mind if I never had to use awk again...

deleted 10y ago
[deleted]
gamegoblin 10y ago
I admit I have never truly learned awk outside of the most dead simple stuff, but one of the most useful python utilities I have ever written is below. It allows you to use python lambdas on lines of stdin. Example usage is: lambda "x0 + x1 * x2" int int int Code: import sys # libs I use commonly... import random, itertools, re # parse the types of the columns types = map(eval, sys.argv[2:]) # craft a lambda with the args x0, x1, ... xN f = eval("lambda " + ','.join("x"+str(i) for i in xrange(len(types))) + ":" + sys.argv[1]) # apply lambda to stdin, don't print results of None for line in sys.stdin: args = [] for t, e in zip(types, line.strip().split()): args.append(t(e)) result = print f(*args) if result != None: print result Examples: Where data.txt is john doe 37 jane doe 35 jack bob 20 bill bob 40 And I do cat data.txt | lambda "x0 if x2 < 36 else x1" str str int It will output doe jane jack bob You can use this sort of tool for a million things. e.g. sample 1 out of every 1000 lines: lambda "x0 if random.random() <= .001 else None" str It's probably the same power and whatnot as awk, but I know and am much more familiar with python, so it's useful for me.
- claystu 10y ago
  To accomplish the same thing in awk: $ awk '{if ($3 < 36) print $1; else print $2}' data.txt doe jane jack bob Python has a lot of strengths and is better than awk at a lot of things, but one-liner column based text processing on the command line is literally awk's bread and butter.
  - gamegoblin 10y ago
    Definitely. I don't think of it as a better awk, just a drop-in to minimize my cognitive overhead. It's basically Python-flavored awk.
  - visarga 10y ago
    I made a similar tool in Perl. It eval's a Perl command passed as string, on each line of stdin. In the middle I can go wild with regexes and hash/dictionaries and whatnot. It's one of my most used tools. Instead of writing scripts for each little task, I just write one-liners. When they become more than 2 lines long it becomes unwieldy and I switch it to a regular script.
    - iheartmemcache 10y ago
      I grew up data munging with awk/sed/tail/head manipulations, avoiding Perl 5 (not out of any antipathy, but I just worked off of what my fathers bookshelf had and there was no Camel book on there). Back in the 90s we'd publicly post our dotfiles to our Apache 1.3 servers amongst peers, but (perhaps this was just a component of the IRC community I was a part of) we didn't share much of scripts-sets we built up overtime. The furthest we'd go is "I want to do foo", someone with more knowledge than you would give you a series of invocation parameters and over time you'd osmotically acquire enough knowledge to be the one dispensing knowledge. From what I hear, Perl 5 is prime for the problem set you defined, but I've never seen any aggregated resource of people's Perl5 munging scripts. Do us all a favor and post a Github Gist of that tool (along with common invocations of you going wild with regexes and hashmaps). If you're feeling overly generous, post the source of the commonly used regular scripts as well.
    - claystu 10y ago
      You should check out Minimal Perl [https://amzn.com/1932394508 https://amzn.com/1932394508] if you haven't already done so.
  - doug1001 10y ago
    nice one. small, sharp tools in the unix toolbox. given that this is the type of processing one might routinely over a file with tens of millions of lines, what's your guess on the difference in performance between the python snippet above your your awk one-liner? (my guess is about 100x in favor of awk)

javier2 10y ago

Go has replaced all my 'scripting' needs. Way easier to deploy static binaries and I get static typing for my scripts. Maintaining a python install on multiple servers and OSes is a nightmare