7 ms·

Weirdly, perl. I found myself on a client site in a locked down environment with only the Windows version of GIT installed. Powershell disabled, no other progra

by Zhyl 8y ago

Weirdly, perl. I found myself on a client site in a locked down environment with only the Windows version of GIT installed. Powershell disabled, no other programs allowed. I had to produce reports on the log files for this machine, but had to obfuscate information before taking it off the server. I discovered that git had bash installed quite early on, so my initial automation and reporting was based on UNIX pipelines. As the data volumes became bigger and the customer requirements became more demanding, I eventually bit the bullet and started learning perl one-liners to do some gymnastics that just weren't possible or practical with just bash (first big win was adding the MD5 of the line so that I could dedupe the obfuscated data). This lead to eventually writing full scripts in perl and now I have basically all of my reporting automated, even able to take up some 'value add' reporting that I didn't think was going to be possible. So yeah, not a new technology but I'm now a perl convert and will almost certainly use it in the future if I'm faced with a similar situation.

elorm 8y ago
Hey, there’s an old school community(they prefer monastery) called PerlMonks[0]where you can share your Perl oneliners and also pick up some mind blowing stuff. Come hang around sometime [0]https://perlmonks.org/ https://perlmonks.org/
- collyw 8y ago
  Worth noting that PerlMonks functions similarly to stack overflow (for Perl at least), with the difference that discussion is actually encouraged.
  - eitland 8y ago
    Somebody should pick up that idea and create a site like stack overflow but that doesn't discourage discusssion or reward destroying useful knowledge.
    - raverbashing 8y ago
      The discussion post is hard as you have to filter the noise and people who know less than they think they know. It might be doable but it is a hard problem
    - lucb1e 8y ago
      But... it does discourage discussion. It's a Q&A site, not a forum. You don't just post some thoughts and reply to other people's thoughts: you post either a question, or an answer. If you need clarification on a post, you comment. It's not a discussion forum. And I'm not a new, disgruntled user. I like the site a lot and am quite active on the IT Security stackexchange site. From my semi-moderator's point of view, the only place for discussion is the chat function (which only a few regulars use).
      - amenod 8y ago
        Yeah... the problem is that SO is becoming the only place where people hang out, and it's really difficult to find active programming forums nowadays. Except, as you say, SO is not a forum, it's a QA site. Which means there is no place left for discussions. While I certainly use SO a lot to find those small bits of missing knowledge, I think it has made search for help (in specific cases) actually much harder.
        lucb1e 8y ago
        Agreed! I moved to HN and SO after two forums slowly died. I have some friends from those forums, but a chat group is different from a forum with dozens or sometimes hundreds of active topics every day. Not just in activity, but also the kind of things you can talk about. I'm not quite sure what is going to fill this gap. I don't always have a place to post things. Blogging doesn't exactly work without an audience. The mainstream things these days appear to be just chats. There are still forums, mailing lists... heck, there are BBSes but it's niche and hard to find anything that isn't already on the decline.
        eitland 8y ago
gronne 8y ago
Im curious to what tasks you felt you could not solve in bash incl. the nix toolchain?
- Zhyl 8y ago
  Many of my issues will be specific to the environment. The thing that made me use the first perl one-liners is because I was used to using 'rename' in the command line and I was able to emulate this with perl more easily (using a regex substitution) than 'mv'. The second and more pivotal one liner was when I needed to create a primary key to ensure I wasn't getting any dupes later down the pipeline (a requirement that came about as some of the other contractors on the project were messing with my exports and then blaming my files for their conflicting numbers). My first approach was to use the MD5 tool included in GIT bash, but this was prohibitively slow, taking 45 mins to loop through each line of a 4mb file. Not sure if there was a performance penalty on the box I was using, but the perl one liner did the job in a couple of seconds for all the files I had. After that it mostly became a question of maintainability, with the performance being a boosting factor. I found my perl deduping one liner was faster than either 'sort -u' or 'sort | uniq'. I was able to add new features and account for weird issues more easily. Regex was better than sed, syntax was more flexible than awk, there were more filtering options than grep. Plus it was good fun. Perl has so many things that just feel so nice and tidy compared to bash. Having said that, I still use Unix pipelines extensively, they just tend to be for ad hoc queries rather than for my automated reports.
  - gronne 8y ago
    All right - thanks for the elaboration. The perfomance diff sounds interesting. I might check it out at some point :)
- zdw 8y ago
  perl has a less divergent set of options across different unix versions - for example, trying to run `sed` in place across MacOS and Linux you'll soon find that the options are different. There are similar issues with BSD vs GNU options in other programs as well. There's an old quip that "perl is portable sed" - in my experience this is pretty true.
tmaly 8y ago
My team has used Perl successfully for 13 years. We use modern Perl and the code is clean. In the process of switching new team members to Python as it is hard to hire people with Perl. I still think the meta capability in Perl is stronger than Python.
- lucb1e 8y ago
  I'm a security guy but like to develop as well. I'd apply for a job in Perl any day, despite not knowing it very well and Python being my main language. But I guess it's easy to say when I just read that you're having trouble hiring, as my expectations of entry requirements are now different. Anyway, it's a shame you have to make such a big change for what shouldn't be a big deal. The language is a tool -- I've even written C# in Visual Studio on Windows 8 for a year in two different companies, which is about as Microsoft-oriented as you can get (I'd rather use something Linux-based). Unless there are major deficiencies and the team is looking to move away already, it shouldn't be a problem to adapt to the team's standards.
- yellowapple 8y ago
  Meanwhile, I had a hard time finding (or getting) a Perl programming job (though I resisted coming to the Bay Area for a long time, so that was likely a significant factor). I've had to settle with other languages (mostly Ruby, Elixir, and now Python professionally, though I did sneak a bit of Perl into the Ruby one ;) ). Where are y'all based? I'm reasonably happy with my current job, but I'm definitely curious about what else is out there (especially if it's not in the Bay Area).
  - tmaly 8y ago
    We are in Greenwich CT close to NYC but we also have an NYC office. https://careers-interactivebrokers.icims.com/jobs/1777/programmer---compliance-technology/job https://careers-interactivebrokers.icims.com/jobs/1777/progr...
- C1sc0cat 8y ago
  You haven't tried training them ?
catacombs 8y ago
In this day and age, with Python and all, is Perl still worth learning?
- Zhyl 8y ago
  It's worth getting a few insights into what it does well. As with any language it _can_ do anything, but a few key features that made it stand out for me: * Perl is set up for really terse one-liners. You can basically emulate or surpass most Unix tools with only a cursory knowledge of the syntax * Perl syntax is highly geared towards text processing and regular expressions. File handling and regexes are so much quicker and easier to use than Python. If I were in a situation where I had a bunch of files I needed to process and the task didn't easily fall into something that could be done with a bash command, I'd probably reach for perl over python for a quick and dirty disposable script.
  - catacombs 8y ago
    > * Perl is set up for really terse one-liners. You can basically emulate or surpass most Unix tools with only a cursory knowledge of the syntax This definitely interests me. How does perl surpass Unix tools? I guess perl has the benefit of handling the equivalent of sed and awk.
    - Zhyl 8y ago
      Yeah, it has equivalents for each. Sed: perl -pe 's/foo/bar/' file1 Grep: perl -pe '/foo/' file1 uniq: perl -ne 'print unless $seen{$_}++' file1 But then you can combine conditions: perl -ne 's/foo/bar/ if /baz/ && /yot/; print "Line number $. changed to $_"' file1 And do some awk-ish line splitting: perl -F, -ane '$sum += $F[2]; END{ print "The sum of column 3 is $sum" }' file1 It absolutely flies in the face of the unix philosophy and is much more complicated than picking up each tool individually, but once you get to grips with 'em they're very handy.
      - a_bonobo 8y ago
        This blew my mind recently, when I had to delete millions of files in a directory and rm didn't make the cut (too many arguments), and find -delete took forever: https://unix.stackexchange.com/a/79656 https://unix.stackexchange.com/a/79656 perl -e 'for(<*>){((stat)[9]<(unlink))}' seems to be the fastest possible way to delete enormous amounts of files
singingfish 8y ago
perl is a fine language. It scales quite well from terse one liners to large (and with appropriate application of discipline, readable) applications. It's a shame it's so unfashionable at the moment.

by Zhyl 8y ago

elorm 8y ago

Hey, there’s an old school community(they prefer monastery) called PerlMonks[0]where you can share your Perl oneliners and also pick up some mind blowing stuff. Come hang around sometime [0]https://perlmonks.org/ https://perlmonks.org/

collyw 8y ago
Worth noting that PerlMonks functions similarly to stack overflow (for Perl at least), with the difference that discussion is actually encouraged.
- eitland 8y ago
  Somebody should pick up that idea and create a site like stack overflow but that doesn't discourage discusssion or reward destroying useful knowledge.
  - raverbashing 8y ago
    The discussion post is hard as you have to filter the noise and people who know less than they think they know. It might be doable but it is a hard problem
  - lucb1e 8y ago
    But... it does discourage discussion. It's a Q&A site, not a forum. You don't just post some thoughts and reply to other people's thoughts: you post either a question, or an answer. If you need clarification on a post, you comment. It's not a discussion forum. And I'm not a new, disgruntled user. I like the site a lot and am quite active on the IT Security stackexchange site. From my semi-moderator's point of view, the only place for discussion is the chat function (which only a few regulars use).
    - amenod 8y ago
      Yeah... the problem is that SO is becoming the only place where people hang out, and it's really difficult to find active programming forums nowadays. Except, as you say, SO is not a forum, it's a QA site. Which means there is no place left for discussions. While I certainly use SO a lot to find those small bits of missing knowledge, I think it has made search for help (in specific cases) actually much harder.
      - lucb1e 8y ago
        Agreed! I moved to HN and SO after two forums slowly died. I have some friends from those forums, but a chat group is different from a forum with dozens or sometimes hundreds of active topics every day. Not just in activity, but also the kind of things you can talk about. I'm not quite sure what is going to fill this gap. I don't always have a place to post things. Blogging doesn't exactly work without an audience. The mainstream things these days appear to be just chats. There are still forums, mailing lists... heck, there are BBSes but it's niche and hard to find anything that isn't already on the decline.
        eitland 8y ago

gronne 8y ago

Im curious to what tasks you felt you could not solve in bash incl. the nix toolchain?

Zhyl 8y ago
Many of my issues will be specific to the environment. The thing that made me use the first perl one-liners is because I was used to using 'rename' in the command line and I was able to emulate this with perl more easily (using a regex substitution) than 'mv'. The second and more pivotal one liner was when I needed to create a primary key to ensure I wasn't getting any dupes later down the pipeline (a requirement that came about as some of the other contractors on the project were messing with my exports and then blaming my files for their conflicting numbers). My first approach was to use the MD5 tool included in GIT bash, but this was prohibitively slow, taking 45 mins to loop through each line of a 4mb file. Not sure if there was a performance penalty on the box I was using, but the perl one liner did the job in a couple of seconds for all the files I had. After that it mostly became a question of maintainability, with the performance being a boosting factor. I found my perl deduping one liner was faster than either 'sort -u' or 'sort | uniq'. I was able to add new features and account for weird issues more easily. Regex was better than sed, syntax was more flexible than awk, there were more filtering options than grep. Plus it was good fun. Perl has so many things that just feel so nice and tidy compared to bash. Having said that, I still use Unix pipelines extensively, they just tend to be for ad hoc queries rather than for my automated reports.
- gronne 8y ago
  All right - thanks for the elaboration. The perfomance diff sounds interesting. I might check it out at some point :)
zdw 8y ago
perl has a less divergent set of options across different unix versions - for example, trying to run `sed` in place across MacOS and Linux you'll soon find that the options are different. There are similar issues with BSD vs GNU options in other programs as well. There's an old quip that "perl is portable sed" - in my experience this is pretty true.

tmaly 8y ago

My team has used Perl successfully for 13 years. We use modern Perl and the code is clean. In the process of switching new team members to Python as it is hard to hire people with Perl. I still think the meta capability in Perl is stronger than Python.

lucb1e 8y ago
I'm a security guy but like to develop as well. I'd apply for a job in Perl any day, despite not knowing it very well and Python being my main language. But I guess it's easy to say when I just read that you're having trouble hiring, as my expectations of entry requirements are now different. Anyway, it's a shame you have to make such a big change for what shouldn't be a big deal. The language is a tool -- I've even written C# in Visual Studio on Windows 8 for a year in two different companies, which is about as Microsoft-oriented as you can get (I'd rather use something Linux-based). Unless there are major deficiencies and the team is looking to move away already, it shouldn't be a problem to adapt to the team's standards.
yellowapple 8y ago
Meanwhile, I had a hard time finding (or getting) a Perl programming job (though I resisted coming to the Bay Area for a long time, so that was likely a significant factor). I've had to settle with other languages (mostly Ruby, Elixir, and now Python professionally, though I did sneak a bit of Perl into the Ruby one ;) ). Where are y'all based? I'm reasonably happy with my current job, but I'm definitely curious about what else is out there (especially if it's not in the Bay Area).
- tmaly 8y ago
  We are in Greenwich CT close to NYC but we also have an NYC office. https://careers-interactivebrokers.icims.com/jobs/1777/programmer---compliance-technology/job https://careers-interactivebrokers.icims.com/jobs/1777/progr...
C1sc0cat 8y ago
You haven't tried training them ?

catacombs 8y ago

In this day and age, with Python and all, is Perl still worth learning?

Zhyl 8y ago
It's worth getting a few insights into what it does well. As with any language it _can_ do anything, but a few key features that made it stand out for me: * Perl is set up for really terse one-liners. You can basically emulate or surpass most Unix tools with only a cursory knowledge of the syntax * Perl syntax is highly geared towards text processing and regular expressions. File handling and regexes are so much quicker and easier to use than Python. If I were in a situation where I had a bunch of files I needed to process and the task didn't easily fall into something that could be done with a bash command, I'd probably reach for perl over python for a quick and dirty disposable script.
- catacombs 8y ago
  > * Perl is set up for really terse one-liners. You can basically emulate or surpass most Unix tools with only a cursory knowledge of the syntax This definitely interests me. How does perl surpass Unix tools? I guess perl has the benefit of handling the equivalent of sed and awk.
  - Zhyl 8y ago
    Yeah, it has equivalents for each. Sed: perl -pe 's/foo/bar/' file1 Grep: perl -pe '/foo/' file1 uniq: perl -ne 'print unless $seen{$_}++' file1 But then you can combine conditions: perl -ne 's/foo/bar/ if /baz/ && /yot/; print "Line number $. changed to $_"' file1 And do some awk-ish line splitting: perl -F, -ane '$sum += $F[2]; END{ print "The sum of column 3 is $sum" }' file1 It absolutely flies in the face of the unix philosophy and is much more complicated than picking up each tool individually, but once you get to grips with 'em they're very handy.
    - a_bonobo 8y ago
      This blew my mind recently, when I had to delete millions of files in a directory and rm didn't make the cut (too many arguments), and find -delete took forever: https://unix.stackexchange.com/a/79656 https://unix.stackexchange.com/a/79656 perl -e 'for(<*>){((stat)[9]<(unlink))}' seems to be the fastest possible way to delete enormous amounts of files

singingfish 8y ago

perl is a fine language. It scales quite well from terse one liners to large (and with appropriate application of discipline, readable) applications. It's a shame it's so unfashionable at the moment.