6 ms·
Show HN: Bsed – Simple, English syntax on top of Perl text processing
- bihla 8y agoI'd love to hear any questions or feedback. I've gotten quite a bit of use from this tool personally and have had a lot of fun making it.
- king_magic 8y agoReally looking forward to trying this out. The syntax looks very clean & clear. I hate the insanity of sed, and assuming the performance is here, I’d switch to this in a heartbeat.
- bihla 8y agoIn my experience, Perl outperforms sed on most equivalent commands. I should do some benchmarking and add it to the documentation.
- asicsp 8y agothat would depend on a lot of factors sed provides only BRE/ERE which has lot less features than perl, however in my experience speed is better with BRE/ERE than PCRE, except patterns with backreferences. see also [1] a simple example: $ time sed 's/at/AT/g' /usr/share/dict/words > /dev/null real 0m0.041s user 0m0.037s sys 0m0.004s $ time perl -pe 's/at/AT/g' /usr/share/dict/words > /dev/null real 0m0.056s user 0m0.051s sys 0m0.004s $ time LC_ALL=C sed -nE '/^([a-z]..)\1$/p' /usr/share/dict/words > /dev/null real 0m0.049s user 0m0.048s sys 0m0.000s $ time perl -ne 'print if /^([a-z]..)\1$/' /usr/share/dict/words > /dev/null real 0m0.041s user 0m0.033s sys 0m0.007s [1] https://swtch.com/~rsc/regexp/regexp1.html https://swtch.com/~rsc/regexp/regexp1.html
- rurban 8y agoYou should really use psed then, which came with perl5 from 5.002 until v5.21.1 and is now on CPAN. This is a proper sed, implemented in perl. https://metacpan.org/pod/release/RJBS/perl-5.21.0/x2p/s2p.PL https://metacpan.org/pod/release/RJBS/perl-5.21.0/x2p/s2p.PL https://metacpan.org/pod/release/LEONT/App-s2p-1.000/script/s2p https://metacpan.org/pod/release/LEONT/App-s2p-1.000/script/...
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- kylek 8y agoGreat, now I only have to google SQL syntax :) (Sorry, had to take the easy one. Tooling using something as wide-spread as SQL is definitely a good thing!)
- Sir_Cmpwn 8y agoThis is very cool for simplifying interactive use :) but I'd like to caution people away from immortalizing it in their scripts. sed and awk are both standardized by POSIX and can be found on pretty much any system, and trivially ported to new systems. Porting Python, on the other hand, is comparitavely a mammoth proposition, and often impossible on some platforms. For this reason, I'm generally allergic to "new $x replaces $y!" for any $y which is governed by portable standards.
- bihla 8y agoI agree. For anything longer term, I'd recommend finding the right bsed command and then using -t flag to get the Perl translation. Perl is installed by default on common operating systems.
- Sir_Cmpwn 8y ago>Perl is installed by default on common operating systems. Aye, but what about the uncommon ones? For your consideration, I'd like to share Perl's configure script: https://github.com/Perl/perl5/blob/blead/Configure https://github.com/Perl/perl5/blob/blead/Configure It's my favorite configure script of all time, but I would hate to be the guy who has to extend it with a new platform :) And I have!
- username223 8y agoThis is something I love about Perl: in an age of many subtly incompatible operating systems, Perl said "F it, we'll do what it takes to compile and run your program everywhere." People don't appreciate this now, when everything is Linux, Mac, or Windows, but back when you regularly had to deal with various BSDs, HPUX, AIX, Mac OS 9, and even the occasional VAX, Perl was a godsend.
- snazz 8y agoAnd even though Perl grew up on and for Unix, there are a couple of distributions that work good on Windows, too, even though I think Perl feels more at home on Unix, where it came from.
- jaytaylor 8y agoI'd definitely incorporate it into my projects if it were ported Go, Rust, or any reasonable redistributable form, and continued to use a BSD license. Way clearer than cryptic sed and awk one-liners! Big kudos to you for making this and sharing it with the world! Python apps which are more than a single script are a non-starter for me. It's a big pain to get everyone on my 120+ person dev team to install it properly, they're all over the place with regard to environments. "Oops, I'm on Python 3.4", or python 2.6, or "I use this other python path"; it's inevitably a headache.
- yoklov 8y agoYep, this. Python is a headache for this sort of thing IME. (Nothing against the language though, although I don't know it)
- anoncake 8y ago> IME > I don't know it Huh?
- elliotec 8y agoOne can have “experience” with something, and not “know” it. Could just as easy say: “I have experience in scenarios where I must use python or run its scripts in an evironment, and it has been a headache, but I can’t say I dislike the language because I haven’t needed to explicitly create programswith it.”
- deleted 8y ago
- asicsp 8y agosome notes on the sample awk/sed/perl one-liner given: # input line has to be explicitly printed awk '{gsub(/Jack/,"Jill")} 1' file.txt # -i will do inplace editing, unlike the awk command # -i by itself won't work on non-GNU versions, needs backup extension sed 's/Jack/Jill/g' file.txt # use single quotes always, unless double is needed # -p will behave like default sed perl -pe 's/Jack/Jill/g' file.txt personally, I prefer the terseness of these commands over verbose SQL like syntax (and also the fact that I don't know SQL like tools) However, I would agree that initial learning curve is tough for sed/awk/perl. Once you get familiar with their idioms, they become the swiss army knife of cli text processing (along with grep). I have an entire repo dedicated to such tools[1] [1] https://github.com/learnbyexample/Command-line-text-processing https://github.com/learnbyexample/Command-line-text-processi...
- chmln 8y agoShameless plug for even terser syntax: https://github.com/chmln/sd https://github.com/chmln/sd
- asicsp 8y agolooks interesting, bookmarked (will possibly add tutorial on this someday) one question though, sed is not just search and replace, it is filter + search and replace (without going into arcane commands like n,N,x,h,etc) - filtering can be done based on regex, line number, combination of these two for blocks of line, etc.. Does sd support these?
- chmln 8y agoNot to derail this submission, but thanks for the consideration! See the first sentence at https://github.com/chmln/sd#comparison-to-sed https://github.com/chmln/sd#comparison-to-sed tl;dr its only find & replace with smart defaults and easy, straightforward syntax because I hated wrangling with sed's quirks. For more features, you're better off with sed. But if you're like me and only really use sed for find/replace, you will have a good time with sd.
- disposedtrolley 8y agoThis is cool! I’ve wanted to explore something similar but incorporating a broad range of Unix commands. I imagined a drag and drop interface (sort of like the Scratch programming language) with English descriptors of the functions being performed. You’d make it a web UI or something that generates a bash script you can paste into the terminal.
- purplezooey 8y agoAh, googling for some awk wizardry. Best done with a sandwich.
- chmln 8y agoHow does it compare performance-wise to other tools with similar goals? Seeing this is written in 100% python, I suspect it will be an order of magnitude slower.
- bihla 8y agoThe python code only carries out translation into Perl. The actual execution of the text transformation is Perl, which is generally the fastest of the available tools.
- ilovetux 8y agoPerl aside, I would say that's almost imperative to learn sed and awk. They invaluable tools for getting things done in *nix. They seem cryptic at first, but put a weekend into learning them and you learn that they are simply concise.
- acqq 8y agoAnd as an extremely good free resource for awk I recommend: https://www.gnu.org/software/gawk/manual/gawk.html#Getting-Started https://www.gnu.org/software/gawk/manual/gawk.html#Getting-S...
- lapnitnelav 8y agoI use Grymoire as my goto sed / awk reference. http://www.grymoire.com/Unix/Awk.html http://www.grymoire.com/Unix/Awk.html http://www.grymoire.com/Unix/Sed.html http://www.grymoire.com/Unix/Sed.html
- acqq 8y agoThanks! It has really good coverage of all the shell-related problems.
- wink 8y agoAnd then you have a script that needs to run on a mac as well and you can either fight that the people will install the gnu version or you just rewrite your one-liner in perl. I'm not a huge fan of perl but I've been writing my one-liners in it for a few years now and I've had less problems with portability.
- ilovetux 8y agoI would use another language for distributing a script (probably) python. Awk and sed I use more on a one-off basis although I do have a list of ones I use frequently.
- 8y ago
- ncmncm 8y agoHe lost me at "Perl".
- bihla 8y agoCan you elaborate? The tool doesn't require you to learn/know perl, that's just what operates under the hood. I actually chose to exclude any reference to Perl in the title, though it appears a moderator changed it.
- otherflavors 8y agothey are trying for cool points by hating on perl
- Yetanfou 8y ago...which is such an odd thing to do on a forum frequented by people who pride themselves on their breadth of knowledge and openness to new ideas.
- acqq 8y ago> I actually chose to exclude any reference to Perl in the title, though it appears a moderator changed it. On HN, the title of the submission is expected to match the title on the target page, except if the later is a clickbait.
- bihla 8y agoGood to know this, thank you.
- techcode 8y agoThat's based on having experience working with Perl?
- deleted 8y ago[deleted]
- salutonmundo 8y agoOdd that it installs with pip but uses perl.
- jancsika 8y ago> bsed giant_malformatted.json replace '\'' with '\"' | bsed replace 'True' with 'true' | bsed replace 'False' with 'false' bsed giant_malformatted.json replace '\'' that begins or ends a string with '\"' | bsed replace unquoted 'True' with 'true' | bsed replace unquoted 'False' with 'false'
- bihla 8y agoFair point. I mostly put those commands together to illustrate concepts. Two things you've touched on though that I haven't implemented yet: compound conditionals, and a nice way to indicated word boundaries (as opposed to the regex solution)
- peteretep 8y agoKinda off that this is written in Python, and not Perl. If it was written in Perl, it'd have the portability that the underlying translated commands have! I'd be happy to help you port it to modern, high-quality, readable Perl if that's something you're interested in. Being able to install this by dropping a single Perl file into ~/bin/ would be neat.
- ChuckMcM 8y ago@bihla take this man up on his offer :-). I thought it was an April fools joke where it talks about perl and then implements everything in python. But no, its actually python. If you (peteretep) do port it to perl, please post a show HN: :-)
- bihla 8y agoI agree, if it were entirely Perl it would have essentially no dependencies. Right now the audience is limited to those with a currently functioning Python3 environment or willing to configure one. My only concerns are: I do not know what the process of getting comparable tab completion would be in Perl. I depended on the argcomplete package for this feature which I consider to be essential...the clarity of the syntax depends on being a bit verbose, and this is only usable because tab completion makes it fast to use. Also, I'm somewhat reluctant to develop in Perl, though I'm willing to do it provided the benefit + demand is there. It would be ironic to have learned Perl as a consequence of building a tool whose primary function is to prevent one having to learn Perl. I'll send you an email to discuss further.
- acqq 8y agoI haven't used the following (I simply never had to implement the shell autocompletion as try I avoid the shell as much as I can, preferring to organize my code in the actual files that I can reuse from my editor) but I guess it can be the best start to include significant parts of this code inside of your final script, if the goal can be to have a single easily usable script doing everything, as somebody mentioned that ideal here (still take care to keep the compatible/same license): https://fastapi.metacpan.org/source/PERLANCAR/Complete-Bash-0.320/lib/Complete/Bash.pm https://fastapi.metacpan.org/source/PERLANCAR/Complete-Bash-... You can surely omit the lines between =pod and =cur (they are used to create the documentation) and most probably also the "description" parts. > It would be ironic to have learned Perl as a consequence of building a tool whose primary function is to prevent one having to learn Perl. It's less ironic than you think, I consider it the bigger irony that Python turned to be the major compatibility problem in all but single-user single-program environments (it's really not trivial to have to work with more Python variants as the dependencies for different code). You can be sure that any Perl 5 skills you obtain can remain quite portable and usable across the different environments. I've also already written here, I consider Perl significantly "safer" language than Python: when I modify some big Perl scripts, very often the Perl compiler (when used with the "use strict" and -w option) will tell me early enough what is missing for a program to function correctly again, as Perl simply more explicit, in some weird way almost like the typed languages. I never have the same feeling of "it will work" with the modification of the Python scripts. That "line noise" is actually meaningful, much more consistent than the shell languages, and safer in case of the changes than Python.
- HerrMonnezza 8y agoThe HN entry title might be misleading: this is a Python tool, Perl is only mentioned in the Readme for comparison.
- bihla 8y agoPerl is actually executing the text transformations. The python is a translation layer between the bsed syntax and the Perl command, so you get the performance of Perl with a very minor time cost for interpretation done in Python.
- singingfish 8y agoYeah, there's not a lot of point in having the python layer. Perl excels at these kinds of things. I'm guessing that the original developer just doesn't know perl well enough to implement that part. Which is not to denegrate his work, but this could easily be a drop in single file perl executable if someone does the actual (not that much) work.
- PhantomBKB 8y agoBased
- mhd 8y agoTEXT PROCESSING DIVISION. PERFORM REGEXP-REPLACE "LETTER A; LETTER B; ONE-OR-MORE-CHARACTERS;" EMPTY-STRING .
- JdeBP 8y agoOr, rather: PIC A(2)X(integer01)
- seiferteric 8y agoNeat, but a little scary because you rely on it to interpret your intentions correctly. Usually when I am trying to do something, I build up my command in multiple steps, printing outputs and verifying it is correct before actually running the command. Maybe you could have a "diff" or "dryrun" mode that would show the potential changes. Another idea would be for this script to "compile" a CLI command using grep/awk/sed etc instead of doing the actual editing itself.
- bihla 8y agoThe interpretation is only translating into standard Perl one-liners, and by default the execution writes to stdout instead of modifying the file. I haven't written code to actually do the text transformations. My normal workflow is to write commands, pipe them together as needed, and when I think I am satisfied I'll store to a temporary file. If I really need to be extra sure then I will use my preferred ddiff tool to compare against the original before doing the swap. I may have mis-interpreted but I hope between this and the unit tests your concerns are alleviated
- seiferteric 8y agoOkay I misunderstood, thanks!