10 ms·
Modernizing “less”
- JoshTriplett 12y ago> Make less use getopt() instead of its byzantine option parser (it needed that for PC operating systems. We don't need or want this complexity on POSIX.) This is the kind of thing that always astonishes me to see in a codebase: why reinvent something rather than just finding and including a compatibility implementation? Just grab an appropriate getopt.c and compile it in if the platform doesn't have one, then let the rest of the code pretend every platform has one. (Preferably an implementation of getopt_long; a quick search turned up some licensed under 3-clause BSD.)
- roeme 12y agoless was started (~ '83) before getopt() was made available to the 'general' public ('85). My guess is nobody bothered to replace these parts, since options were only added gradually; if at all. Coincidentally i'm in the same predicament with a tool a coworker of mine (initially) wrote, convoluted option parsing to say the least; but I'm too busy with fixing other parts or adding proper functionality to it than to replace it. Sometimes, 'less is more'.
- AceJohnny2 12y agoYou couldn't help it, could you? :)
- dredmorbius 12y agoMany moons ago I mentioned the story of "more" and "less" to our (female) DBA, who was unfamiliar with the commands, and the expression "less is more". "It's right in the manpage, actually". "No." "Yes, I'll send it to you." And so I did, with the subject line "man less". She sat at a desk right in front of mine, and I detected a somewhat painful silence as the email arrived. And realized I'd just inadvertently commented on her social life (confirmed through later conversations).
- deleted 12y ago[deleted]
- DSMan195276 12y agoIn general I agree with your sentiment, but getopt is one where I'm willing to make an exception. getopt_long is barely any actual code, and honestly its design stinks for an argument parser. It's also just standard C (IE. No platform issues to worry about assuming you're going to get argv and argc). I've written my own argument parser in just around 100 lines that's pretty similar to getopt and so far it's been well worth it.
- JoshTriplett 12y agoDoes your hand-written parser handle all the corner cases getopt does? Ending option processing with "--"? Interleaving options in any order (ls foo -l)? Stacking short options (ls -la) or writing them separately (ls -l -a)?
- DSMan195276 12y agoI had to go check if it does '--', but yes it does all of those things you listed. My parser is similar to getopt, but it's less clunky and avoids the duplication that getopt_long results in.
- JoshTriplett 12y agoThank you for your diligence, then; most hand-written parsers fail one or more of those. > less clunky and avoids the duplication that getopt_long results in. Duplication because of the ugly flag/val logic, which in practice is typically passed as either NULL, 's' (long option for short option 's') or NULL, OPTION_FOO (long option with no short option, OPTION_FOO > 255)? Yeah, that does seem silly. I think they did that to simplify the setting of boolean parameters, by passing &some_flag, 1, but that seems woefully insufficient when you need to handle arguments. I'd love to have a C library as capable as Python's argparse, instead.
- DSMan195276 12y agoOf course, I wasn't going to half-do it if I was gonna replace getopt. I'm with you in that I get annoyed when I run into programs which do just half-done job at a getopt replacement and you can't do the normal stuff like "-df" instead of "-d -f". And yes, that's pretty much what I'm getting at. The issue really stems down to the fact that getopt_long is just bolted on to getopt, so there's still the string short-opt syntax like "sf:t", and then you duplicate the options in the array of long opts with the chars or some random number > 255 otherwise. The string is really the most annoying part because there's no good way to generate it via the C preprocessor that I could come-up with, leading to some duplication between the long-opts and short-opts. It also can't easily generate help text for you from your arguments, which bugged me. My argument parser basically works off of a single xmacro header that holds all the argument information for getopt (Which gets organized via a few macros into an enum and an array). It's dead simple to add new arguments and there's no duplication or separate strings or etc. that you have to update at the same time besides adding code to handle that argument. Personally, I wrote my argument parser specifically because I couldn't find any that I was happy with after looking around. They were either clunky to use (getopt_long), or were full libraries and seemed like it would be a hassle to get it integrated into my code. I'd love to see a 'standard' argument parser that works with long options well and doesn't just feel like an afterthought like it does with getopt_long, but I think the chance to make such a thing has been missed. Personally, if I would ever use such a thing then it needs to be a single or just a few .c and .h files that I can stick directly into my program. I wouldn't want to have to bet on the distribution having it or not, and I'm not going to add a separate dependency just for argument parsing.
- _delirium 12y agoInteresting! From what I'm reading so far, this seems not to be Illumos-specific (despite being motivated by Illumos's needs), but rather a cleanup that'd be applicable to any POSIX-like system. A fork of less that assumes POSIX-like functionality and cleans up a lot of things accordingly does seem like a worthwhile project. A bit more "unixy" design that uses the system versions of available functionality (like globbing and UTF-8!) should also reduce the risk of weird bugs & inconsistencies with how the rest of the system operates.
- nn3 12y agoA faster string search would be nice. I used to (and still sometimes do) use less to analyze large trace files. With a few hundred MB the searching becomes a real bottle neck. Couldn't be that hard to do a boyer moore for non RE substrings.
- porker 12y agoWould using ag help? No longer maintained AFAIK, but fast for me.
- chrismonsanto 12y agowhere did you get that idea? last commit was 7 days ago, no mention of abandonment
- porker 12y agoFrom the status of the Ubuntu ppa, and (I thought) comments on his site. Thanks for pointing out that's not the case.
- e12e 12y agoAs far as I know, it still holds that ag isn't fast, ack is slow ;-) That is to say: grep is pretty fast (too). In other words, ack improved the user-interface and api for search across files, with an eye towards programming and editors, but was relatively slow -- ag aims to keep the improved ui/api/output but bring speed back up to gnu grep-like levels.
- porker 12y agoI think if your grep has the -s flag (the GNU 'treat as string' option) you're correct. If not, it may be faster :)
- e12e 12y agoYou might be thinking of -F/--fixed-strings. -s is slient (long option --no-messages). For GNU grep 2.12, anyway. Or you might be thinking of BSD grep: http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu... edit: Eg, with warm cache: :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |wc -l) 2765699 real 0m5.021s user 0m0.144s sys 0m0.792s :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E 'Some pattern' -v -c) 2765700 real 0m5.133s user 0m0.264s sys 0m0.852s :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E 'Some..pattern' -v -c) 2765700 real 0m5.144s user 0m0.400s sys 0m0.768s # "%% " used for leading comment lines in some of this code: :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E '^%% ' -c) 27535 real 0m5.597s user 0m0.520s sys 0m0.788s :~/tmp/riak/riak-2.0.0pre5/deps $ du -hcs . 405M . 405M total :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |ag '^%% ' >/dev/null) real 0m5.735s user 0m1.480s sys 0m0.876s #actually find/cat is pretty slow -- I guess both GNU grep and ag #use nmap to good effect: $ time rgrep '^%% ' . > /dev/null real 0m0.539s user 0m0.404s sys 0m0.128s :~/tmp/riak/riak-2.0.0pre5/deps $ time ag '^%% ' . |wc -l 27500 real 0m0.252s user 0m0.284s sys 0m0.068s :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E '^%% ' . |wc -l 27553 real 0m0.535s user 0m0.396s sys 0m0.140s Note that grep clearly goes looking in more files here (more mathcing lines). Still, I guess ag is indeed faster than grep in some cases (even if it might not be apples to apples depending how you count -- of course the whole point of ag is to help search just the right files). :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E 'Some pattern' . |wc -l 0 real 0m0.266s user 0m0.128s sys 0m0.132s :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E 'Some..pattern' . |wc -l 0 real 0m0.338s user 0m0.212s sys 0m0.120s :~/tmp/riak/riak-2.0.0pre5/deps $ time ag 'Some..pattern' . |wc -l 0 real 0m0.111s user 0m0.100s sys 0m0.076s I guess ag is indeed faster, even if it might not be due to fixed string search... [edit2: For those wondering that's an (old) ssd, on an old machine -- but with ~4G ram the working set should fit, as soon as some of my open tabs in ff are paged to disk...]
- gatehouse 12y agoSeems weird to hear about less being used "to meet POSIX" requirements, since I use it at least 10 times a day -- interesting read though.
- hayksaakian 12y agoNote: this is not referring to the compile-to-CSS language named Less ----- Downvotes for clarifying an ambiguity, really?
- justincormack 12y agoWhat does it add to the discussion?
- dredmorbius 12y agoClarity, without having to follow through to the article. The lack of any context for HN posts (also reddit link shares) is a significant disadvantage of both sites. I've always been partial to Slashdot's link summaries, and wish that style were more widely used. See also Jakob Nielsen and microcontent.
- unethical_ban 12y agoClicking on the article also immediately clarifies the meaning.
- comboy 12y agoI've switched from less to most* some time ago, no idea how it looks underneath, but it works great for me. It seems to be available for most (tee hee) distributions. * http://linux.die.net/man/1/most http://linux.die.net/man/1/most
- txutxu 12y agoI used 'most' a few years ago because of the windowing support (this is before 'screen' did get support for vertical split). Also I did love about 'most' that it used to colorize manpages, when other pagers didn't. I stopped using it, like 4 years ago, when someone did told me on IRC that it was unmaintained and did have some bugs. Note that I did never notice any 'bug' in my time as 'most' user, even if they may exist, and indeed, it's still available at least in all Debian versions. It's curious that distributions are able to "maintain" a package (maybe even with custom patches) which is not maintained or updated upstream, for years. It's encouraging to see actions like the one performed by this IllumOS developer. If a program is opensource, we can fork, improve and share. Or as users, we can take a look at the code when making choices, lots of people forget this in favor of search engine recommendations.
- e12e 12y agoThe level of love packages receive varies -- I'm a great fan of Debian, but in the case of "most" support appears to be less than stellar from the Debian side: https://bugs.debian.org/cgi-bin/pkgreport.cgi?which=maint&data=mako@debian.org&archive=no&raw=yes&bug-rev=yes&pend-exc=fixed&pend-exc=done#_1_4_5 https://bugs.debian.org/cgi-bin/pkgreport.cgi?which=maint&da...
- reconbot 12y agoIt's crazy to me that something as core as less has such bit rot. I'm aware GNU rebuilt a bunch of tools a while ago and added a slew of common features such as `-h` for human readable mode, and `--` long arguments. Does this rewrite have a name yet? and can it be brought into the core project at all? (or the GNU project?)
- jordigh 12y agoBeing part of the GNU project doesn't mean it's going to be maintained. For example, bzr supposedly is a core GNU package, and it's dead. Being a GNU package just means you agree to behave in a GNUly way. It's a very informal and easily-granted qualification.
- noonespecial 12y agoSome yaks look really good bald. Well done.
- Tloewald 12y agoCool, but I was hoping he meant less the CSS extension :-)
- dredmorbius 12y agoAmbiguity is program / feature names is a constant (and growing) issue. Though less(1) is the rather more established of these two instances.
- Tloewald 12y agoWell, it's certainly older. It's not really so ambiguous since the CSS version is "LESS". Speaking as someone who cut his teeth on the command line, I clicked the article thinking that it was referring to LESS since the idea of modernizing less never occurred to me (even though it's a much older piece of technology; LESS is much more broken...)
- oblio 12y agoIt seems to me that many core tools could use such updates. They're widely used yet most of them predate modern development techniques and tools such as widespread linter usage, unit tests, peer reviews before committing, appearance of various stable and portable libraries or even compilers, security scans. I wonder what other nasty things would appear after a serious code review of the GNU and BSD core code bases. I do agree that the code has been read probably hundreds of times and executed millions of time, but I doubt there have been many formal attempts at improvement and the overall methodoloy resembles brute forcing to me. I could be wrong.
- vezzy-fnord 12y agoMost of the base system on Linuxes (which is GNU) certainly have unit tests and static analyzers run on them, to the best of my knowledge (just looked at the coreutils). Most of these programs were maintained by Cygnus, and now by Red Hat after they acquired them. less is an exception to the rule, as it's a small, standalone program and not as critical. Its initial development start seems to predate the official announcement of the GNU Project. I don't think most people really expend much thought on something like a pager. EDIT: Brute forcing is also a very pragmatic approach. Ken Thompson uttered his famous adage for a reason, even if it was somewhat humorous.
- kev009 12y agoFreeBSD has always done peer reviews which cover the entire base system, and it's now mechanized a bit: http://julipedia.meroh.net/2014/05/code-review-culture-meets-freebsd.html http://julipedia.meroh.net/2014/05/code-review-culture-meets.... Automated testing (kyua) and CI (jenkins) are also in progress.
- justincormack 12y agoThere are also regular Coverity scans on the whole system (kernel and userspace) for the BSDs; the NetBSD one has been ongoing for quite some time and a lot has been fixed (a lot of the errors now seem to be in gdb...)
- _delirium 12y ago
- clarry 12y agoCool, but terminfo and curses are another bunch of things I'd like to see go.. or at least delegated from the core of every fullscreen terminal application to a special compatibility layer (e.g. tmux or screen) for those using terminals which don't speak ANSI. I've looked at it before, and I agree less is a mess.
- nn3 12y agoAlso it would be nice if the original author could put this work up as a tarball with standard make files somewhere so that other OS can benefit.
- 0xeeeeeeee 12y agoA lot of people are talking about revamping these old programs. I don't see what the problem is with less, cat, vim, etc... I've never been using a program and wished for better functionality. Even when this was all new to me, it was never a problem figuring these out, using them, and I was always satisfied with them.. So...here's the question. I don't think these are broken, so what are you fixing?
- reubenmorais 12y agoDid you read the blog post? Here's what he fixed: >So, now it: >Speaks terminfo (X/Open Curses) instead of ancient BSD termcap >Uses glob(3C) instead of a hack involving the shell and a helper program (lessecho, which I've removed from my tree.) >Functions properly as /usr/bin/more, both with and without -e (even on broken xterms) >Is fully ANSI C (or ISO C, if you prefer) >Passes illumos' cstyle code style checks >Is lint(1) clean
- paddyoloughlin 12y agoOne thing I have noticed is that less's filtering ('&<pattern>') is sloooooooooow compared to grep. If that were improved I'd never have to use tail -f ever again. Someday, I'll look into it.
- coldtea 12y ago>So...here's the question. I don't think these are broken, so what are you fixing? First, it's not just about something not working. It's about creating tools that are extensible and understandable and hackable. Open Source is not just about "working", it's about being modifiable by the end user. All this cruft (a mess of 200 obsolete architectures, dead code and deprecated library support that nobody used since 1988) works against that goal. Second, there are things that would be essential for some people, like international users (e.g proper multibyte support) that cannot be added due to dependancy of some custom methods of handling encodings. That's not some wishy washy magical unicorn feature request, it's essential for the main operation of what less does for those that have to deal with these encodings. Third, there's nothing wrong in taking pride and crafting finely your tools. UNIX is supposed to be made of things that "do one thing and do it well". Less having its own utf-8 support breaks this division of responsibility. We have libaries for that. Same for getopts vs it's custom options parsing.
- miah_ 12y agoI just use 'view'; its the same as running 'vim -R'.
- _delirium 12y agoI find 'view' fine for things like looking at a logfile, but not great for one of my more common use-cases for a pager, which is looking at something from stdin that's either large or slow-to-produce. You have to wait until vim reads in the entire stream before you can do anything with it. With 'less' you can immediately navigate/search/etc. while the stream is still coming in.
- paddyoloughlin 12y agoAlso, the default action of the cursor keys is more useful for paging in less than in view. In view, cursor keys move the cursor; in less, they scroll the screen. And less has built-in tailing which you can start and stop at any time. That's its killer feature for me. On the other hand, vim/view can have some nice syntax highlighting for syslog format log files. I haven't found that enough to switch though.
- gknoy 12y agoHave you tried using the less-like script distributed with vim? The path varies depending on your distribution of vim # ubuntu 14.04 alias less='/usr/share/vim/vim74/macros/less.sh' # ubuntu 12.04: vim73, if I recall right alias less='/usr/share/vim/vim73/macros/less.sh' This behaves like `less` in many ways, and uses your syntax highlighting from vim. My only complaint is that some things with escape codes for colors are not flattened, but instead you see the escape codes. (Diffs seem to work fine, at least.) It also appears to read the whole thing into vim, which is likely not what you want for large files. I also use the 'vimcat' script from the vimpager project [0] as well. (I'm not sure why I haven't just used the whole thing. I must not have realized there was more when I first grabbed it.) 0: https://github.com/rkitover/vimpager/blob/master/vimcat https://github.com/rkitover/vimpager/blob/master/vimcat
- jwr 12y agoThank you for this effort. It is badly needed.
- stuaxo 12y agoThese changes really need tk be upstreamed...
- paddyoloughlin 12y agoIt's probably not that simple. The article indicates that the refactor removed support for a lot of old platforms which the project may very well prefer to keep.
- onan_barbarian 12y agoI got excited to click on this, imagining that someone was modernizing less. Which is true in a narrow sense, but am I the only one to feel a lingering disappointment that we run shells in xterms (or slightly modified equivalents) that emulate ancient terminals, then implement pagers to page through screen after screen? Can anyone point to work that starts with the combination of the two following propositions: 1. User interface elements invented since the 1970s are a pretty neat thing. 2. Text-based shells and the command line are also a pretty neat thing.
- rogual 12y agoThere was TermKit (https://github.com/unconed/TermKit https://github.com/unconed/TermKit) but the author has stopped working on it.
- onan_barbarian 12y agoYes. His sentiments here http://acko.net/blog/on-termkit/ http://acko.net/blog/on-termkit/ are very close to mine: "It makes me wonder, when sitting in front of a crisp, 2.3 million pixel display (i.e. a laptop) why I'm telling those pixels to draw me a computer terminal from the 80s". I'm not crazy about every last aspect of his design there - but it's a start.
- maxerickson 12y agoThere's Acme: http://en.wikipedia.org/wiki/Acme_(text_editor) http://en.wikipedia.org/wiki/Acme_(text_editor) I have never tried to use it, and I'm not sure it's mixing the ingredients in the way you are thinking.
- dredmorbius 12y agoWhile we're talking less(1) features, one I stumbled across a few years ago when wishing really hard that less supported an interactive regex line filter, was its interactive regex line filter: &pattern Which will: Display only lines which match the pattern; lines which do not match the pattern are not displayed. If pattern is empty (if you type & immediately followed by ENTER), any filtering is turned off, and all lines are displayed. I'd prefer this followed a syntax closer to mutt's filters, and that the patterns were editable (e.g., typing '&' during a filter would show the currently extant filter for modification), but it's handy.
- dbdr 12y ago> I'd prefer [...] that the patterns were editable (e.g., typing '&' during a filter would show the currently extant filter for modification) If you type '&' then the up arrow, you'll get the previously entered pattern(s), which you can then edit.
- dredmorbius 12y agoBless you. I didn't know that. Which is why I share tips like this -- it's almost always a win.
- kazinator 12y agoNobody needs POSIX conformance in an interactive pager; that's just conformance for the sake of conformance. Or do they? What is the economic justification ("business case") for working on POSIX compliance in a "more" command? You should never invoke "more" directly in a script anyway, but rather observe the PAGER environment variable, and fall back on a plain "more" only if that isn't set. (Speaking of which, PAGER isn't described in POSIX, oops!) If the user wants the pager to exit when the last line is reached, the user can specify the necessary option in PAGER, if their pager supports it. PAGER just has to be properly expanded: treated as a command, not a command name.
- snoopybbt 12y agoModernizing less to me means using most: http://www.jedsoft.org/most/ http://www.jedsoft.org/most/