9 ms·

Make your STDERR red

• joh6nn 15y ago
  thank you. this is probably the cleanest way to do this i've seen, and i've seen a LOT of ways to do this.
  • anon_d 15y ago
    How the hell is this even remotely clean?
    • burgerbrain 15y ago
      Yeah, this is a tad hackish for my tastes honestly, makes to many assumptions. The way I came up with while brainstorming a while ago (never got around to fully implementing it) is to create a PTY wrapper that you can fire off a program with. Basically it creates a new PTY and wires it up to it's own controlling PTY, and forks of a child under the new one. You can then trivially do a lot of things transparently, like separating stdout and stderr (normally both stdout and stderr would be attached to the slave side of the PTY (/dev/tty), but if you attach them to a different fd then the select on the master/parent side can do things like color them). EDIT: if it helps you picture it, this is basically doing userland STREAMS ;) I'd estimate you could do this for around 250 lines of C. Maybe I'll see what I can do after dinner.
      • burgerbrain 15y ago
        Ok, banged this together: https://gist.github.com/1474748 https://gist.github.com/1474748 Based off jgreco's 'vindication': https://github.com/jgreco/vindication https://github.com/jgreco/vindication Usage: `./redwrapper python -c 'import os; print "Yo!"; os.write(2, "Jola\n\r");'` `./redwrapper zsh` (zsh works with it, but bash doesn't currently for some stupid reason. something about what it expecting stderr to be /dev/tty or something I believe). This is rough code, if you want to use regularly/seriously I suggest reviewing the code. I didn't pay attention much to standards compliance, I've only tested on Linux right now. I know that PTY stuff can get hairy on different *nix's, so beware of that. EDIT: whoops, remove that '-g -lefence' from the Makefile too before you use this.
        • roel_v 15y ago
          You'd have to modify every script you ever run on a machine, and modify every command you ever type in, to make this work. Any serious solution to this problem needs to be 100% transparent to the user - i.e., wrapper programs are not an option.
          • joeyh 15y ago
            Or you can just modify screen(1)
            • deleted 15y ago
              [deleted]
          • burgerbrain 15y ago
            No, you can just run your shell under it. You certainly wouldn't want to actually modify anything. The following sequence works: `./redwrapper zsh `python -c 'import os; print "Yo!"; os.write(2, "Jola\n\r");'` This is because the values of stderr/stdout are inherited, unless they are purposely overwritten (redirection or anything that allocates it's own PTY (like 'ssh'), both of which of course disables 'red-ification'). Anyway, neither of these solutions are something I would ever deploy in an "always on" setup (although it could be done in mine as well). You certainly wouldn't want to do it for your users, so they're always going to have to type something (or mess their shell's init files, which should work for mine as well). So this is about as transparent to the user as I dare make it. Unfortunately it is not 100% transparent to the programs you are running under it since they can figure out that stderr is not /dev/tty. Nothing seems to care, with the exception of bash, which uses stderr for seemingly everything inexplicably.
      • bdonlan 15y ago
        If you split stdout and stderr, you can end up with the order of output changing - your output process might not get around to reading its input ends until it has data at both, and then it can't tell which is which.
        • burgerbrain 15y ago
          "If you split stdout and stderr, you can end up with the order of output changing" That is true. I find it doesn't matter terribly much in practice though. Obviously with some programs it might. "your output process might not get around to reading its input ends until it has data at both, and then it can't tell which is which." I'm not quite sure what you mean.
    • joh6nn 15y ago
      "cleanest way i've seen" != "ideal" it isn't a horrible mangling of pipes and/or file descriptors, or some custom wrapper script that you have to run before every command, or any of the other frightening ways to do this that i've seen. it's a simple library that you can load or unload easily, that works system wide, without being an impenetrable, unmaintainable mess. as far as these things go, that's clean.
• J_Darnley 15y ago
  Er, why would you want all the text to be red?
  • burgerbrain 15y ago
    Not all of it, just stderr (so not stdout). This is useful because many things use stdout for regular output and stderr for error output (some things just use stdout for both, but that's not very nice as it disables doing things like this).
    • bouncingsoul 15y ago
      I think they're suggesting just styling "Error:" in red and then using normal text for the actual message. npm does something like that: http://i.imgur.com/7oBLP.png http://i.imgur.com/7oBLP.png
      • burgerbrain 15y ago
        Well you can do that with a standard unix filter. This does all of stderr.
        • bouncingsoul 15y ago
          Whoops. I thought this was a tool for script authors, not the end user. Now I get it, and I think it's way cool. And I think J_Darnley doesn't like red.
  • wladimir 15y ago
    Well if you don't like red, with a 256-color terminal emulator you could change it into a nice shade of purple instead: static const char PURPLE[] = "\x1b[38;5;129m" #define STDERR_COLOR PURPLE Or something more sane: http://www.mudpedia.org/wiki/Xterm_256_colors http://www.mudpedia.org/wiki/Xterm_256_colors
  • J_Darnley 15y ago
    To reply to everyone here, I still don't get why you want all text on stderr to be coloured, be it red or purple. Most programs can write data to stdout so they put all messages on stderr. Making it all coloured hides the errors in the rest of the text. Making errors red, or any other colour, is a good idea. It makes "under-experienced users" notice them.
• Xuzz 15y ago
  On OSX, you might be able to do something similar using dyld's interposing feature (http://opensource.apple.com/source/dyld/dyld-132.13/include/mach-o/dyld-interposing.h http://opensource.apple.com/source/dyld/dyld-132.13/include/...) to replace write() and the dyld equivalent to LD_PRELOAD, DYLD_INSERT_LIBRARIES.
• wmobit 15y ago
  The alloca usage here terrifies me.
  • burgerbrain 15y ago
    That is a good point. I don't know if write has a limit on 'count' (I can't find anything saying that it does), but if it's simply "whatever you can fit in size_t" then you certainly want to be careful about putting that on your stack. Failed alloca's don't really give you any indication, though you might sigsegv later.
    • sickill 15y ago
      What would you suggest instead then? I'm not C programmer.
      • burgerbrain 15y ago
        Avoid the allocation entirely. Use writev for example (or just use multiple 'write's, if you don't care about it being atomic).
        • sickill 15y ago
          Thx. Improved version in master: https://github.com/sickill/stderred/blob/master/stderred.c https://github.com/sickill/stderred/blob/master/stderred.c
• tlrobinson 15y ago
  Nice hack. Ideally it would just be a feature of the shell. At least it does a isatty check though. Is there a reason you allocate a bunch of memory and copy the strings rather than just doing multiple writes? (*lol_write)(fd, STDERR_COLOR, STDERR_COLOR_SIZE); (*lol_write)(fd, buf, count); (*lol_write)(fd, COL_RESET, COL_RESET_SIZE);
  • bdonlan 15y ago
    Or use writev for that matter. Note that using ordinary write multiple times has atomicity implications - if you have multiple processes writing to STDERR, you might find their output interleaved in unfortunate ways. Using a single write with a temporary buffer or writev avoids this (provided you're not writing more than PIPE_BUF bytes total)
    • nviennot 15y ago
      Indeed: https://github.com/sickill/stderred/pull/1 https://github.com/sickill/stderred/pull/1
      • agwa 15y ago
        I agree that writev is a better approach. But your patch doesn't look thread-safe because it uses that static iovec struct.
        • nviennot 15y ago
          You are right, and there are other issues: - I don't return the proper return value (should not return the number of bytes from the color strings) - I don't handle the case where writev() returns a number of bytes between 0 and count (e.g. You are writing 10Mb of data, a signal arrives, and you only wrote 5Mb -- the color thing will get all confused)
  • kelnos 15y ago
    The reason would be interleaving if more than one process is writing to stderr. I was more bothered by the use of alloca(). Even on a 32-bit system with a 32-bit size_t, you can blow your stack to hell with a single call to this modified write(). Having to do a full malloc() here would be pretty lame, but as others suggested, writev() should do nicely. The other nice property of using writev() is that you don't have to bother with the dlopen/dlsym crap. Can just call writev() directly from the overridden write() in either case. Of course, that doesn't catch people writing to stderr using writev() directly, but I think that's ok. (The current impl doesn't catch that case anyway.) (edit: pull request submitted! https://github.com/sickill/stderred/pull/6 https://github.com/sickill/stderred/pull/6)
• derleth 15y ago
  Ubuntu does indeed allow multi-arch distros. I've run 32-bit Firefox on 64-bit Ubuntu for years; in fact, I'm doing so now.
  • sickill 15y ago
    Allright, maybe. But it still has DST ($LIB token) disabled in LD_PRELOAD.
    • derleth 15y ago
      Maybe? What's maybe about it? I can build (using -m32) and run 32-bit x86 binaries on an x86-64 platform.
      • sickill 15y ago
        I didn't mean that you can't run 32-bit binaries on 64-bit OS, what I meant was that AFAIK you can't install 32-bit deb packages together with 64-bit ones on 64-bit Ubuntu. Correct me if I'm wrong.
        • obtu 15y ago
          You can install both. The new multiarch support builds blahblah:i386 and blahblah (implicitly blahblah:amd64) debs for source packages that have it enabled. Before that, packages that wanted to provide i386 binaries on amd64 wrote their own extra build support, and added a suffix to the binary package names to avoid collisions. Here's what I found on how to support LD_PRELOAD without a $LIB dynamic string token: Looking at this: http://lists.debian.org/debian-devel/2011/12/msg00300.html http://lists.debian.org/debian-devel/2011/12/msg00300.html it seems you could put a relative path in LD_PRELOAD and count on the linker to use architecture-dependent search paths. I'll also link these two threads for context, though they didn't provide answers: http://comments.gmane.org/gmane.comp.lib.glibc.user/974 http://comments.gmane.org/gmane.comp.lib.glibc.user/974 (chatting) http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=372626 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=372626 (concatenating the two libs in LD_PRELOAD works, but is noisy).
• tlrobinson 15y ago
  On OS X: gcc stderred.c -D_GNU_SOURCE -Wall -ldl -fPIC -shared -o lib/stderred.dyld alias stderred='DYLD_INSERT_LIBRARIES=$PWD/lib/stderred.dyld DYLD_FORCE_FLAT_NAMESPACE=1' stderred python -c 'import os; print "Yo!"; os.write(2, "Jola\n\r")' (replace $PWD with the absolute path if you're adding this to your shell config, of course)
  • sickill 15y ago
    Cool, looks like the same gcc flags works for OSX.
• raldi 15y ago
  Why override write() when you can just use pipes? http://distfiles.exherbo.org/distfiles/hilite-1.5.c http://distfiles.exherbo.org/distfiles/hilite-1.5.c
  • vilhelm_s 15y ago
    One problem is stdio buffering. By default the C stdio functions will flush buffers after every line for file descriptors that are connected to terminals, but only every now and then for file descriptors connected to disk files or pipes. So if a program outputs messages on both stdout and stderr (like a compiler), then after filtering one of the streams through a pipe, the messages are not interleaved properly.
    • colmmacc 15y ago
      One way is to wrap the program with a parent process that can handle the colourising. Here's a version I've been using for 6 years or so; http://people.apache.org/~colm/utils/sexec.c http://people.apache.org/~colm/utils/sexec.c it uses regular select semantics and a trivial state machine to pick the active fd, and has an XML mode - which is convenient for colourising in XHTML (if you want to record an automated process say).
      • vilhelm_s 15y ago
        Here is an example illustrating what I mean: #include <stdio.h> int main() { while (1) { fprintf(stdout, "Hi.\n"); fprintf(stderr, "Hello.\n"); } return 0; } If you run this program directly from the shell, every odd line says Hi and every even line says Hello. But if you run it through sexec (on a Linux machine at least), you get a large block of lines saying Hi, followed by a large block saying Hello, etc. Whether this is a problem or not of course depends on the use case. But to avoid it you need something more sophisticated than just pipes, e.g. the LD_PRELOAD hack in the original post.
        • colmmacc 15y ago
• udp 15y ago
  I may be wrong, but I always thought stderr was for diagnostic messages as well as errors, in which case red wouldn't really be appropriate. I almost think we need another output stream for something that's neither an error nor output of the program.
  • jzwinck 15y ago
    C++ has std::clog (http://www.cplusplus.com/reference/iostream/clog/ http://www.cplusplus.com/reference/iostream/clog/) but most people haven't heard of it and it maps to stderr anyway.
  • Jach 15y ago
    I don't know, I kind of have mixed feelings about adding another stream. I'd probably find stdinfo and stdwarn useful. But it's a slippery slope to duplicating the features of a full logging system; you might add stdinfo, stdwarn, stdfine, stdfiner, stdfinest, stdconfig, std$custom, etc. etc. One nice thing about having only stdout and stderr is it pushes you to ask: "Is this piece of information actually necessary to display to the user or should it just be logged to a file somewhere?" I'm not sure everyone really asks that though. Every time someone shoves a bunch of needless information to stderr "just in case" is a time I have to 2> /dev/null. (At least the useless stuff is often in stderr so I don't have to grep it out of stdout.)
  • anko 15y ago
    I think this is a marvellous idea. I would love to see a meta stream, that allows you to mark up semantics of stdout/stderr. How would you go about creating a new standard stream?
• strlen 15y ago
  Why did you truncate a size_t to an int?
  • sickill 15y ago
    Will fix that, thx.
• apinstein 15y ago
  Or, add this to .zshrc: # colorize all stderr red exec 2>>(while read line; do print '\e[91m'${(q)line}'\e[0m' > /dev/tty; print -n $'\0'; done &) works like a charm.
  • 1amzave 15y ago
    Though I'm not a zsh user, my first reaction was likewise "meh, my shell can do that", and so I whipped up a pretty much identical command (only using sed). However, when it didn't behave quite as I expected, I dug a little deeper and ended up with (what was to me) a curious discovery: bash prints its prompt to stderr rather than stdout (see http://git.savannah.gnu.org/cgit/bash.git/tree/parse.y#n5013 http://git.savannah.gnu.org/cgit/bash.git/tree/parse.y#n5013). Anyone know why it does that?
    • burgerbrain 15y ago
      That's a great question. Zsh and tcsh both use stdout for their prompt, ksh and dash use stderr for the prompt itself but stdout for whatever the user is typing into the prompt. Bash, at least as it's configured on my system, seems to use stderr for the prompt and whatever the user types on it. I really can't think of a reason that the way tcsh and zsh handle this isn't the proper way. Surely the principle of least surprise applies here.
  • sickill 15y ago
    This is the solution I started with. However it's messing line order sometimes due to race conditions/buffering.
    • apinstein 15y ago
      Yeah we're getting that issue, too. And sometimes the prompt doesn't appear properly.
• tzury 15y ago
  This is my python version (for logger) https://gist.github.com/1474991 https://gist.github.com/1474991
• noeleon 15y ago
  can it somehow incorporate the blink tag as well?
  • burgerbrain 15y ago
    (Unfortunately) yes, only some terminal emulators support it though.
• bdonlan 15y ago
  This actually doesn't work for processes using stderr on eglibc 2.13 - fprintf internally outputs via a different path that can't be hooked in this manner. On x86, you could try overwriting the system call gate pointer to a hook function, but on x86_64 you're basically stuck, unless you attach a debugger or something. Also, the write prototype is incorrect (using int instead of size_t) and could break on 64-bit machines.
  • sickill 15y ago
    Thx, will fix int -> size_t.
• mwsherman 15y ago
  I like command lines, but often make fun of people that prefer them over well-produced GUIs. I suppose the reason is that well-produced GUIs are rare. That being said: turning text red is notable? In 2011? (Will accept downvotes with dignity.)
  • cleverjake 15y ago
    It's less of a unique idea, and more of a why-didn't-I-think-of-that sort of thing, in my opinion.
  • wladimir 15y ago
    What, you're trying to start a GUI versus command-line argument? On hacker news? In 2011?
  • bnegreve 15y ago
    Turning all error messages red systemwide is a neat result.
• deleted 15y ago
  [deleted]
• deleted 15y ago
  [deleted]
• deleted 15y ago
  [deleted]
• deleted 15y ago
  [deleted]
• sovande 15y ago
  The concept may be useful for someone, but the implementation is just slow, naive and definitely not something you want to interpose your libc write function with. If the author is reading this; 'man (2) writev' for a quick code and efficiency improvement.
  • sickill 15y ago
    I am not C programmer. Actually this piece of code is my first C code since over 10 years. I might have forgotten about all good practices and stuff. But it works for me so I shared. Would you like to contact me and point slow, naive points or even better improve it and send pull request?
    • deleted 15y ago
      [deleted]
    • sovande 15y ago
      The slow point mainly refers to allocating and copy into a new buffer. The naive points refers to that you use 1) alloca to allocate an unknown amount of memory on the stack and 2) assume that write is atomic, never fails and will write the whole buffer (you return count). Instead you should return what write actually wrote or not. There are also a whole can of worms with regards to what happens during (signal) interrupt or if write failed to write the whole buffer. In which case the reset color code may not be sent etc etc. Solutions to this require some fine-thinking, but for the slow stuff, you can simply replace all the allocation and copy with something like this struct iovec iov[3] = {{STDERR_COLOR, STDERR_COLOR_SIZE},{buf, count},{COL_RESET, COL_RESET_SIZE}}; ssize_t n = 0; do { n = writev(2, iov, 3);} while (n == -1 && errno == EINTR); return n;
• cnvogel 15y ago
  While it might not be relevant in practice, there's the possibility of data corruption if someone closes all filedescriptors (for whatever reason) and then operates on the newly generated ones. This implementation just checks for filedescriptor number 2, a more sensible implementation for sure should hook the stdio FILE or iostream mechanisms to restrict colorization only to the "standard error" case. for(i=0;i<1024;i++) close(i); /* close a lot of fd / / open will yield the lowest free fds, here it's 0,1,2 / f=open("inputfile",O_RDONLY); g=open("outputfile1",O_RDWR|O_CREAT,0666); h=open("outputfile2",O_RDWR|O_CREAT,0666); read(f,buf,1024); write(g,buf,1024); write(h,buf,1024); / <-- data is colorized! */
  • kelnos 15y ago
    The isatty() check should cover most cases of badness. Assuming "outputfile2" is a regular file, the data will not get colorized in your example.
  • sickill 15y ago
    Thx for catching this. Interesting in making pull request? ;)
    • sickill 15y ago
      Btw, as kelnos said, isatty() check "should" prevent all bad scenarios.
• jrabone 15y ago
  Well, I suppose it's one step up from embedding ANSI escape sequences in your output. Various crappy Ruby tools do this and it makes Windows development SUCH a joy. Free clue: the world is not a 1960s UNIX terminal. If you don't want to support Windows, that's fine - don't. I'll use something else or write my own. If you DO claim to "work" on Windows, read the goddamn docs and don't just blindly emit escape sequences on std(out|err) because it DOES NOT WORK. It's not that hard to support Windows console IO, and it's well documented. I've previously written a (commercial) Java native library to provide curses-style character-at-a-time support for Java running in a terminal, using curses on UNIXalikes and a choice of Win32 console or Cygwin curses on Windows, although Cygwin curses at the time didn't correctly recognise Ctrl-Space (makes supporting Emacs keybindings a non-starter). Win32 has a considerably saner console implementation IMO.
  • roel_v 15y ago
    It's quite a pain though to get console coloring to work properly on Windows. I wrote a utility once that batch scripts could call to set the fore/background color (just a call to SetConsoleTextAttribute() really) but in practice it's quite a pain to write scripts that color their output properly (especially once you start mixing languages of those scripts). The terminal-emulator-that-interprets-escape-sequences-as-color-change-commands is much easier for that purpose.
  • alextingle 15y ago
    What a bizarre rant. This is a libc/LD_PRELOAD hack, so it's clearly a Unix-only solution. Where did all these "you'll break my Windows console" tears come from??
    • jrabone 15y ago
      What's bizarre about it? Clearly this library isn't the problem. What'd be even better would be a sane console standard that isn't stuck with being backwards compatible with a 7 bit everything-is-a-character protocol. The acoustic coupler is dead people, get over it! Relegate termcap / terminfo / curses / getty to the same bin we dumped Gandalf serial line multiplexors in, and move forward!
• anko 15y ago
  with the stderred LD_PRELOAD alias this works for me: stderred python -c 'import os; print "Yo!"; os.write(2, "Jola\n\r")' but this doesn't; stderred ruby -e 'puts "Yo!";STDERR.write "Jola\n\r"' if i do an strace the writes are going to the same file descriptors.. any ideas?
  • sickill 15y ago
    Maybe ruby's write() is not using write() from libc but other function like fprintf?