33 ms·
Xterm(1) now UTF-8 by default on OpenBSD
- igravious 11y agoI've been trying to teach myself some unicode code points because I'm getting sick and tired of continually Googling them and copying and pasting the result or bringing up a symbol character table. In fact, I'd say keyboards are woefully out to date. Specifically, I keep looking up † dagger (U+2020) and ‡ double-dagger (U+2021) for footnotes, black heart (U+2065) to be romantic, black star (U+2605) to talk about David Bowie's last album and ∞ to talk about actual non-finite entities. I olny found out recently that Ctrl+Shift+u and then type unicode hexadecimal outputs these in Ubuntu, presumably all Linuxen. AltGr+8 is great for diaeresis while we're at it so you can go all hëävÿ mëtäl really easily. edit: black heart and star are not making it through, why Lord, why?!
- scrupulusalbion 11y agoA few months ago, I had the idea to remake the old Space Cadet keyboard. One change was to make the bucky bits (e.g. control, alt, meta, super, etc.) allow you to type unicode characters instead of APL characters. Other than that and having lower case parentheses (not needing to use shift to type ( or ) ), the keyboard would be like any other mechanical keyboard.
- lokedhs 11y agoDid you follow through on it? I have wanted a modern space cadet keyboard for a long time, but the current trend in keyboards seems to be to have less keys, not more (a trend that just doesn't make much sense to me).
- JdeBP 11y agoThat's slightly ambiguous. Is your definition of a "modern space cadet" keyboard that it has more modifier types? Or that it has more keys? (Bear in mind that one could go back to a 101-key U.S. PC/AT keyboard and still have more keys than the 100-key space cadet keyboard.) For keyboards with more keys (and yet no vendor-defined private HID usages), one can look at the keyboards available in Brazil. Some of the "multimidia" keyboards from the likes of Multilaser, C3 Tech, and Leadership have the 107-key Windows ABNT2 physical layout, with anywhere up to 20 further multimedia keys. But these keyboards don't have keys engraved with more modifier types beyond the usual five.
- lokedhs 11y agoI am referring to a keyboard with a multitude of extra keys. The layout doesn't have to be exactly like the original space cadet keyboard of course, but having some 50 extra keys surrounding the normal qwerty portion, as well as some 10 modifier keys is basically what I'm looking for.
- scrupulusalbion 11y agoI think I came up with the idea about 6 months ago. It is on the back burner until I can find a way to mount the switches (Cherry MX Blues (had enough of those lying around)) [0] without resorting to a PCB. Any ideas on that front are welcome. Regarding the number of buttons, the Space Cadet had 100 buttons and no number-pad [1], whereas most modern keyboards have 104 buttons. I suppose I could add a number-pad to my design (117 buttons), but then I could also use that area for extra user-definable buttons (20 buttons in a 4x5 grid -> 120 buttons). The Space Cadet is a bit larger than most IBM-style keyboards, so more keys means more real-estate; this is not to mention yet further divergence in design from the original Space Cadet keyboard. Beyond hardware issues, there are software issues to resolve, like whether to include the macro functionality of the original. I can't find any documentation on how it worked, so I get to start from fresh. [0] = I really wanted to use some hall-effect switches, but nobody makes them anymore, because they are allegedly the most luxurious switches ever. I would probably have to tear apart an original Space Cadet keyboard to get some. Thus, I would probably just use Cherry MX switches [1] = https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/Space-cadet.jpg/1024px-Space-cadet.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/Sp...
- setra 11y agoAs someone who programs in modern APL (no joke check out dyalog) I think this is terrible. Besides the APL characters are just the greek characters, and all sorts of things use those.
- scrupulusalbion 11y agoThe original keyboard design was meant to be used to type the characters present in the APL code page [0]. My intention was that one would still be able to type the same (or perhaps mostly the same) characters as the original keyboard but using Unicode instead the APL encoding (which is based on EBCDIC, yuck). My design may or may not contain the same keys, because I am not sure how many would want the original APL character set. APL keyboards are available, so the market exists [1], but that doesn't mean much. I plan to have the micro-controller user-configurable and replaceable, so one could change what symbols were type-able with the same keyboard. As I expect to use UTF-8, this keyboard could be used to type any character in the UTF-8 code pages. [0] = https://en.wikipedia.org/wiki/APL_(codepage) https://en.wikipedia.org/wiki/APL_(codepage) [1] = https://geekhack.org/index.php?action=dlattach;topic=69386.0;attach=92387;image https://geekhack.org/index.php?action=dlattach;topic=69386.0...
- elros 11y agoOn OS X, if you type Command+Control+Space, it brings up a character insertion menu where you can search by character name. I can get both daggers, black star and black heart quite quickly that way.
- msbarnett 11y agoAlso on OS X, † is option-t, and ‡ is option-shift-7.
- igravious 11y agoOk, On Linux I have found ‡ and † † is AltGr-Shift-%, and ‡ AltGr-Shift-: I'll never remember them :(
- Tiksi 11y agoThere's a program called gucharmap which does similar on linux, hotkeys probably vary though. http://paste.click/MYRVrF http://paste.click/MYRVrF
- comex 11y agoYou can also set up custom text macros in the keyboard preferences, which are a bit faster to input. I have :lod: mapped to ಠ_ಠ...
- jcranmer 11y agoThat Ctrl+Shift+u hint is nice. Now I can type all the time I want without having to browse for the emoji page to copy it. And it sucks that I have to use so much that I know the code point for it (1F4A9) off the top of my head. :-( Edit: I'm definitely putting in U+1F4A9 (the PILE OF POO character), but apparently hacker news strips it out. I'm guessing it's filtering everything that has a symbol character class?
- igravious 11y agoYes! The Ctrl+Shift+u hint is nice. I can't believe I only just learnt it. How many _years_ have I been Googling unicode characters for? I am ashamed to think. I am glad PILE OF POO does not work for you. does (U+2603) snowman work? edit: noooo, no snowman
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- plugnburn 11y agoAnd still U+5350 works. Better than a thousand words sometimes.
- kbd 11y agoI have a stupid little 'clip' program I wrote that has a dictionary of common texts that I can call by name and have added to the clipboard. $ clip lod $ pbpaste ಠ_ಠ Maybe you can do the same without needing to remember code points. Something like TextExpander would accomplish the same thing.
- igravious 11y agoI don't think that's stupid. I think that's a great idea. I might have to use 'xclip' to go about making something equivalent.
- kbd 11y ago> I might have to use 'xclip' to go about making something equivalent. Right now my code just shells out to pbcopy on Mac, but you may be interested in pyperclip[1] which provides cross-platform access to the clipboard. [1] https://github.com/asweigart/pyperclip https://github.com/asweigart/pyperclip
- aktau 11y agoCan't remember where I got it from, but: https://github.com/aktau/dotfiles/blob/master/.zshrc-extra#L114-123 https://github.com/aktau/dotfiles/blob/master/.zshrc-extra#L...
- dylan-m 11y agoAnother really handy thing is the Compose key. If you're using GNOME it's under Keyboard Settings, under Shortcuts / Typing. I have it set to Right Alt. The idea is there's just a whole bunch of memorable key sequences for various common Unicode characters. For example, Alt + o + o = °; < + 3 = black heart, < + " = “, etc. It doesn't have all of the ones you like, but it's helpful :)
- winestock 11y agoThree years ago, I wrote a review of three programs that simulate a Compose key on Windows. I included some history behind that key, as well. https://windows.appstorm.net/roundups/utilities-roundups/add-a-virtual-compose-key-to-your-pc/ https://windows.appstorm.net/roundups/utilities-roundups/add...
- ori_b 11y agoI usually map my insert key to compose -- I never use 'insert' for the default functionality, but I also don't type funny characters often enough to justify getting rid of an alt or control key.
- Zancarius 11y agoI love reading about how different people map different things. For me, it would be a disaster to map insert since I use that for pasting (shift+insert--I'm a lefty and it's a bad habit I grew attached to years ago). My choice for Compose is the right Windows key, which I think I eventually settled on because I use the left one in some keybinds (winkey+s for shell, etc.) and like you, I couldn't part with an alt or ctrl. I've often wondered what other folks tend to use. To the grandparent: I'm sometimes amused by what Compose defines. There's ∞ (compose + 8 + 8), (compose + # + #), and oddly (compose + C + C + C + P). I think it may depend on system configuration, but I believe libX11 is responsible. (On my system, Arch, the key combinations appear to be documented under /usr/share/doc/libX11/i18n/compose for my locale.)
- davidp 11y agoVery cool, thanks. I found a page that shows all the key combinations: [0]: https://help.ubuntu.com/community/GtkComposeTable https://help.ubuntu.com/community/GtkComposeTable
- unfamiliar 11y agoI have an Alfred workflow that fuzzy-searches through all unicode characters by name and inserts the character when selected. All it takes is a good interface to make it fluid.
- scrollaway 11y agoYou can make your own keyboard layouts with X11. This is mine: https://github.com/jleclanche/dotfiles/blob/master/X11/xkb/symbols/us https://github.com/jleclanche/dotfiles/blob/master/X11/xkb/s...
- Grue3 11y agoCtrl-Shift-u also works in GIMP, even on Windows. I guess it's a GTK feature.
- wtbob 11y agoXCompose is your friend: black hear suit is Compose-< 3, and with additional configuration[1] dagger can be Compose-| -, double dagger Compose-| = and black star Compose-S S. [1] https://github.com/cofi/dotfiles/blob/master/XCompose https://github.com/cofi/dotfiles/blob/master/XCompose
- JdeBP 11y ago> In fact, I'd say keyboards are woefully out to date. I wrote a virtual terminal subsystem a while ago. I gave it keyboard layouts with the ISO 9995-3 common secondary group. No daggers, alas. But ISO 9995-3 does have pretty much all of the combining diacritical marks. <Group2> <Level3>+D05 is combining diaeresis. In practice I find myself not appreciating that as much as I appreciate being able to type U+00A7 as <Group2> <Level2>+C02.
- zkirill 11y agoThis is really great! Just a few days ago I got very confused when I saw tofu characters in xterm and had to switch to uxterm to see them (or set some locale flag in my home dir).
- deleted 11y ago[deleted]
- kazinator 11y agoGreat! Now just drop the embarrassing man(1) page reference, and you can call it modernized. Wow, I'm surprised that the people whose buttons this pushes are able to make(1) a HN account, let alone have enough points to downvote. Think about it. There is only one man page for xterm. I fyou type "man xterm" with no section number you get that man page. If there existed an xterm(7) page, you'd still get the xterm(1) man page by default. So why the hell write the (1) notation every time you type the word xterm? Man page section numbers are not useful or relevant, by and large and mentioning them only adds noise to a paragraph. Even stupider is when the worst of the Unix wankers write man page section numbers after ISO C function names. Example sentence: "Microsoft's malloc(3) implementation is found in MSVCRT.DLL". #facepalm#
- klodolph 11y agoDon't take the downvotes personally, it's just uninteresting content getting moderated.
- neerdowell 11y agoOpenBSD's malloc(3) implementation is found in sys/kern/kern_malloc.c, and OpenBSD's malloc(9) implementation is found in lib/libc/stdlib/malloc.c
- gjvc 11y ago>Think about it. There is only one man page for xterm. I fyou type "man xterm" with no section number you get that man page. If there existed an xterm(7) page, you'd still get the xterm(1) man page by default. So why the hell write the (1) notation every time you type the word xterm? Because the convention exists to define the type of the component. It's a handy convention, and I'm betting there are a few people reading this who have never used anything other than GNOME terminal so appending the section number immediately helps the reader to place the component, otherwise they'd have to look it up. etc
- jhallenworld 11y agoI and others have pushed changes into XTerm to improve mouse support of terminal-based applications. All terminal emulators should implement XTerm's command set, especially these: Bracketed paste mode: allows editor to determine that text is from a mouse paste instead of typed in. This way, the editor can disable auto-indent and other things which can mess up the paste. Libvte now supports this! Base64 selection transfer: this is a further enhancement which allows the editor to query or submit selection text to the X server. This allows editors to fully control the selection process, for example to allow the selection to extend through the edit buffer instead of just the terminal emulator's contents. One patch of mine didn't take, but I think it's still needed: allow mouse drag events to be reported even if the coordinates extend beyond the xterm window frame. Along with this is the ability to report negative coordinates if the mouse is above or to the left of the window. Why would this be needed? Think of selecting text which is scrolled off the window. The distance between edge and the mouse controls the rate of selection scrolling in that direction. BTW, it's fun to peruse xterm's change log. For example, you can see all the bugs and enhancements from Bram Moolenaar for VIM. http://invisible-island.net/xterm/xterm.log.html http://invisible-island.net/xterm/xterm.log.html Thomas Dickey maintains a lot of other software as well, in particular ncurses, vile and lynx: http://invisible-island.net/ http://invisible-island.net/
- caf 11y agoBracketed paste mode is also useful for IRC, to prevent misfiring a huge paste into a channel.
- deleted 11y ago[deleted]
- sherr 11y agoYes, Thomas Dickey's been maintaining xterm and a lot else for donkey's years now. A lot of people owe him a big "thank you" for all his hard work. Thanks Thomas.
- thisrod 11y agoThis reminded me of a Rob Pike comment. I can't find the text, but it was along the lines of, "I recently tried Linux. It was as if every bug I fixed in the 1980s had reverted."
- kazinator 11y agoThat was baseless posturing. A famous study and its follow-up found that the utilities on GNU/Linux are more robust, and that was twenty years ago: ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-revisited.pdf [1995] "This study parallels our 1990 study (that tested only the basic UNIX utilities); all systems that we compared between 1990 and 1995 noticeably improved in reliability, but still had significant rates of failure. The reliability of the basic utilities from GNU and Linux were noticeably better than those of the commercial systems." I doubt there has been much improvement in those commercial Unixes; they are basically dead. (What would be the business case for fixing something in userland utility on commerical Unix?) The maintainers of the free BSD's have been carrying that torch, but they don't believe in features. Stepping into a BSD variant is like a trip back to the 1980's. Not exactly the real 1980's, but a parallel 1980's in which Unix is more robust---but the features are all rolled back, so it's just about as unpleasant to use.
- setra 11y agoThen the commercial systems must have been horrible. Take a look at GNU code, then at plan 9 code. Combine a few GNU core utils, and you have more code than the whole plan 9 kernel. Granted Plan 9 came out a little passed 1990.
- kazinator 11y agoOnly by not using subordinate clauses did you just avoid saying "plan 9" and "commercial" in the same sentence! Where is Plan 9 deployed? Who are the customers? Plan 9 is a strawman representative of "commercial Unix". > Combine a few GNU core utils, and you have more code than the whole plan 9 kernel. When you actually sit down and think of the cases that can occur, that translates into code. Handle this, handle that, and the KLOC's just pile up. Speaking of kernel size, what makes a kernel code base big? Why, device drivers for lots of hardware.
- spedru 11y agoEvery time some link or headline reads "now UTF-8 by default", the only reasonable response in 2016 is "about time".
- JoachimSchipper 11y agoThat's not why this article is interesting. Rather, it highlights how profoundly not UTF-8 ready the (terminal) world is. (It does work in practice, but in-band signaling over a channel carrying complex data that receiver and sender interpret according to settings that do not appear in the protocol at all is, predictably, terrible.)
- mobystrip 11y agoThe reason for their delay in moving to it was this. So that when eventually someone found a way to exploit a system using utf-8 rendering, they could show how much more secure they were by proclaiming "We are so secure this bug does not affect us!".
- deathanatos 11y agoRepeat after me: UTF-8 is the sane default in this day and age. This is a good change. The whole "the ISO 6429 C1 control code 'application program command'" thing is a bit surprising though. (I'm guessing this change doesn't actually avoid this directly? If you sent an APC it'd still do it, it's just that APC is multiple bytes in UTF-8, and hopefully a bit rarer?) > Reinterpreting US-ASCII in an arbitrary encoding This way will likely work — at least, I thought. The vast majority of encodings are a superset of ASCII, so reinterpreting ASCII as them is valid. The only one I know of that isn't is EBCDIC, and I've never seen it used. (Said differently, non-superset-of-ASCII codecs are incredible rare to encounter, so the above assumption usually holds.) (The reverse, reinterpreting arbitrary data as ASCII, is not going to work out as well.) Though it is rather horrifying how easily it is to dump arbitrary data into a terminals stream. Unix does not make this easy for the program. The vast majority of programs, I'd say, really just want to output text. Yet, they're connected to a terminal. Or better, if perhaps a program could say, "I'm outputting arbitrary binary data", or even "I'm outputting a application/tar+gzip"; the terminal would then know immediately to not interpret this input. And in the case of tar+gzip, it would have the opportunity to do something truly magical: it could visualize the octets (since trying to interpret a gzip as UTF-8 is insane); it could even just note that the output was a tar, and list the tar's contents like tar -t. If the program declares itself aware, like "application/terminal.ansi", then okay, you know: it's aware; interpret away. But it doesn't, so it can't. Part of the difficulty is probably that the TTY is both input and output (not that the input can't also declare a mimetype or something similar). And the vast majority of programs don't escape their user input before sending it to a terminal; it's like one giant "terminal-XSS" or "SQL-injection-for-your-terminal". And it is probably unreasonable to expect it; I don't really know of any good libraries around terminal I/O; most programs I see that do it assume the world is an xterm and just encode the raw bytes, right there, and pray w.r.t. user input. catting the linux kernel's gzip into tmux can have consequences from "lol" to "I guess we need a new tmux session". It was also just today that I discovered that neither GNU's `ps` nor `screen` support Unicode, at least, for characters outside the BMP.
- comex 11y agoUTF-16 isn't a superset of ASCII, for one. Doesn't seem that anyone uses a native UTF-16 terminal, but if you're trying to use grep or whatnot on a UTF-16 encoded file, it'll happily silently not do what you want...
- plugnburn 11y agoUTF-8 must be the default and only encoding. Why does anything else still exist?
- jmnicolas 11y agoYes but UTF-8 with or without byte order mark ? ;-)
- plugnburn 11y agoWithout. BOM (when used for UTF-8) is an obsolete crap invented by necrosoft in order to make their software incompatible with normal.
- TazeTSchnitzel 11y agoIt's not a Microsoft invention, and MS's use of it is really quite sensible. They had a problem of distinguishing UTF-16, UTF-8 and non-Unicode (possibly a single-byte "extended ASCII" type encoding, possibly some multi-byte monstrosity) text files. Since UTF-8 and ASCII-compatible encodings look similar when there aren't many >U+007F characters in use, and identical if none are in use, they could get confused. Prepending a Byte Order Mark solves this problem, in that it makes a file unambiguously UTF-8 (or UTF-16, for that matter).
- tempodox 11y agoHow do you have a BOM in the shell?
- plugnburn 11y agoSome masochist M$-fan could invent even this just in order to justify the difference from civilized world.
- Tiksi 11y agoANSI must be the default and only encoding. Why does anything else still exist?
- gpvos 11y agoWouldn't it be better if all those dangerous escape sequences (like Application Program-Control, redefining function keys, alternate character sets, etc.) were disabled by default in xterm? Anyone using the obsolete software that uses them could enable them if they wish.