8 ms·
"River" detection in text
- drivebyacct2 14y agoThis is massively distracting for me at certain times and I was always considered a "good, fast" reader. The only thing I detest more in a block of text than rivers, is justified alignment. Ick.
- hzy 14y agojustified alignment has a time and a place (newspapers), but for the most part I agree, the differing line lengths make keeping your place far easier.
- ygra 14y agoIn general text in multiple columns; ragged-right columnised text is sometimes horrible to read.
- homosaur 14y agoGood typography is about making the correct decisions regarding your text. If justification is making a passage hard to read then it's simply bad typography and doesn't mean justification is bad as a concept. Ragged-right isn't necessarily going to save you from rivers either although they are far less likely due to uniform spacing.
- drivebyacct2 14y agoOh, I know. I hate them as separate entities. I'm not even convinced necessarily that justified text causes more rivers - they make awkward gaps and ruin my pace, but I don't usually get distracted by rivers with justified text.
- barrkel 14y agoJustification without gaps works best with aggressive hyphenation. If the hyphenation isn't good, the text is full of surprises at line wraps. It's hard to do justification well as a result. It's usually restricted to newspapers with limited space these days.
- iSnow 14y agoAnd different languages - German, for example, has massively longer words, so the problem is worse over here.
- andreasvc 14y ago> It's usually restricted to newspapers with limited space these days That's false. Almost any printed publication uses justification and hyphenation by default: books, magazines, scientific papers. The big exception is websites.
- homosaur 14y agoThey usually do cause more rivers simply because on many lines the spaces are significantly larger. Since you only would letterspace as an absolute last resort (and probably not even then), it tends to create more dangerous whitespace per line.
- jtchang 14y agoDefinitely cool. How about the reverse for an art project? Maybe the Constitution or Bill of Rights with rivers forming the statue of liberty.
- shitlord 14y agoSomething similar: http://stackoverflow.com/questions/8479058/how-do-i-find-waldo-with-mathematica http://stackoverflow.com/questions/8479058/how-do-i-find-wal... These sorts of image processing questions are incredibly entertaining imo (and I don't mean it in a bad way).
- mcav 14y agoNext, someone should put that in LaTeX to automatically fix it rather than just detect it. (Or some sort of HTML/JS plugin, since using LaTeX again is somewhat disconnected from my immediate life goals).
- fusiongyro 14y agoI wouldn't hold my breath. This sounds like a huge, messy, heuristic coding effort to solve a rare, über cosmetic problem. TeX and typography in general seem to attract really obsessive folks, but this doesn't strike me as harboring a hidden ideal that's computationally well-defined and feasible the way other parts of TeX turned out to be. I guess we'll find out if somebody does it.
- jamesjporter 14y agoI like the idea that mistercow proposes above: use an algorithm to sample the space of possible typsettings and find one with less that some minimum amount of rivers or river length.
- fusiongyro 14y agoI get that, but here's the problem: there's no guarantee that any perturbation isn't going to produce more rivers. It's guess-and-check. For simulated annealing to work, you have to believe that large perturbations are more likely to produce large differences and small ones produce small differences. I am not convinced that property holds here, because a river is an optical phenomenon that doesn't directly relate to the spacing size, which is the one variable you're interested in manipulating. A small change in space might have a small effect on the river you noticed, but create an entirely new and worse one. Another problem with simulated annealing is that you want to start with large changes and work your way to small changes, but TeX has already optimized the spacing, so you would really rather start with small changes. Your next problem is that TeX doesn't know what glyphs look like, just how much space they take up. If the "riverbanks" are made up of As and Vs, the river will be extremely pronounced. But it may not be noticeable at all if they're made up of Ms and Xs. Worse, it will also depend on the font in play. But TeX doesn't know what the glyphs look like while it's adjusting the whitespace, so you now need to introduce a feedback loop between the eject-page phase and the paragraph layout phase. I'm under the impression you'd be jumping over three or four phases to do that, because TeX lays out lines first, then paragraphs, then emits pages. This is a huge amount of work. How big a problem are rivers?
- virb 14y agoIs this really a problem? In how many texts does this occur? It seems quite unlikely. Show me a book where this has happened.
- daeken 14y agoI see it all the time on my Kindle at least (I don't read paper books anymore, due to nerve damage in my wrists). That said, I actually quite enjoy seeing 'rivers' in text, for whatever odd reason.
- anigbrowl 14y agoSo do I. I'm rather annoyed at this attempt to engineer them away.
- PavlovsCat 14y agoAh, but remember that this sword cuts both ways! That is, these methods can also be used to ensure a given text is displayed with the maximum possible river amount and length.
- anigbrowl 14y agoOK, you've convinced me :-)
- kenko 14y agoYou are unlikely to see many books where this has happened, because professionals try to make sure that it doesn't, and fiddle with things if it does.
- mistercow 14y agoI'm pretty sure I've seen it in plenty of books, and it's pretty distracting.
- 14y ago
- d23 14y agoI've been seeing these all my life and for some reason never thought to ask if anyone else noticed them. I never found them particularly distracting and didn't know they were a sign of bad typography. Still, cool to see an article about them.
- firefoxman1 14y agoSame here. Did you tend to notice them out the corner of hour eye more than straight-on?
- tekromancr 14y agoI think the vision mode that detects rivers is more of a "big-picture, perhriphery" type where the vision mode that you use for for reading is a focused, small area type. I think you will always catch them in the big picture, but may not notice when reading.
- pygy_ 14y agoThe peripheral vision is also blurry. Rivers stand out more in blurred text, even in central vision.
- martinced 14y agoDefinitely. People typesetting books are supposed to get rid of them but it's a manual process. As I've written in another comment I'd typically "blur" my vision a bit and quickly scan all the pages of the book to track these down (on screen) and then remove them. Don't know what was the technique of other typesetters when it came down to finding these. Some proofreaders would note them should they find / notice any but some wouldn't. Now that I think of it, using OP's findings it would probably wouldn't really be too hard to write a "plugin" taking screenshots of Quark XPress / Adobe InDesign / LaTeX preview and showing "rivers" in a HUD on top of the screen-rendered pages. This would make chasing "rivers" quite easy.
- homosaur 14y agoIf you start to study typography at a relatively deep level, there will be two things that will absolutely kill you on a daily basis. One is rivers. The second is bad kerning on signs/displays.
- mistercow 14y agoI wonder how hard it would be, given the detection algorithm, to fix this automatically. Seems like simulated annealing would be a pretty good fit, where you perturb the word spacing on each iteration and the energy function is the total river length. The fitness landscape is rich enough with good solutions that you should rarely need more than a few iterations to arrive at one. Still, that's a pretty expensive fitness test, and I wonder if there's a more elegant and efficient approach than Monte Carlo methods.
- kordless 14y agoThanks for this. I just spent 30 minutes reading up on calculating pi with a simple Monte Carlo algo. Thinking about doing it d3.js...
- benatkin 14y agoWhy are they using bogus examples?
- mark-r 14y agoIt's probably Lorem Ipsum, which is commonly used when you want to focus on layout rather than meaning. Since the text is meaningless it won't be distracting, yet it looks close enough to real to be useful. See http://en.wikipedia.org/wiki/Lorem_ipsum http://en.wikipedia.org/wiki/Lorem_ipsum
- tekromancr 14y agoI love the fact that all three answers used totally different techniques and each found the answer.
- Zolomon 14y agoI've always wondered if there exist any book that uses rivers as an encryption technique for hidden messages. Like the author has hidden an easter egg or something more exciting.
- taltman1 14y agoTeX uses a dynamic programming algorithm to perform its advanced hyphenation, which allows the text to fill the page "beautifully": http://en.wikipedia.org/wiki/TeX#Hyphenation_and_justification http://en.wikipedia.org/wiki/TeX#Hyphenation_and_justificati... If TeX is already using dynamic programming in order to improve the visual appearance of the words, I imagine that the same can be done for the space between the words without having to resort to image processing of the TeX document rendered as PDF.
- limmeau 14y agoThe thread at SO discusses this; both glyph positions and glyph shapes are important for a river to become noticeable.
- GhotiFish 14y agoI think a false positive or two is worth it for glyph based analysis. You could even do some sensible analysis per glyph. width of base: A vs width of top: V
- limmeau 14y agoIs glyph width at base and top available to the layout algorithm? I don't know TeX that well.
- taltman1 14y agoWhether vanilla TeX supports this out of the box or not, the TeX technology stack (including MetaFont and friends) has intimate knowledge of the size and shape of all glyphs. It is also not random as to which types of letters accentuate or obfuscate the border of a river in text. It is hard for me to believe that one cannot use dynamic programming to minimize the occurrence of rivers in the final layout, in the same way that TeX can minimize ugly borders using sophisticated hyphenation algorithms.
- Kiro 14y agoI want to know if it can be done on HTML texts with JavaScript.
- lifeisstillgood 14y agoI was thinking about developer continuing education yesterday, and this underlines the need for some way of keeping sharp - without wasting time or direction. I simply don't know what a Hough Transform is and rather than Wikipedia, I would rather a coursera course on image processing - it's to get things in context that matters. As I get older i am not afraid of hard work, just afraid of exploring in every direction - the waste of time and effort is the problem, time and effort finding out what to learn rather than the practise of learning Oh, just answered my own question - coursera!
- martinced 14y agoIf there are any typesetting geek on HN, any idea when this would be incorporated in TeX / LaTeX and other typesetting programs? Oh the memories when I used to write (and typeset) books using LaTeX and Quark XPress: "rivers" had to be tracked down manually, by eye-ball searching. You'd basically "blur" your vision a bit and quickly scan through all the pages of the book. I didn't take that long and I don't write books anymore but I'd still be curious as to when that technique is going to be implemented (maybe in InDesign which I never used!?).
- Samuel_Michon 14y agoIn Adobe InDesign, one chooses to set type using the Single-line Composer or the Paragraph Composer. The latter analyzes the entire paragraph and tweaks spacing and hyphenation to minimize rivers, ragged lines, orphans and widows. So nowadays, I don't manually check each page for rivers anymore. When designing the page layout, I decide on the 'color' (the density of the paragraph) that works best, so that rivers are avoided, but I leave it at that. I understand how river detection can be a fun mathematical puzzle, but it's moot. Adobe built the Multi-line Composer into InDesign since version 1.0 (1999). And in my experience, with every version, it gets better. From MacWorld's review of InDesign 1.0: "InDesign's text features will appeal to designers and production people tired of the drudgery of manual copyfitting and kerning. The Multi-line Composer feature can calculate hyphenation and justification settings by examining an entire paragraph (or as many lines as you specify), instead of just a single line, to create better-looking text. In the process, it notably reduces the amount of manual tweaking necessary and will be especially helpful if you're trying to avoid multiple word breaks in a design with awkward text wraps. Similarly, the program's Optical Kerning feature does its best to find optimal character spacing, even if you've mixed different type sizes and faces. " http://www.macworld.com/article/1014955/k2.html http://www.macworld.com/article/1014955/k2.html http://blog.paragonpress.net/2010/08/04/adobe-indesigns-paragraph-composer/ http://blog.paragonpress.net/2010/08/04/adobe-indesigns-para...
- arrrg 14y agoWoah. InDesign’s Paragraph Composer detects rivers? I didn’t know that. Now I wonder how it’s doing that.
- oofabz 14y agoIt's really impressive how their sophisticated image processing algorithms are only a few lines long. If I wrote an algorithm to detect rivers, it would be hundreds of lines long and less effective. These guys tackle it mathematically and it seems almost magical. I wish I could think that way.
- maurits 14y agoThere is no panacea in image processing. These methods work for this particular image, and give an idea about the possible solution, but the thresholding and morphological operations are not scale invariant, and very sensitive to light changes and noise. Any algorithm that can detect rivers in text from any image, acquired by some device, such as mobile phone or a scanner, in any light circumstances, will run hundreds of lines long.
- deleted 14y ago[deleted]
- phil 14y agoSure, but the problem here is "we'd like to detect rivers in TeX, but it doesn't know about glyph shape which we know is important." They're talking about clean images only.
- Chirono 14y agoI find this comment interesting given the recent (ok, continuous) talk about not using maths puzzles in programmer interviews. This is perhaps an interesting real-world case that demonstrates how a mathematical outlook leads to much cleaner code. Ok, solving this kind of thing is rare, but still...
- thomasz 14y agoThe secret sauce here is not the ability to solve "maths puzzles", but specialized domain knowledge.
- 14y ago
- qscesz 14y agoReally impressed by the short length and high effectiveness of the code. Thanks!
- fnordfnordfnord 14y agoI'd like to have the tex, to translate the letters to blocks before doing image processing, but it's almost as good to simply convert it to a binary mask, and then "dilate" or "grow" the pixels. I think it makes it a simpler problem both in processing time and conceptually. Not sure about how I'd decide, identify what vertical lines to pick out, there are a lot of choices for that.
- Bryan22 14y agoThis seems to me, like the least efficient method ever for dealing with the problem. Ok you want to find 'rivers' in text. To do so you have to turn that text into an image and then run it through your app, then go back and manually correct the problem? What was the point of the app again? I don't care if it took 1 line of code or 1000, this seems completely useless, when the anomaly is blatantly obvious when proofreading. Not to mention if you wanted to fix it dynamically, you'd end up with an image of a block of text instead of text, no big deal for print, but devastating for SEO.. "ok smart ass what would you recommend?" is that what your thinking? Well, since you asked; why not take an open source text editor and add an algorithm that stores each line of text in an array, find the index of all the spaces and compare it to the previous and next lines of text. If the index of spaces are relatively 0 between lines; you have a river. If the indexes increment or decrement; you have a river. Now add an extra space somewhere or move the last word on the last line to the next line. Whatever solution is most aesthetically pleasing. Now your rivers are getting fixed on the fly and you don't need to take a screenshot of a block of text to analyze it. Maybe I missed the point of the article, but i thing those of you praising their solution aren't taking into account what the problem actually is, and the fact that their app doesn't actually include a solution to the problem.
- pdeuchler 14y agoI love the variance in answers. You get several different sig-pro solutions and then at the bottom also a machine learning answer. IMHO the machine learning solution could be better, as rivers are defined more aesthetically than functionally.