6 ms·

Boring Problems Need Attention Too

57 points by simplegeek 5y ago

PaulHoule 5y ago
There is no such thing as a "word". Look at the waveform of speech and you will see long silent gaps inside "words" as well as there frequently being no gap between "words". There are phrases like "Skinny Puppy" that can do the same job as a word, there are also structures smaller than words that people smush together to make words. The two even work together: missile anti-missile missile anti-anti-missile missile missile If you see "words" as the molecules of text there will always be an asymptote you can't overcome because segmenting text into words will sometimes introduce errors that you might not be able to recover from.
- belval 5y ago
  One step further, I was doing text processing and my very western education taught me that characters were the smallest relevant construct when dealing with text. Except that's simply not true for most eastern languages. Devanagari, Arabic, Bengla, Thai and so many other languages are packed with diacritics and graphemes, conditional rendering based on surrounding letters and much more that force you to completely rethink your approach.
  - whatshisface 5y ago
    Cursive writing in English has conditional rendering.
- lucasmullens 5y ago
  I don't understand this. The article showed 4 text processors have a different definition of "word". How does that mean "there is no such thing"? Just because no one agrees on a definition for "word" doesn't mean words don't exist.
  - mgbennet 5y ago
    GP's point is more about how words are spoken. When we hear speech in a language we understand, we parse the sounds as a collection of distinct units with individual meaning we call "words". But the sound wave itself isn't actually segmented in this fashion, and the separation into words occurs in our perception and interpretation. But I agree, words definitely exist, just as interpretations of sound and written characters.
    - zorked 5y ago
      I don´t think it has anything to do with our interpretation: the illiterate have no concept of words. They come from writing systems and are not a part of languages. German has very long "words" just because they decided not to add lots of spaces when creating a writing system for it. There is no particular reason why it's "Bundesverfassungsgerich" instead of "Bundes verfassungs gericht".
LarryMade2 5y ago
So they just leave us hanging there, no analysis, no reasoning of the results...?
- magneticnorth 5y ago
  Boring problems need to be solved too, but not by the author, apparently.
- deleted 5y ago
  [deleted]
- yellowriver9 5y ago
  Discovery of a problem is also not useless. It is also possible that the author is working on it (his Github profile suggests he open-sourced some work in NLP).
thehappypm 5y ago
I mean.. are numbers words? Is a floating ! or - a word? Are hyphenated words 2 or 1? How about when a word is broken over two lines with a hyphen? Is ... a word? Is an unrecognized string a word? Does a figure caption count towards a word count? How about chapter titles? Or page numbers?
- Rioghasarig 5y ago
  Yeah the issue seems to be the concept of a "word" not being precisely defined. But I don't think that's a problem. I'd be concerned if the numbers were dramatically different but they're pretty close so there isn't an issue. If exact precision is necessary you probably shouldn't rely on imprecise terms like "word".
  - vinnymac 5y ago
    Agreed. If you are going to base a salary on imprecise concepts, then the pay will be imprecise. Concepts from the real world don't always fit nicely with the strict rules that we attempt to program them in. Trying to solve this problem by changing the constraints may prove to be easier than programming a precise solution around an imprecise concept.
version_five 5y ago
What this highlights most to me is the false precision people have come to expect. This is a real problem I've seen as an engineer / physicist working with business people. Things that I would consider "the same" because they are well withing the noise of one another get all kinds of "hold on now, I added up the numbers in the columns and got 22.1 but the columns total says 22" from people that don't understand rounding, significant figures, experimental error, etc. I know it's important to hide this from people because they end up getting distracted by it (or to occasionally let slip so someone can feel like they have done a thorough review because they found a column didnt add up to the rounded total). But I think it's more important overall to educate people about what kind of variation is important and what isn't, vs. getting hung up on the definition of a word.
- 6gvONxR4sf7o 5y ago
  I have a ton of experience with that too. You have two people try to estimate a thing, and they make different choices and come up with answers that are totally consistent with each other, but aren't the same. Then someone sees that they're different and omg omg omg everything's on fire. It's amazingly hard to communicate "variability" in an answer. Whenever I communicate experimental results, my business partners always want to hone in on the simplest explanation (e.g. "treatment effect somewhere between 1 and 3" becomes "treatment effect = 2"). And that's just the uncertainty part of variability. Then there's the kinds of variability you're talking about. There are a ton of tricks to mitigating this, like reporting confidence intervals instead of point estimates, or giving the right number of digits to communicate the key message, or making super sharable/linkable results to minimize the game of telephone. I'd love someone to write a book on how to communicate this sort of "variability" in a business setting.
  - weaksauce 5y ago
    > like reporting confidence intervals Then there is the problem of how insanely innumerate society at large is to the point where something like this doesn't register for huge swaths of people. might work in the context of engineers or people with some higher education but not all contexts.
lbriner 5y ago
Also boring businesses need attention. Loads of people would love to run a gym but you could make 10x more by selling toilet paper or paper clips.
- Trufa 5y ago
  I'm always looking for boring ideas, I'm a full stack web dev, and honestly my dream is to have some running side business that's extremely boring, I never seem to find good ol' boring ideas. Open to ideas from HN too :)
  - maxwell 5y ago
    Property and casualty insurance. Work with the brokers (and reinsurers if possible) and go after the carriers. Follow the Zillow model instead of Zenefits. Edit, more: Businesses want to work with established P&C brokers, and brokers have ensured (in most states) that they have to anyway. Provide the right data-driven SaaS tool that gives brokers the power to sell/renew better. Offer it for free unless it converts but require reporting conversions, then charge a percentage. You could call up brokers and they would all try it out. Then focus on solving boring, mostly statistical, problems in an elegant way behind the scenes.
  - wilde 5y ago
    The classic advice is to talk to people. Ask your friends in other industries what the most bullshit part of their job is. The most frustrated friend will probably have something. Especially if their answer is “eh, I have this Excel sheet that kinda does the job”.
- jjk166 5y ago
  I used to work for a machine shop that made lots of stuff - parts for cars, aircraft, medical devices, all very exciting. But the real money maker? Little clamps for holding sections of drill shafts together. Oil extraction companies go through literal tons of them every day and don't give a damn about cost. It was quite an eye opening experience.
- Akronymus 5y ago
6gvONxR4sf7o 5y ago
I'm sad that the author didn't do a manual word count. If it's only 1660 words or so, it would surely take less time than it took to write this post. It would probably make the author confront the difficulty in precisely defining word count. It turns out that a lot of things that initially seem trivial to precisely define aren't actually that precisely defined, like the length of the california coastline. This is, in my mind--and as a complete tangent--a great argument for wide programming and math education. When you're forced to be so goddamned precise all the time, it's very clear when an idea isn't fully defined.
AbrahamParangi 5y ago
The words of someone who's never tried to write a parser that operates on natural language. "I've seen things, you people wouldn't believe"
asciimov 5y ago
I'd rather see an explanation behind the issue rather than a complaint about people not working on boring stuff because a bunch of text editors can't agree on a word count. I remember taking a typing class some 25 years ago and being told a that a word count is typically every 5 characters. That way someone doesn't pad out their word count by using lots of small words.
- Pet_Ant 5y ago
  > typing class ... word count is typically every 5 characters. That might be only within the context of "words per minute" in typing. I mean otherwise Finnish typists would be terribly slow.
Pet_Ant 5y ago
I remember during my PhD, a casual discussion on that figuring out the end of a sentence on it's own is surprisingly more complex then you'd think: "I work at the F.B.I. I like it there." "I work at the F.B.I., I like it there." "I work at the F.B.I. I like it there!" It's not as simple as counting periods. So that counting words can have corner cases is definitely understandable. Is "&" a word? It is literally just 'e' + 't' superimposed and "et" is definitely a word.
jakub_g 5y ago
IMO the issue here is that: everything that is not carefully specified as a precise algorithm, will be implemented differently. This is why each browser used to parse HTML differently. This is why you'd have compat or even security issues because some software used \r\n for newlines splitting while other used \n. Luckily the browser vendors formed WHATWG which created pretty precise specs which are maybe convoluted but at least everyone parses HTML in the same way, and each browser pretends to be every other browser for compatibility. 2021 is really great for web compat, maybe not all browsers implement every API, but existing APIs are accompanied by very thorough test suites (Web Platform Tests): https://github.com/web-platform-tests/wpt https://github.com/web-platform-tests/wpt Live results from nightly builds: https://wpt.fyi/results/?label=experimental&label=master&aligned https://wpt.fyi/results/?label=experimental&label=master&ali... Having said that I don't see vendors aligning on definition on word count any soon due to corporate inertia, lack of incentives and lack of "Word editors consortium" (or is there any?) A truly good function for word count might be pretty complex, and perhaps different for every language.
sgtnoodle 5y ago
At some FAANG companies, fixing bugs and other "boring" projects don't get you promoted. I can tell when there's a perf/promo cycle coming up at Google because all the core apps on my phone change their UIs and get buggier and slower.
bilater 5y ago
The variance is probably due to a difference in what counts as a word + ambiguity around white spaces etc in each text editor. So it's not that the word counting problem is unsolved it's a problem of different companies adopting different definitions.
SethTro 5y ago
I diffed down to understand why Open Office (1663 consistent with Microsoft Office) differs from Google Docs (1655). "ago--never": G docs 1 word vs OpenOffice 2 words. "tiger-lillies--what": G docs 1 word vs OpenOffice 2 words (IDK what this should "really" be) "Wanting?--Water": GD 3 words vs OO 2 words In this case the disagreement springs, exclusively, from if the docs engine believes that double hyphens make a compound word (and potentially handling punctuation in the middle of such a compound word)
- jkingsbery 5y ago
  This is an excellent answer. The original post, insinuating that engineers are demonstrating "engineering mediocrity" or not "correct," is lazy for not understanding why this problem is ill-defined.
  - horsawlarway 5y ago
    Absolutely. The irony is that his title is correct - Boring problems need attention. But the content is jarringly wrong - The lack of attention didn't lie with the engineers of these companies, but with the author themselves. For ~1600 words, they could have very easily tried to count it and quickly realized that the answers they got weren't wrong - the question was lazy and lacking attention. Do you mean dictionary words? Compound words? Whitespace separated tokens? What about stand alone text that isn't a word (ex - page numbers? Title numbers?) Words that the author invented? Words use non-traditional-spacing or grammar?(and maybe a missing space) ---- But nope, it's those damn lazy engineers!
- randompwd 5y ago
  > "tiger-lillies--what": G docs 1 word vs OpenOffice 2 words (IDK what this should "really" be) i think 3 words. The tiger lily is a type of lily. plural is tiger lillies. Fair to say hypen is stylistic (& after a quick web search , I can't find any other usage like that). That aside, I can't believe how simplistic Google Docs word recognition/count engine is.
juancn 5y ago
Counting words is easy, defining what a word is, is not.
dcolkitt 5y ago
I think a complement to this article is the essay "Reality has a Surprising Amount of Detail"[1]. The point is that things which superficially seem easy often have a lot of hidden complexity when you drill down to all the low-level details and corner cases. [1]http://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...
Zababa 5y ago
There's a 2% variation between the lowest and the biggest. Is there a need to know a precise word count, where 2% of margin of error is not acceptable? If there's not, there's no problem. The author doesn't attempt to define what a word is, so maybe all of these are correct for their own definition of word? Maybe the definition of what counts as a word is the boring problem that needs a solution, but it doesn't look boring or easy to me. You can't ask people to count correctly something that you haven't even defined. Also, believing that complex problems are easy is not something only programmers do. I work currently a lot with Excel automation, and most people have no idea of what can be automated easily and what can't. I have some people coming that ask for automating a task they've never done manually and don't really know how to do precisely. I think that's the same mechanism of "overabstraction" that leads to people to say "WET instead of DRY" (Write Everything Twice instead of Don't Repeat Yourself).
spoonjim 5y ago
The "boring problem" that always needs more attention than it gets is performance. Plenty of companies ship features that only reduce performance by 10,20,50 milliseconds but it all adds up and applications end up feeling very sluggish. Apple is one of the few where I feel like the perf doesn't regress over time (likely because the features don't really expand much either).
deleted 5y ago
[deleted]

57 points by simplegeek 5y ago

There is no such thing as a "word". Look at the waveform of speech and you will see long silent gaps inside "words" as well as there frequently being no gap between "words". There are phrases like "Skinny Puppy" that can do the same job as a word, there are also structures smaller than words that people smush together to make words. The two even work together: missile anti-missile missile anti-anti-missile missile missile If you see "words" as the molecules of text there will always be an asymptote you can't overcome because segmenting text into words will sometimes introduce errors that you might not be able to recover from.

belval 5y ago
One step further, I was doing text processing and my very western education taught me that characters were the smallest relevant construct when dealing with text. Except that's simply not true for most eastern languages. Devanagari, Arabic, Bengla, Thai and so many other languages are packed with diacritics and graphemes, conditional rendering based on surrounding letters and much more that force you to completely rethink your approach.
- whatshisface 5y ago
  Cursive writing in English has conditional rendering.
lucasmullens 5y ago
I don't understand this. The article showed 4 text processors have a different definition of "word". How does that mean "there is no such thing"? Just because no one agrees on a definition for "word" doesn't mean words don't exist.
- mgbennet 5y ago
  GP's point is more about how words are spoken. When we hear speech in a language we understand, we parse the sounds as a collection of distinct units with individual meaning we call "words". But the sound wave itself isn't actually segmented in this fashion, and the separation into words occurs in our perception and interpretation. But I agree, words definitely exist, just as interpretations of sound and written characters.
  - zorked 5y ago
    I don´t think it has anything to do with our interpretation: the illiterate have no concept of words. They come from writing systems and are not a part of languages. German has very long "words" just because they decided not to add lots of spaces when creating a writing system for it. There is no particular reason why it's "Bundesverfassungsgerich" instead of "Bundes verfassungs gericht".

LarryMade2 5y ago

So they just leave us hanging there, no analysis, no reasoning of the results...?

magneticnorth 5y ago
Boring problems need to be solved too, but not by the author, apparently.
deleted 5y ago
[deleted]
yellowriver9 5y ago
Discovery of a problem is also not useless. It is also possible that the author is working on it (his Github profile suggests he open-sourced some work in NLP).

thehappypm 5y ago

I mean.. are numbers words? Is a floating ! or - a word? Are hyphenated words 2 or 1? How about when a word is broken over two lines with a hyphen? Is ... a word? Is an unrecognized string a word? Does a figure caption count towards a word count? How about chapter titles? Or page numbers?

Rioghasarig 5y ago
Yeah the issue seems to be the concept of a "word" not being precisely defined. But I don't think that's a problem. I'd be concerned if the numbers were dramatically different but they're pretty close so there isn't an issue. If exact precision is necessary you probably shouldn't rely on imprecise terms like "word".
- vinnymac 5y ago
  Agreed. If you are going to base a salary on imprecise concepts, then the pay will be imprecise. Concepts from the real world don't always fit nicely with the strict rules that we attempt to program them in. Trying to solve this problem by changing the constraints may prove to be easier than programming a precise solution around an imprecise concept.

version_five 5y ago

What this highlights most to me is the false precision people have come to expect. This is a real problem I've seen as an engineer / physicist working with business people. Things that I would consider "the same" because they are well withing the noise of one another get all kinds of "hold on now, I added up the numbers in the columns and got 22.1 but the columns total says 22" from people that don't understand rounding, significant figures, experimental error, etc. I know it's important to hide this from people because they end up getting distracted by it (or to occasionally let slip so someone can feel like they have done a thorough review because they found a column didnt add up to the rounded total). But I think it's more important overall to educate people about what kind of variation is important and what isn't, vs. getting hung up on the definition of a word.

6gvONxR4sf7o 5y ago
I have a ton of experience with that too. You have two people try to estimate a thing, and they make different choices and come up with answers that are totally consistent with each other, but aren't the same. Then someone sees that they're different and omg omg omg everything's on fire. It's amazingly hard to communicate "variability" in an answer. Whenever I communicate experimental results, my business partners always want to hone in on the simplest explanation (e.g. "treatment effect somewhere between 1 and 3" becomes "treatment effect = 2"). And that's just the uncertainty part of variability. Then there's the kinds of variability you're talking about. There are a ton of tricks to mitigating this, like reporting confidence intervals instead of point estimates, or giving the right number of digits to communicate the key message, or making super sharable/linkable results to minimize the game of telephone. I'd love someone to write a book on how to communicate this sort of "variability" in a business setting.
- weaksauce 5y ago
  > like reporting confidence intervals Then there is the problem of how insanely innumerate society at large is to the point where something like this doesn't register for huge swaths of people. might work in the context of engineers or people with some higher education but not all contexts.

lbriner 5y ago

Also boring businesses need attention. Loads of people would love to run a gym but you could make 10x more by selling toilet paper or paper clips.

Trufa 5y ago
I'm always looking for boring ideas, I'm a full stack web dev, and honestly my dream is to have some running side business that's extremely boring, I never seem to find good ol' boring ideas. Open to ideas from HN too :)
- maxwell 5y ago
  Property and casualty insurance. Work with the brokers (and reinsurers if possible) and go after the carriers. Follow the Zillow model instead of Zenefits. Edit, more: Businesses want to work with established P&C brokers, and brokers have ensured (in most states) that they have to anyway. Provide the right data-driven SaaS tool that gives brokers the power to sell/renew better. Offer it for free unless it converts but require reporting conversions, then charge a percentage. You could call up brokers and they would all try it out. Then focus on solving boring, mostly statistical, problems in an elegant way behind the scenes.
- wilde 5y ago
  The classic advice is to talk to people. Ask your friends in other industries what the most bullshit part of their job is. The most frustrated friend will probably have something. Especially if their answer is “eh, I have this Excel sheet that kinda does the job”.
jjk166 5y ago
I used to work for a machine shop that made lots of stuff - parts for cars, aircraft, medical devices, all very exciting. But the real money maker? Little clamps for holding sections of drill shafts together. Oil extraction companies go through literal tons of them every day and don't give a damn about cost. It was quite an eye opening experience.
Akronymus 5y ago

6gvONxR4sf7o 5y ago

I'm sad that the author didn't do a manual word count. If it's only 1660 words or so, it would surely take less time than it took to write this post. It would probably make the author confront the difficulty in precisely defining word count. It turns out that a lot of things that initially seem trivial to precisely define aren't actually that precisely defined, like the length of the california coastline. This is, in my mind--and as a complete tangent--a great argument for wide programming and math education. When you're forced to be so goddamned precise all the time, it's very clear when an idea isn't fully defined.

AbrahamParangi 5y ago

The words of someone who's never tried to write a parser that operates on natural language. "I've seen things, you people wouldn't believe"

asciimov 5y ago

I'd rather see an explanation behind the issue rather than a complaint about people not working on boring stuff because a bunch of text editors can't agree on a word count. I remember taking a typing class some 25 years ago and being told a that a word count is typically every 5 characters. That way someone doesn't pad out their word count by using lots of small words.

Pet_Ant 5y ago
> typing class ... word count is typically every 5 characters. That might be only within the context of "words per minute" in typing. I mean otherwise Finnish typists would be terribly slow.

Pet_Ant 5y ago

I remember during my PhD, a casual discussion on that figuring out the end of a sentence on it's own is surprisingly more complex then you'd think: "I work at the F.B.I. I like it there." "I work at the F.B.I., I like it there." "I work at the F.B.I. I like it there!" It's not as simple as counting periods. So that counting words can have corner cases is definitely understandable. Is "&" a word? It is literally just 'e' + 't' superimposed and "et" is definitely a word.

jakub_g 5y ago

IMO the issue here is that: everything that is not carefully specified as a precise algorithm, will be implemented differently. This is why each browser used to parse HTML differently. This is why you'd have compat or even security issues because some software used \r\n for newlines splitting while other used \n. Luckily the browser vendors formed WHATWG which created pretty precise specs which are maybe convoluted but at least everyone parses HTML in the same way, and each browser pretends to be every other browser for compatibility. 2021 is really great for web compat, maybe not all browsers implement every API, but existing APIs are accompanied by very thorough test suites (Web Platform Tests): https://github.com/web-platform-tests/wpt https://github.com/web-platform-tests/wpt Live results from nightly builds: https://wpt.fyi/results/?label=experimental&label=master&aligned https://wpt.fyi/results/?label=experimental&label=master&ali... Having said that I don't see vendors aligning on definition on word count any soon due to corporate inertia, lack of incentives and lack of "Word editors consortium" (or is there any?) A truly good function for word count might be pretty complex, and perhaps different for every language.

sgtnoodle 5y ago

At some FAANG companies, fixing bugs and other "boring" projects don't get you promoted. I can tell when there's a perf/promo cycle coming up at Google because all the core apps on my phone change their UIs and get buggier and slower.

bilater 5y ago

The variance is probably due to a difference in what counts as a word + ambiguity around white spaces etc in each text editor. So it's not that the word counting problem is unsolved it's a problem of different companies adopting different definitions.

SethTro 5y ago

I diffed down to understand why Open Office (1663 consistent with Microsoft Office) differs from Google Docs (1655). "ago--never": G docs 1 word vs OpenOffice 2 words. "tiger-lillies--what": G docs 1 word vs OpenOffice 2 words (IDK what this should "really" be) "Wanting?--Water": GD 3 words vs OO 2 words In this case the disagreement springs, exclusively, from if the docs engine believes that double hyphens make a compound word (and potentially handling punctuation in the middle of such a compound word)

jkingsbery 5y ago
This is an excellent answer. The original post, insinuating that engineers are demonstrating "engineering mediocrity" or not "correct," is lazy for not understanding why this problem is ill-defined.
- horsawlarway 5y ago
  Absolutely. The irony is that his title is correct - Boring problems need attention. But the content is jarringly wrong - The lack of attention didn't lie with the engineers of these companies, but with the author themselves. For ~1600 words, they could have very easily tried to count it and quickly realized that the answers they got weren't wrong - the question was lazy and lacking attention. Do you mean dictionary words? Compound words? Whitespace separated tokens? What about stand alone text that isn't a word (ex - page numbers? Title numbers?) Words that the author invented? Words use non-traditional-spacing or grammar?(and maybe a missing space) ---- But nope, it's those damn lazy engineers!
randompwd 5y ago
> "tiger-lillies--what": G docs 1 word vs OpenOffice 2 words (IDK what this should "really" be) i think 3 words. The tiger lily is a type of lily. plural is tiger lillies. Fair to say hypen is stylistic (& after a quick web search , I can't find any other usage like that). That aside, I can't believe how simplistic Google Docs word recognition/count engine is.

juancn 5y ago

Counting words is easy, defining what a word is, is not.

dcolkitt 5y ago

I think a complement to this article is the essay "Reality has a Surprising Amount of Detail"[1]. The point is that things which superficially seem easy often have a lot of hidden complexity when you drill down to all the low-level details and corner cases. [1]http://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...

Zababa 5y ago

There's a 2% variation between the lowest and the biggest. Is there a need to know a precise word count, where 2% of margin of error is not acceptable? If there's not, there's no problem. The author doesn't attempt to define what a word is, so maybe all of these are correct for their own definition of word? Maybe the definition of what counts as a word is the boring problem that needs a solution, but it doesn't look boring or easy to me. You can't ask people to count correctly something that you haven't even defined. Also, believing that complex problems are easy is not something only programmers do. I work currently a lot with Excel automation, and most people have no idea of what can be automated easily and what can't. I have some people coming that ask for automating a task they've never done manually and don't really know how to do precisely. I think that's the same mechanism of "overabstraction" that leads to people to say "WET instead of DRY" (Write Everything Twice instead of Don't Repeat Yourself).

spoonjim 5y ago

The "boring problem" that always needs more attention than it gets is performance. Plenty of companies ship features that only reduce performance by 10,20,50 milliseconds but it all adds up and applications end up feeling very sluggish. Apple is one of the few where I feel like the perf doesn't regress over time (likely because the features don't really expand much either).

deleted 5y ago

[deleted]