15 ms·
What can’t the internet handle in 2022? Apostrophes
- leovander 4y agoCopy pasting code shared on slack, why do the apostrophe characters get converted into a different character that looks like an apostrophe?
- bombcar 4y agoOften it’s smart quotes or similar being stupid.
- thangalin 4y agoCurling smart quotes correctly is a deceptively hard problem. I implemented a basic natural language parser that does a decent job. However, straight quotes are ambiguous. Compounding the problem is that curled apostrophes and right single quotes are indistinguishable, which is an oversight in Unicode, IMO, because they are semantically different. Here's my swat at the puzzle: https://whitemagicsoftware.com/keenquotes/ https://whitemagicsoftware.com/keenquotes/ When exporting Markdown documents to HTML or PDF, my text editor uses KeenQuotes to curl straight quotes to their most likely candidate: https://github.com/DaveJarvis/keenwrite#download https://github.com/DaveJarvis/keenwrite#download
- GolDDranks 4y agoMakes it even harder that some languages use straigth quotes as the gold standard – so whether to curve them or not is language dependent.
- zokier 4y agoWhich languages prefer straight quotes? Wikipedia doesn't mention any: https://en.wikipedia.org/wiki/Quotation_mark https://en.wikipedia.org/wiki/Quotation_mark
- GoblinSlayer 4y agoLooks like hair splitting to me, they are all interchangeable. Apparently I used Dutch an Danish variants and nobody ever batted an eye. Cursive quotes look like just cursive version of straight quotes, you can't possibly differentiate them.
- bloak 4y agoQuotation marks arguably depend on the country more than on the language, so Switzerland quotes like «this» in French, German and Italian, while the Germans, at least, don't quote like that. It seems that Sweden and Finland use the same character for opening and closing quotes, like ”this”, which is similar to using straight quotes.
- GolDDranks 4y agoI was going to say "Finnish" (that's my native language), but to my surprise, after checking it, we use "”" after all. (We use that both as the opening and closing quotation mark.)
- jeroenhd 4y agoWhile I don't blame you or anything, this only works for English conventions. Dutch quotation marks are different, at least if you try to do smart quotes correctly. Expected output: „Goedemorgen,” zei hij. Returned output: “Goedemorgen,” zei hij. And that's just language to take care of! Smart quotes for computers are hell and there should be a library to do all this so everyone doesn't need to reinvent the wheel.
- P5fRxh5kUvp2th 4y agoThis was going to be my point (roughly). It's like when you travel and you start to judge gas stations by their bathroom. Anyone who has traveled a lot knows exactly what I mean. copy/paste worked more reliably 20 years ago than it does today. So much so that I've started judging software based upon their ability to reliably copy/paste. Seems stupid, right? Just like judging gas stations by their bathroom seems stupid to people who rarely go into them because they don't travel. But it flat out amazes me how bad copy/paste is today. I once spent _hours_ trying to figure out why a PHP script was shitting all over itself. It turns out I copy/pasted the email text from Monday.com and it copied non-printing characters that caused PHP to fall over. Thing is, at that point, I was an old hand at that stupid crap, it didn't take me that long to find it. I learned years ago not to trust copy/paste.
- bentcorner 4y agoIf you're on windows Ctrl+Shift+V is your friend.
- puyoxyz 4y agohttps://archive.ph/K4gV2 https://archive.ph/K4gV2
- colejohnson66 4y agoOnce again, programmers forget alphabets other than the Latin one exist. And by Latin, I mean the one containing only A through Z, with no “fancies”. Then there’s the issue of character encodings, which the article does a decent job explaining! Nitpick though: they claims it’s a UTF-8/ASCII thing, but it’s actually a UTF-8/Windows-1252 issue. In related news, even GNU coreutils fails to support UTF-8 properly despite a claim of support for multibyte character sets: https://catgirl.ai/log/cut-c-harmful/ https://catgirl.ai/log/cut-c-harmful/
- mc32 4y agoI want URLs that handle spaces. Like I could enter www.coca cola.com. [and not www.coca-cola.com. nor www.cocacola.com.) and it would work. "Simple space". Even the Latin/Roman alpha had space(s).
- teeray 4y agowww.google.com [a ton of spaces that HN won’t let me type] jknotgoogle.com
- mc32 4y agoI just want a single contiguous space character so I don't have to suppress the space or substitute with an alternate character --also URL boxes can make it conspicuously obvious for better UI. And while they're at it, allow & (ampersand).
- andrewxhonson 4y agoMaybe the location bar should accept the space but strip it? That would work for input if not reading the URL. Maybe only allowing single consecutive spaces to avoid the address spoofing example. I could go for that. I would love & natively though. For my wedding (bride&groom.com for example) I looked into faking it with punycode but nothing was close enough, and anything other than an ampersand would have been too hard to input.
- 4y ago
- nicbou 4y agoIs there a good dataset that can be used to test such things? I'm thinking of a database of all kinds of names, addresses and emails that could be found in the wild.
- colejohnson66 4y agoFor fuzzing, the Big List of Naughty Strings is a nice starting point: https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings
- myself248 4y agoI think we need a Big List of Nice Strings that your program must handle correctly. It would be forgiven for rejecting an evil string, but names with apostrophes must be handled.
- Beltalowda 4y agoI usually write a test with 'Россия&, '€& [shrug emoji in there dropped by HN idiocy], or the like as the input. If it handles that correct, then it'll probably be fine for everything else too.
- user_named 4y ago
- evronm 4y agoSorry, mate, usage defines language, and there are many more of us than there are of you.
- deleted 4y ago[deleted]
- galangalalgol 4y agoWait what? It is normal now to use an apostrophe for plurals? That just seems like extra work.
- beowulfey 4y agoThank you for your suggestion.
- toast0 4y agoA) sounds like you can't handle them. B) Style norms change over time. Using an apostrophe in that case used to be the common style, now it's the uncommon style. It's likely to disappear over time, like the periods that used to be present to let us know ETF was an acronym.
- 4y ago
- nayuki 4y agoIf you don't trust developers to use prepared statements, banning input strings with apostrophes can be a blunt way to avoid SQL injection attacks. Robert'); DROP TABLE Students;-- https://xkcd.com/327/ https://xkcd.com/327/
- vincent-manis 4y agoAh, the old “make things easy for the computer” philosophy, much less work than diligently sanitizing input.
- LinAGKar 4y agoBut it's not easy for the computer, since it needs to recompile the statement every single time. It's just easy for the programmer, and even that is questionable.
- andreareina 4y agoNot trusting your developers to use prepared statements is a big problem beyond the sqli
- yosito 4y agoA big problem that many, many organizations have, unfortunately.
- P5fRxh5kUvp2th 4y agoI _just_ rejected a PR yesterday from a contractor that did the initial checkin of a new project with no .gitignore file and all the build artifacts (everything under obj and bin directories). Why in the world am I having to protect the project from such things? I understand ignorance, but the person putting together the base of the project should know what the hell is going on.
- andreareina 4y ago
- tapoxi 4y agoI have an Irish O' surname, and I've been to too many conferences where it's been changed to O%apos; on my nametag, and in once case a registration failed because there is a second unicode apostrophe that some text editors use which means my name won't match what's in their database. Not to mention many sites consider it an illegal character and drop the apostrophe all together. It's infuriating, I've seriously considered changing my name because of it.
- sacrosancty 4y agoImagine if even strongly typed languages had different string types for various encodings like that and enforced them. Then such bugs would be as rare as, say, a date appearing as a huge integer.
- rhn_mk1 4y agoWhen Python did that to separate raw bytes and proper Unicode, people pointed this out as a reason they don't want to use the language any more.
- sacrosancty 4y agoSounds like Python wasn't right for those people in the first place. Abstracting away the internal data format is typical of Python.
- Thorrez 4y agohttps://angular.io/api/platform-browser/SafeHtml https://angular.io/api/platform-browser/SafeHtml https://google.github.io/closure-library/api/goog.html.SafeHtml.html https://google.github.io/closure-library/api/goog.html.SafeH...
- Cu3PO42 4y agoTo a degree, Rust does this. The standard string type is always guaranteed to be valid UTF-8. Then there's a string type for the OS native format and a string type for what constitutes a valid path on the file system. Of course, arrays of bytes also exist and so does a type for a null-terminated "CString". Converting between them requires runtime checks (unless you invoke unsafe code and promise you know better than the compiler). It's cumbersome at times, but a perfectly reasonable tradeoff in my opinion. Some libraries add other string types when they're needed to interact with a system. For example, the windows crate has wide strings.
- Foobar8568 4y agoGartner top lead quadrant solution cannot handle dot., minus - and the likes, fun stuff too
- Pinus 4y agoIt only takes a quick glance at sites like Stack Overflow to see that things like character encodings, the difference between the external representation of a string and its actual content, different quoting rules in different contexts (CSV versus XML versus JSON versus ...) is a complete mystery to many people trying to work as developers. In print, code is often represented in a monospace font, materials intended for beginners sometimes indicate stuff that you are actually supposed to type with little keycap symbols, etc. Have there been any programming languages (with associated IDE:s) where strings were represented by typography (or colour, or whatever) rather than by surrounding them with some sort of quotation marks?
- boltzmann-brain 4y agoThis is like if civil engineers were allowed to build bridges without any formal education. I'm not even talking about a full 4-year course, the majority of stupid moves in programming can be taught away in a course that's a few weeks of learning, especially once you've had a bunch of experience already.
- Nitrolo 4y agoAs someone who codes a fair bit at work without having a CS degree, is there any such course online you could recommend? Like a coursera course titled "How not to be dumb at coding"out something of the sorts.
- Karellen 4y agoThe set of search results for "falsehoods programmers believe about" could each be a single lesson in "How not to be dumb at coding 101" https://duckduckgo.com/?q=falsehoods+programmers+believe+about https://duckduckgo.com/?q=falsehoods+programmers+believe+abo...
- DangitBobby 4y agoThankfully most code isn't mission critical and bugs aren't life threatening. The things that are seem to be held to a higher standard.
- drooopy 4y ago* Cries in Portuguese *
- gdprrrr 4y agoOne Person once sued their Bank to correct their Name, under the GDPR right oft correction. The Bank initially refused, because they considered their legacy EDCDIC system unreasonable. The Bank lost the case. https://gdprhub.eu/index.php?title=Court_of_Appeal_of_Brussels_-_2019/AR/1006 https://gdprhub.eu/index.php?title=Court_of_Appeal_of_Brusse...
- lloydatkinson 4y agoThe title doesn’t do this justice. As it goes on to say it’s not just apostrophes it’s practically everything outside the incredibly limited and non-inclusive range of 26 “ascii” letters. It’s bad enough that apparently no one can spell my name correctly but when my partner and family with letters such as ö or ł can almost never type their correct name it irritates me greatly. A few months ago at work I had to actually kick off at product managers and QA that these letters are not “special characters” in the same way ? or & are. In the end I had to refuse to implement the regex in the form validation and just said that it was done and that was the end of it. So incredibly dumb that peoples names are classified as “special characters” by some arbitrary set of rules people subscribe to.
- fanf2 4y agoOne of the nice little details of email address syntax is that ' is not a special character, so niall.o'reilly@example.ie is just fine. One of the vexing things about MS Outlook and Exchange is that they insist on using a non-standard syntax in which ' is used as a quote, so they make an appalling mess of mail headers.
- dublin 4y agoWith very few exceptions (Gab, et al), the Internet cannot handle Free Speech in 2022. Especially speech that contravenes the official government narratives - that gets brutally shut down. This will not end well.
- dublin 4y agoOK, I'll whack the hornets' nest here: Base 10 Arithmetic should be the standard, with others being available only when wise programmers explicitly choose it because they have a good reason. Every tangle I've ever had with IEEE 754 would have been unnecessary if our computers and languages used the same numbers that humans do! (And we have way more than enough compute power now to make Base10 arithmetic the default - I'll cut the folks in the '60s some slack due to resource limits, but why are we still dealing with this brokenness in the 21st century?)
- xeonmc 4y agoI disagree: instead humans should drop base-10 entirely everywhere and use base-2 or base-16 instead.
- an1sotropy 4y ago"IEEE754" is ubiquitous because it was the first non-platform-specific solution to a fundamentally hard engineering problem (how to approximate the entire real number line with a finite number of bits), and because it is supported in hardware (on the CPU itself), so that the representation of the number fits within the same registers as used for pointers, integers, and everything else, which makes it fast. If you want base-10 encoding of non-integer values, and base-10 arithmetic on them, you're probably going to need a software library like [1], which implements the decimal math that is also in IEEE754, just not part of common hardware. It also may be that you actually want infinite-precision arithmetic, which definitely requires a library, because the representation of a number becomes its own dynamic data structure (like with a string). Can you say more about what exactly is the "brokenness" that you're running up against? [1] https://www.intel.com/content/www/us/en/developer/articles/tool/intel-decimal-floating-point-math-library.html https://www.intel.com/content/www/us/en/developer/articles/t...
- GoblinSlayer 4y agoIEEE 754 does define decimal types in addition to binary ones.
- bloak 4y agoIt annoys me that apostrophe and English closing single quotation mark are the same character in Unicode. I'm tempted to use ' for apostophe so that the quotation marks (‘...’) match up properly.
- zamadatix 4y agoThere are dedicated opening/closing quotation mark characters too, U+2018 and U+2019: ‘’
- kps 4y ago> It annoys me that apostrophe and English closing single quotation mark are the same character in Unicode. They're not. You should be more than just tempted to use ' (U+0027 APOSTROPHE) for apostrophe, because it is. Closing single quotation mark is ’ U+2019 RIGHT SINGLE QUOTATION MARK.
- eigengrau 4y ago> When text is set, U+2019 right single quotation mark is preferred as apostrophe, but only U+0027 is present on most keyboards. Software commonly offers a facility for automatically converting the U+0027 apostrophe to a contextually selected curly quotation glyph. https://www.unicode.org/versions/Unicode14.0.0/ch06.pdf https://www.unicode.org/versions/Unicode14.0.0/ch06.pdf Typographically, U+2019 is the appropriate glyph for quotation marks in English and in some other western languages.
- kps 4y agoThat doesn't reflect actual practice any more than the previous Unicode recommendation to use U+02BC MODIFIER LETTER APOSTROPHE (which at least had some semantic justification in that quote marks are not part of words). Yes, typographically an apostrophe looks much like a right single quote (although the position may be different); in the current century fonts handle that.
- eigengrau 4y ago
- NoGravitas 4y agoFurther evidence that O'Brien must suffer.
- vivegi 4y agoTyranny of the ASCII and EBCDIC.
- NautilusWave 4y agoThe US Federal government (and I imagine most/all states) only being able to handle names comprised of unaccented ASCII characters certainly doesn't help anything.