9 ms·
Drag an emoji family with a string size of 11 into an input with maxlength=10
- SquareWheel 4y ago> Except in Safari, whose maxlength implementation seems to treat all emoji as length 1. This means that the maxlength attribute is not fully interoperable between browsers. No, it's definitely not. You can read the byte length more directly in JS, and use that to inform if more text is allowed or not. const encoder = new TextEncoder(); const currentBytes = encoder.encode(inputStr).byteLength; But the maxlength attribute is at best an approximation. Don't rely on it for things like limiting length for database fields (not that you should trust the client anyway).
- RobotToaster 4y ago>Don't rely on it for things like limiting length for database fields (not that you should trust the client anyway). Do database engines all agree on edge cases like this?
- Manfred 4y agoNo, they don't even agree between engine implementations within the same database server. Generally limits in the database are defined as storage limits for however a character is defined. That usually means bytes or codepoints.
- jonhohle 4y agoOr act like MySQL and define a code point as fixed to 3-bytes when choosing UTF-8 .
- bonsaibilly 4y agoThankfully MySQL also offers a non-gimped version of UTF-8 that one should always use in preference to the 3-byte version, but yeah it sucks that it's not the "obvious" version of UTF-8.
- tragomaskhalos 4y agoIs this part of MySQL's policy of "do the thing I've always done, no matter how daft or broken that may be, unless I see an obscure setting telling me to do the new correct thing" ?
- bonsaibilly 4y agoThat'd be my guess, but I don't really know. They just left the "utf8" type as broken 3-byte gibbled UTF-8, and added the "utf8mb4" type and "utf8mb4_unicode_ci" collation for "no, actually, I want UTF-8 for real".
- jonhohle 4y agoIt will be a fun day when Unicode crosses the 5-byte UTF-8 encoding threshold :/
- otabdeveloper4 4y agoIt won't. We settled on using stateful combining characters instead. (Remember when the selling point of switching the world to Unicode was "represent all writing systems with a single stateless 16 bit encoding"? Yeah, well, lol.)
- bonsaibilly 4y agoAnything beyond four bytes is composed of multiple code points, happily
- WJW 4y agoNo the default these days is the saner utf8mb4, if you create a new database on a modern MySQL version. But if you have an old database using the old encoding then upgrading databases doesn't magically update the encoding because some people take backwards compatibility serious.
- bruce343434 4y agoto be fair, there's utfmb4
- ASalazarMX 4y agoApple's take seems more reasonable. When a user uses an emoji, they think of it as a single symbol, they don't care of the Unicode implementation, or its length in bytes. IMO this should be the standard, and all other interpretations are a repeat of the transition from ASCII to Unicode.
- Sharlin 4y agoIndeed Swift is one of the very few languages that even have a stdlib API for working with this intuitive definition of a "character" (the technical Unicode term is "(extended) grapheme cluster" and the division of text into EGCs is called Unicode text segmentation, described by [uax29] – spoiler: definitely not trivial). [uax29]: https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/
- alwaysbeconsing 4y agoThere's no Swift involved here; this is in WebKit.
- alpaca128 4y agoAgreed, I'm surprised the linked OP filed a bug for Webkit when I'd say it's the only correct implementation. HTML is for UIs and I don't think many users would expect an emoji to be counted as 11 symbols when it looks like just one. If this wasn't an emoji but a symbol from a more complex writing system splitting that up would clearly be unacceptable, so I don't see why one should make an exception here just because this one known example of a family emoji still mostly survives the truncation.
- scatters 4y agoThe number of graphemes an emoji displays as depends on the platform and font. How many "characters" does Safari think Ninja Cat[1] is? It displays as a single grapheme on Windows 10. 1. https://emojipedia.org/ninja-cat/ https://emojipedia.org/ninja-cat/
- ASalazarMX 4y ago
- hn_throwaway_99 4y agoLol, sorry kid, we can't afford the minivan, you've got to go.
- graypegg 4y agoOdd, it makes sense on a technical level when it comes to these ZWJ characters, but hiding the implementation makes sense from Safari’s point of view. I’d actually prefer that as a UNIVERSAL standard, visible symbols vs characters. (When it comes to UI) But I can also imagine this is problematic when setting validation rules else where and now there’s a subtle foot gun buried in most web forms. I guess the thing to learn here is to not rely on maxlength=
- zuhsetaqi 4y agoThe thing every developer should have learned from the beginning is to never rely on anything that comes from user input or client
- turmeric_root 4y agoagreed, i discard user input in all of my apps
- graypegg 4y agoI hope no one is relying on it! But in the context of a regular user, maxlength measuring all chars is failsafe (user is allowed to submit a long string that should be caught by validation elsewhere) vs measuring visible chars is faildangerous (user can’t submit a valid string because the front end validation stops them) Which is weird
- Akronymus 4y agoAt work we almost exclusively use backend validation and return a json object with the fields that failed validation + what rules were failed.
- Wowfunhappy 4y agoIf anyone would like to try this out: https://codepen.io/Wowfunhappy/pen/dyqpMXO?editors=1000 https://codepen.io/Wowfunhappy/pen/dyqpMXO?editors=1000
- waltbosz 4y agoI forked your code and made it a bit more fun. Presenting: the Magical Shrinking Emoji Family. https://codepen.io/waltbosz/pen/zYJKBEE https://codepen.io/waltbosz/pen/zYJKBEE
- brycedriesenga 4y agoThe father seems so much happier to be alone.
- rom-antics 4y agoEdge cases like this will just get more common as unicode keeps getting more complex. There was a fun slide in this talk[1] that suggests unicode might be turing complete due to its case folding rules. I miss when Unicode was just a simple list of codepoints. (Get off my lawn) [1]: https://seriot.ch/resources/talks_papers/20171027_brainfuck_dominos.pdf https://seriot.ch/resources/talks_papers/20171027_brainfuck_...
- GuB-42 4y agoI think we need some kind of standard "Unicode-light" with limitations to allow it to be used on low specs hardware and without weird edge cases like this. A bit like video codecs that have "profiles" which are limitations you can adhere to to avoid overwhelming low end hardware. It wouldn't be "universal", but enough to write in the most commonly used languages, and maybe support a few, single codepoint special characters and emoji.
- themerone 4y agoYou can't express the most commonly used languages without multiple code point graphemes. If you want to eliminate edge cases you would need to introduce new incompatible code points and a 32 or 64 bit fixed length encoding depending on how many languages you want to support.
- GuB-42 4y agoExtra pages for these extra code points wouldn't seem far fetched to me. We already have single code point letters with diacritics like "é", or a huge code page of Hangul, for which each code point is a combination of characters. As for encoding, 32 bits fixed length should be sufficient, I can't believe that we would need billions of symbols, combinations included, in order to write in most common languages, though I may be wrong. Also, "limiting" doesn't not necessarily means "single code point only", but more like "only one diacritic, only from this list, and only for these characters", so that combination fits in a certain size limit (ex: a 32 bit word), and that the engine only has to process a limited number of use cases.
- Manfred 4y agoAny attempt at defining what people think of as characters is going to fail because of how many exceptions our combined writing systems have. See: codepoints, characters, grapheme clusters. A good starting place is UAX #29: https://www.unicode.org/reports/tr29/tr29-41.html https://www.unicode.org/reports/tr29/tr29-41.html However, the gold standard in UI implementations is that you never break the user's input.
- rockwotj 4y agoPersonally I think this is much easier to digest than the above link: https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ https://manishearth.github.io/blog/2017/01/14/stop-ascribing... It starts by breaking down common Unicode assumptions folks have
- Dylan16807 4y ago> See: codepoints, characters, grapheme clusters. People don't think about code points and they definitely don't think about code units. What does "character" mean? Grapheme clusters aren't perfect but they're far ahead of code whatevers.
- ryandrake 4y agoSeems like an obvious root cause for these sorts of things are languages (and developers) who can't or won't differentiate between "byte length" and "string length". Joel warned us all about this 20 years (!!) ago[1], and we're still struggling with it. 1: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
- rockwotj 4y agoString length isn't even a well defined term. Do you mean codepoint length? Or the number of graphemes?
- ryandrake 4y agoGood point. When it comes to text there are a lot of "lengths". Languages should differentiate between each of those lengths, and programmers should pay attention to the difference and actually use the right one for the whatever situation they're programming.
- jonhohle 4y agoIt seems odd to suggest the bug is with Safari. Normal humans (and even most developers!) don’t care that the byte length of an emoji in a particular encoding that may or may not be under their control defines the maximum “characters” in a text box (character used to define a logical collection of code points each of which may fit into one or multiple bytes).
- twic 4y agoIt's a bug with Safari because the HTML spec defines maxlength as applying to the number of UTF-16 code units [1]: > Constraint validation: If an element has a maximum allowed value length, its dirty value flag is true, its value was last changed by a user edit (as opposed to a change made by a script), and the length of the element's API value is greater than the element's maximum allowed value length, then the element is suffering from being too long. Where "the length of" is a link to [2]: > A string’s length is the number of code units it contains. And "code units" is a link to [3]: > A string is a sequence of unsigned 16-bit integers, also known as code units. I agree with your implied point that this is a questionable definition, though! [1] https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#limiting-user-input-length%3A-the-maxlength-attribute https://html.spec.whatwg.org/multipage/form-control-infrastr... [2] https://infra.spec.whatwg.org/#string-length https://infra.spec.whatwg.org/#string-length [3] https://infra.spec.whatwg.org/#code-unit https://infra.spec.whatwg.org/#code-unit
- Someone 4y agoI am not sure it’s a bug. [1] also says (emphasis added) “User agents _may_ prevent the user from causing the element's API value to be set to a value whose length is greater than the element's maximum allowed value length.” and we have “MAY This word, or the adjective "OPTIONAL", mean that an item is truly optional” [2] [1] https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#limiting-user-input-length%3A-the-maxlength-attribute https://html.spec.whatwg.org/multipage/form-control-infrastr... [2] https://www.ietf.org/rfc/rfc2119.txt https://www.ietf.org/rfc/rfc2119.txt
- 4y ago
- a_c 4y agoEdit 2: Actually it took 7 backspace to obliviate the whole family. Tough one. Edit: Oops. I was trying to type emoji into HN comment. apparently it is not supported. I never knew that [emoji of family of 4] takes 5 backspace to eliminate. It goes from [emoji of family of 4] to [emoji of family of 3] to [father and mother] to [father] to [father]. Somehow [father] can take double the (key)punch than the rest of his family.
- WirelessGigabit 4y agoThere is a ZWJ after father. That would explain the double punch.
- lapcat 4y agoThe bigger problem is that web browsers silently truncate strings over the maxlength. This behavior is particularly nasty for secure password fields.
- ok123456 4y agomulti-character graphemes were a mistake.
- deleted 4y ago[deleted]
- test6554 4y agoUnicode... There may be some regrets.
- raffy 4y agoI've created a bunch of Unicode tools for Ethereum Name Service (ENS). https://adraffy.github.io/ens-normalize.js/test/resolver.html https://adraffy.github.io/ens-normalize.js/test/resolver.htm... Also: https://adraffy.github.io/ens-normalize.js/test/emoji.html https://adraffy.github.io/ens-normalize.js/test/emoji.html https://adraffy.github.io/ens-normalize.js/test/chars.html https://adraffy.github.io/ens-normalize.js/test/chars.html https://adraffy.github.io/ens-normalize.js/test/confused.html https://adraffy.github.io/ens-normalize.js/test/confused.htm... https://adraffy.github.io/punycode.js/test/demo.html https://adraffy.github.io/punycode.js/test/demo.html Here is a video of another failure case: https://twitter.com/adraffy/status/1629262969581473792 https://twitter.com/adraffy/status/1629262969581473792
- kgwxd 4y agoI’ve worked on large data entry forms for decades. I stopped using maxlength a long time ago because of this. Entry should be free form, truncation is unexpected behavior. Validation should catch the limitation and never manipulate what the user entered. People paste not realizing text got cutoff and information gets lost. They type without looking at the screen. I’ve even seen it used on a password setting form but not on the sign in form so anyone that thinks they set a password larger than the limit thinks it succeeded but login fails and even tech support gets baffled for hours. We already know client side limits aren’t enough and server validation is required anyway. Trying to cleverly “help” entry usually just causes headaches for users and devs. Dynamically showing/hiding, enabling/disabling, focus jumping, auto formatting, updating other fields based on other values, is usually confusing for the user and more difficult to program correctly. Just show it all, allow all entry and catch everything in the validation code, everyone will be happier.