16 ms·

So Long Surrogates: How We Moved to UTF-8 in Haskell

140 points by wofo 4y ago

nerdponx 4y ago
This site saves 26 "statistics" cookies and 99 "marketing" cookies. Really? Is all that necessary?
- meetups323 4y ago
  Your user agent saves the cookies. If you don't like it, change it.
  - hombre_fatal 4y ago
    I do think cookies get unfair treatment. They are things that your browser happily rebroadcasts back to the server with no real UI for it outside of the shitty devtool bar made for devs, even after all this outcry about cookies. It reminds me of the meme of the guy riding a bicycle, throwing a branch into the spokes (rebroadcasting cookies), and then roaring in pain on the ground about how evil websites/advertisers are tracking him with cookies. That said, what a lame HN thread on a post about Haskell.
    - eklavya 4y ago
      I have come to accept it and just ignore it. Many times there would be a long thread having to do absolutely nothing about the topic at hand. Not a tangent but like completely unrelated, why are we even discussing this here kind of thing. I wish there was a good way to visually differentiate when a new top comment starts except by squinting and figuring out the whitespace from the left of the mobile screen, more painful than necessary I presume.
      - Mindless2112 4y ago
        Use HackerWeb. Top-level comments are highlighted, and it automatically collapses threads to show only top-level comments when there are a lot of comments. https://hackerweb.app/#/item/31181595 https://hackerweb.app/#/item/31181595
      - hombre_fatal 4y ago
        I couldn't help myself. Always desperate to find an opportunity to shove my 2 cents into the world. So imagine my glee when you provided me with another one! Yeah, I think both (A) defaulting to auto-expanded threads and (B) making them annoying to collapse make HN worse than it could be. You tend to read the top-level thread because it's already there. And then it ends up being longer than you expected, or you're trapped in a subtree that just won't end, or you just want to see what other people are saying. And there's no good way to move past it. Would be nice to click the indentation to collapse the thread anywhere inside the tree.
        boogies 4y ago
        I just scroll to the top and use the “next” link on the top comment (added with the prev and context links around October 27–28th last year I think).
        eklavya 4y ago
        Thanks a lot for sharing boogies, much simpler now.
  - bawolff 4y ago
    Ignoring the privacy bit - 125 cookies is quite a bit of per request overhead, especially in http/1.1 where they are not compressed. I would say its poor website design.
  - Rygian 4y ago
    Why shift the burden on the user and the user agent? The website is the only one to blame here.
    - meetups323 4y ago
      Blaming the website for your own agent doing something you don't want it to is learned helplessness. Every marketing cookie generates revenue for the website in some way or another. The website wants revenue, so it asks the user agent to maintain those cookies. The user agent agrees. Then the operator of the user agent gets upset that the website asked their agent to store the cookies? Get upset that your agent agreed, not that a request was made. Or better yet, don't get upset at all and just solve the darned problem yourself. Is this Hacker News or Complier News?
      - jstimpfle 4y ago
        Cookies as a mechanism are useful and required for a solid modern web experience. However, tracking cookies are arguably the opposite of that. A typical modern website with marketing comes with, I don't know, 100s of cookies. Are you really arguing that the user should be required to vet each individual cookie whenever following a link with unvetted cookies? Or how do you solve this problem? Personally, the most I can be arsed to do is install some Adblock Plugin. I did that only a few months ago and I'm not even sure that it improved my experience by a lot.
        zasdffaa 4y ago
        > and required for a solid modern web experience Absence of cookies don't make things unstable (non-solid?), and fuck knows what 'modern' is supposed to mean, or why it's good. > Or how do you solve this problem? Block all cookies except for rare moments like posting on HN, which then immediately get deleted. And no JS, which means CPU is trivial (so no burn-a-core-for-every-open-tab which is so common with page-sized pointless animations). Many problems can be solved if you want them to be.
        4y ago
  - deathanatos 4y ago
    Heh, so I actually do this. An incredible amount of the web just breaks. Twitter, Reuters, Imgur. Like it's one thing if, when I attempt to log in, your log in fails (and usually, logins fail to handle the error & will just loop back to the start, that's at least a start) but a lot of the web will have a flash-of-text and then nothing, & JS has crashed.
languageserver 4y ago
I am always extremely doubtful of these types of blogposts that take a well-known algorithm and somehow beat all others (including academia, bioinformatics tools, etc.) with a fancy implementation in <insert cool programming language 2022>
- Nebasuke 4y ago
  The article is about how they moved an existing (fast) implementation in Haskell in UTF-16 to an even faster implementation in Haskell by switching to UTF-8. This is stated in the first paragraph. The post they reference, is also very honest: ..., the fastest Haskell implementation of the Aho–Corasick string searching algorithm, which powers string search in Channable. Basically the blog posts show that if you want to program in Haskell and still optimise, this is how you can do it. I think both posts are great resources and don't overstate their claims.
- danschuller 4y ago
  I was taught Haskell at university and I'm old. Looking at it's wiki page it's a 32 year old language not that much younger than 37 year old C++.
- poorlyknit 4y ago
  (author here) I wrote this article during a short internship at Channable. Not to be apologetic but I think these kind of articles are so prevalent because young or unpopular languages usually have worse documentation than established ones (naturally). I basically wrote down the things I learned during my internship that I found noteworthy.
crdrost 4y ago
Oh wow. That is really not very much pain, as described. I have to say, I never thought that the benefit of Haskell having a horrible native string type would be "you can just upgrade strings like any other dependency," which is really kinda slick. You think about how much pain there was for Py2 -> Py3 where one of the big sticking factors was all of the distinctions around strings and encoding and byte arrays... this is comparatively quite nice. Makes me wonder how much of a programming language can be hotswappable.
- resoluteteeth 4y ago
  Utf8 vs utf16 as the internal representation of the Unicode string type is mostly just an implementation detail. This is very different from going from python2, which conflated bytes and ascii strings, to python3, which intentionally changed the api to propely distinguish sequences of bytes and strings.
- antonvs 4y ago
  > Makes me wonder how much of a programming language can be hotswappable. For a research language that can make a lot of sense, not so much for a language to be used in industry. The downside is that different libraries will have different string representations, so you can end up being forced to do a lot of conversion if you're using different libraries that have made different choices from each other, or from your own code. There are at least 5 commonly used string types - String (linked list of Char), ByteString lazy & strict, and Text lazy and strict. The latter two have a good rationale for being different - byte strings are not necessarily text - but, for various reasons, they're often used to represent text anyway. These five also have corresponding `readFile` functions - see https://www.snoyman.com/blog/2016/12/beware-of-readfile/ https://www.snoyman.com/blog/2016/12/beware-of-readfile/ . As Snoyman recommends in that post, it's probably best to "Stick with Data.ByteString.readFile for known-small data, use a streaming package (e.g, conduit) if your choice for large data, and handle the character encoding yourself. And apply this to writeFile and other file-related functions as well." The first comment on that post starts out with "This problem extends well beyond readFile." Having the string handling more standardized at the language level can make life quite a bit simpler for developers.
Aardwolf 4y ago
If only Windows, Java and JavaScript could also move away from internal usage of UTF-16, it's purely a legacy format and the worst of both worlds (UTF-32 and UTF-8). Even worse is that unicode itself, which should in theory be a list of codes for glyphs, modifiers and other script related values, that's independent of encoding, had to have some codes reserved for "surrogates" for the UTF-16 encoding anyway. UTF-8 doesn't need such a thing...
- layer8 4y ago
  Besides the surrogate characters there are also some other noncharacters: https://www.unicode.org/faq/private_use.html#noncharacters https://www.unicode.org/faq/private_use.html#noncharacters Because of modifier characters, control characters like for bidi, stuff like soft-hyphens and ligatures, locale-dependent semantics (upper/lowercase, collation etc.), the general discordance between glyphs and characters, and so on and so forth, Unicode is so complex, and in general always requires careful processing of code point (or code unit) sequences, that honestly the surrogate encoding doesn’t make that much of a difference. It’s just an additional wrinkle in a sea of wrinkles.
  - Aardwolf 4y ago
    I still find the surrogates different. Bidi, private use, ligatures, ... are script or locale related. Unicode uses numeric values from 0 to 1112063. You can invent all kinds of methods to encode numbers from 0 to 1112063 (variable length, fixed length, decimal, hexadecimal, anything else). But most ways I can think of to encode these numbers, including variable length ones that would use 8 bit or 16 bit primitives, don't require me to actually reserve some of those to-be-encoded numbers themselves for a special meaning. Yet for UTF-16 they managed to do it. Imagine that all other encodings out there would also want to reserve some Unicode values for their own purpose!
    - layer8 4y ago
      You always have to work with sequences of code units anyway (instead of just single code points), so the individual reasons for that doesn’t make much of a difference. It seems your rejection is more on aesthetic than on practical grounds.
dmz73 4y ago
I don't understand how are surrogates in UTF16 a problem solved by UTF8. From the article it seems that two main improvements were smaller memory footprint due to mostly ASCII data and using better algorithm which resulted in even better performance in UTF16. I had to deal with surrogates in UTF16 and it is much easier than dealing with variable encoding in UTF8. Naive UTF8 decoder is easy and fast but if one takes time to fully validate each UTF8 code point then it becomes much more difficult and a lot slower...and that is going forward, trying to move backwards in UTF8 is again much harder than in UTF16. The main disadvantage of UTF16 is memory usage when dealing with ASCII data but if that is the case then just use array of bytes and don't worry about unicode.

140 points by wofo 4y ago

nerdponx 4y ago

This site saves 26 "statistics" cookies and 99 "marketing" cookies. Really? Is all that necessary?

meetups323 4y ago
Your user agent saves the cookies. If you don't like it, change it.
- hombre_fatal 4y ago
  I do think cookies get unfair treatment. They are things that your browser happily rebroadcasts back to the server with no real UI for it outside of the shitty devtool bar made for devs, even after all this outcry about cookies. It reminds me of the meme of the guy riding a bicycle, throwing a branch into the spokes (rebroadcasting cookies), and then roaring in pain on the ground about how evil websites/advertisers are tracking him with cookies. That said, what a lame HN thread on a post about Haskell.
  - eklavya 4y ago
    I have come to accept it and just ignore it. Many times there would be a long thread having to do absolutely nothing about the topic at hand. Not a tangent but like completely unrelated, why are we even discussing this here kind of thing. I wish there was a good way to visually differentiate when a new top comment starts except by squinting and figuring out the whitespace from the left of the mobile screen, more painful than necessary I presume.
    - Mindless2112 4y ago
      Use HackerWeb. Top-level comments are highlighted, and it automatically collapses threads to show only top-level comments when there are a lot of comments. https://hackerweb.app/#/item/31181595 https://hackerweb.app/#/item/31181595
    - hombre_fatal 4y ago
      I couldn't help myself. Always desperate to find an opportunity to shove my 2 cents into the world. So imagine my glee when you provided me with another one! Yeah, I think both (A) defaulting to auto-expanded threads and (B) making them annoying to collapse make HN worse than it could be. You tend to read the top-level thread because it's already there. And then it ends up being longer than you expected, or you're trapped in a subtree that just won't end, or you just want to see what other people are saying. And there's no good way to move past it. Would be nice to click the indentation to collapse the thread anywhere inside the tree.
      - boogies 4y ago
        I just scroll to the top and use the “next” link on the top comment (added with the prev and context links around October 27–28th last year I think).
        eklavya 4y ago
        Thanks a lot for sharing boogies, much simpler now.
- bawolff 4y ago
  Ignoring the privacy bit - 125 cookies is quite a bit of per request overhead, especially in http/1.1 where they are not compressed. I would say its poor website design.
- Rygian 4y ago
  Why shift the burden on the user and the user agent? The website is the only one to blame here.
  - meetups323 4y ago
    Blaming the website for your own agent doing something you don't want it to is learned helplessness. Every marketing cookie generates revenue for the website in some way or another. The website wants revenue, so it asks the user agent to maintain those cookies. The user agent agrees. Then the operator of the user agent gets upset that the website asked their agent to store the cookies? Get upset that your agent agreed, not that a request was made. Or better yet, don't get upset at all and just solve the darned problem yourself. Is this Hacker News or Complier News?
    - jstimpfle 4y ago
      Cookies as a mechanism are useful and required for a solid modern web experience. However, tracking cookies are arguably the opposite of that. A typical modern website with marketing comes with, I don't know, 100s of cookies. Are you really arguing that the user should be required to vet each individual cookie whenever following a link with unvetted cookies? Or how do you solve this problem? Personally, the most I can be arsed to do is install some Adblock Plugin. I did that only a few months ago and I'm not even sure that it improved my experience by a lot.
      - zasdffaa 4y ago
        > and required for a solid modern web experience Absence of cookies don't make things unstable (non-solid?), and fuck knows what 'modern' is supposed to mean, or why it's good. > Or how do you solve this problem? Block all cookies except for rare moments like posting on HN, which then immediately get deleted. And no JS, which means CPU is trivial (so no burn-a-core-for-every-open-tab which is so common with page-sized pointless animations). Many problems can be solved if you want them to be.
        4y ago
- deathanatos 4y ago
  Heh, so I actually do this. An incredible amount of the web just breaks. Twitter, Reuters, Imgur. Like it's one thing if, when I attempt to log in, your log in fails (and usually, logins fail to handle the error & will just loop back to the start, that's at least a start) but a lot of the web will have a flash-of-text and then nothing, & JS has crashed.

languageserver 4y ago

I am always extremely doubtful of these types of blogposts that take a well-known algorithm and somehow beat all others (including academia, bioinformatics tools, etc.) with a fancy implementation in <insert cool programming language 2022>

Nebasuke 4y ago
The article is about how they moved an existing (fast) implementation in Haskell in UTF-16 to an even faster implementation in Haskell by switching to UTF-8. This is stated in the first paragraph. The post they reference, is also very honest: ..., the fastest Haskell implementation of the Aho–Corasick string searching algorithm, which powers string search in Channable. Basically the blog posts show that if you want to program in Haskell and still optimise, this is how you can do it. I think both posts are great resources and don't overstate their claims.
danschuller 4y ago
I was taught Haskell at university and I'm old. Looking at it's wiki page it's a 32 year old language not that much younger than 37 year old C++.
poorlyknit 4y ago
(author here) I wrote this article during a short internship at Channable. Not to be apologetic but I think these kind of articles are so prevalent because young or unpopular languages usually have worse documentation than established ones (naturally). I basically wrote down the things I learned during my internship that I found noteworthy.

crdrost 4y ago

Oh wow. That is really not very much pain, as described. I have to say, I never thought that the benefit of Haskell having a horrible native string type would be "you can just upgrade strings like any other dependency," which is really kinda slick. You think about how much pain there was for Py2 -> Py3 where one of the big sticking factors was all of the distinctions around strings and encoding and byte arrays... this is comparatively quite nice. Makes me wonder how much of a programming language can be hotswappable.

resoluteteeth 4y ago
Utf8 vs utf16 as the internal representation of the Unicode string type is mostly just an implementation detail. This is very different from going from python2, which conflated bytes and ascii strings, to python3, which intentionally changed the api to propely distinguish sequences of bytes and strings.
antonvs 4y ago
> Makes me wonder how much of a programming language can be hotswappable. For a research language that can make a lot of sense, not so much for a language to be used in industry. The downside is that different libraries will have different string representations, so you can end up being forced to do a lot of conversion if you're using different libraries that have made different choices from each other, or from your own code. There are at least 5 commonly used string types - String (linked list of Char), ByteString lazy & strict, and Text lazy and strict. The latter two have a good rationale for being different - byte strings are not necessarily text - but, for various reasons, they're often used to represent text anyway. These five also have corresponding `readFile` functions - see https://www.snoyman.com/blog/2016/12/beware-of-readfile/ https://www.snoyman.com/blog/2016/12/beware-of-readfile/ . As Snoyman recommends in that post, it's probably best to "Stick with Data.ByteString.readFile for known-small data, use a streaming package (e.g, conduit) if your choice for large data, and handle the character encoding yourself. And apply this to writeFile and other file-related functions as well." The first comment on that post starts out with "This problem extends well beyond readFile." Having the string handling more standardized at the language level can make life quite a bit simpler for developers.

Aardwolf 4y ago

If only Windows, Java and JavaScript could also move away from internal usage of UTF-16, it's purely a legacy format and the worst of both worlds (UTF-32 and UTF-8). Even worse is that unicode itself, which should in theory be a list of codes for glyphs, modifiers and other script related values, that's independent of encoding, had to have some codes reserved for "surrogates" for the UTF-16 encoding anyway. UTF-8 doesn't need such a thing...

layer8 4y ago
Besides the surrogate characters there are also some other noncharacters: https://www.unicode.org/faq/private_use.html#noncharacters https://www.unicode.org/faq/private_use.html#noncharacters Because of modifier characters, control characters like for bidi, stuff like soft-hyphens and ligatures, locale-dependent semantics (upper/lowercase, collation etc.), the general discordance between glyphs and characters, and so on and so forth, Unicode is so complex, and in general always requires careful processing of code point (or code unit) sequences, that honestly the surrogate encoding doesn’t make that much of a difference. It’s just an additional wrinkle in a sea of wrinkles.
- Aardwolf 4y ago
  I still find the surrogates different. Bidi, private use, ligatures, ... are script or locale related. Unicode uses numeric values from 0 to 1112063. You can invent all kinds of methods to encode numbers from 0 to 1112063 (variable length, fixed length, decimal, hexadecimal, anything else). But most ways I can think of to encode these numbers, including variable length ones that would use 8 bit or 16 bit primitives, don't require me to actually reserve some of those to-be-encoded numbers themselves for a special meaning. Yet for UTF-16 they managed to do it. Imagine that all other encodings out there would also want to reserve some Unicode values for their own purpose!
  - layer8 4y ago
    You always have to work with sequences of code units anyway (instead of just single code points), so the individual reasons for that doesn’t make much of a difference. It seems your rejection is more on aesthetic than on practical grounds.

dmz73 4y ago

I don't understand how are surrogates in UTF16 a problem solved by UTF8. From the article it seems that two main improvements were smaller memory footprint due to mostly ASCII data and using better algorithm which resulted in even better performance in UTF16. I had to deal with surrogates in UTF16 and it is much easier than dealing with variable encoding in UTF8. Naive UTF8 decoder is easy and fast but if one takes time to fully validate each UTF8 code point then it becomes much more difficult and a lot slower...and that is going forward, trying to move backwards in UTF8 is again much harder than in UTF16. The main disadvantage of UTF16 is memory usage when dealing with ASCII data but if that is the case then just use array of bytes and don't worry about unicode.