7 ms·
Are there many cases where plain ASCII is okay anymore, anyway? Wouldn't proper UTF-whatever comparisons mandate doing it the slow way, if you wanted counts ver
by tyingq 5y ago
Are there many cases where plain ASCII is okay anymore, anyway? Wouldn't proper UTF-whatever comparisons mandate doing it the slow way, if you wanted counts versus just "same/not the same"?
- bombcar 5y agoThat would be more interesting, something that could handle a UTF-8 string in a similar situation (I guess you'd compare the bytes and then do a walk to see if it was part of a multi-byte character? Maybe the high-bit would help?).
- joawowiij 5y agohex, base64 encoded strings, etc
- tyingq 5y agoThat makes sense for "same/not-same" comparisons to me. I don't know why you would want counts of different characters though.
- bombcar 5y ago"Password must be different by at least 6 characters" perhaps, though that's the inverse, really. Fuzzy matching is another aspect that comes to mind.
- rocqua 5y agoFor UTF-8 I believe the question is ill-posed. The "position" of a "character" in unicode is very hard to determine and depends largely on what you take to be a character.
- bmn__ 5y agoThen UTF-8 should not be operated on directly. When reading from a file etc., first decode the input into an internal string format that has straight-forward linear access to each gc boundary.
- eesmith 5y agoEven then, https://en.wikipedia.org/wiki/Unicode_equivalence https://en.wikipedia.org/wiki/Unicode_equivalence . What does "matching character" mean given combining characters?
- masklinn 5y agoProper comparison would have to be even slower as you’d need to deal with normalisation & equivalence. The « slow way » of the post would also count matching code units which is utterly useless, really.
- Thaxll 5y agoHTTP headers.
- nonameiguess 5y agoThe actual killer application of character count matching is bioinformatics. You only need four distinct characters to encode DNA, which of course, means in reality you only need two bits, but I don't know if anyone has bothered to develop a two-bit character encoding system and instructions that can operate on them. They have come up with extremely optimized implementations that compare byte streams, though, utilizing ASCII GCTA.
- TchoBeer 5y agoexcept amino acids are encoded for with three bases, meaning that the actual character encoding system is 6 bits.
- jakobnissen 5y agoBioinformatician here. Yes, people have come up with many encoding systems: 2-bit encoding systems, 3-bit encoding systems (for N's), various encoding schemes that allows long runs of N's to be stored with little memory costs, and even reduced amino acid alphabets for e.g. 4-bit amino acid sequences. In my experience, the performance bottlenecks in bioinformatics is usually IO, parsing, de/compression and needlessly slow algorithms. You rarely get meaningful speed improvements by internally encoding the sequences one way or another (storing small fixed-size sequences in machine integers is an exception). There can be other advantages, though, like reduced memory consumption. Partly for that reason, most DNA data is just stored as gzipped text files.
- GianFabien 5y agoCould be an area to use FPGA based arrays for far greater throughput. Some recent RISC-V designs include FPGA blocks.
- sfink 5y agoYes! Plain ASCII is great, even when you need to properly handle non-ASCII. When >90% of your strings are ASCII in practice, then having an "is ASCII" bit in your string headers can save a ton of memory, and gain more performance than you lose in flag checks and extra code. Heck, if your usual encoding is UTF-8, the flag bit can be purely advisory and ignored when it doesn't matter.
- redis_mlc 5y agoI get asked this a lot as a DBA. Most strings that are not end-user input related are still ASCII, and you can save a lot of resources by designing schemas with that in mind.