5 ms·
I like the stack trick. But it's still just a membuf library, without any string support. No encoding, no Unicode, no upper/lower/fc/norm support, which would
by rurban 8y ago
I like the stack trick.
But it's still just a membuf library, without any string support. No encoding, no Unicode, no upper/lower/fc/norm support, which would be important to compare or find strings. And coreutils (e.g grep) still have no unicode support. It's 2018, not the seventies anymore. Unicode strings need to be normalized to be able to be found.
- abenedic 8y agoI don't really know why you are being downvoted. There are a lot of string libraries that do not work well with Unicode strings. If you natively speak any non-European language it can be hard to use those libraries to effectively solve your problems. Not to bad mouth this project, but we don't really need a fast incomplete library.
- MrEldritch 8y agoIn the modern development environment, I think it's genuinely fair to say that something which doesn't support Unicode strings cannot be claimed to support strings. It's like saying you have the fastest integer math library ever written, with the minor caveat that it only supports numbers smaller than 256 because it achieves that speed by being simply a hardcoded lookup table, and crashes if you give it anything else.
- mehrdadn 8y agoDoes this library crash like in your example though? There's a difference between "not having Unicode-specific features" and "failing when given Unicode input".
- dwheeler 8y agoIt appears to support Unicode just fine, you just have to use UTF-8. I find that in most cases you should only use UTF-8 anyway, so in most cases that is not a problem.
- vbezhenar 8y agoIf you're using WinAPI, you have to use UCS-16. Converting strings for every call would be abysmal to performance.
- hrydgard 8y agoFor most software it's perfectly fine to convert. What Windows API call that takes string input do you call so often that it would be an issue?
- flohofwoe 8y agoThere are only very few WinAPI functions which use UCS-2 vs ASCII, and none of them are performance-critical. It's totally fine to convert from/to UTF-8 on the fly, especially since the conversion is extremely cheap (and the rest of the world has moved on to UTF-8 anyway).
- iforgotpassword 8y agoAll winapi functions use UCS2. The "ASCII" versions are just wrappers that implicitly convert from the current locale (latin1, ShiftJIS, BIG5, ...) to UCS2. Try passing some Japanese text to the ASCII version of a function on a system that's set to eg. German and see what happens...
- flohofwoe 8y agoMost Win32 API functions don't even take string parameters, so there's no separate ASCII or UCS-2 version of those. Of the functions that take strings, the old ...A() versions should never be called anyway since they depend on the obsolete concept of code pages. The proper way to work with strings on Windows (IMHO) is to encode all string data to UTF-8, only encode/decode from/to UCS-2 when needed, and call the ...W() functions explicitely (don't use the global UNICODE define). Also see: http://utf8everywhere.org/ http://utf8everywhere.org/
- rurban 8y agoNope, big mistake. You cannot just search for "Café" in "Café", as there are multiple representations for the same character. Think of Cyrillic vs Greek, Han vs Hangul. The popular garbage-in, garbage-out doesn't apply to unicode identifiers. That's why the Mac filesystem normalization was correct, and the unix filesystems are all broken. If it's not identifiable it's no identifier. That's why java and cperl are the only languages with proper and safe unicode support. Python 3 is a bit better than Python 2, Perl6 better than Perl5 and the rest, but still don't apply unicode security guidelines for identifiers. It's much better for browsers and email clients. And with case-insensitivity it gets worse, as there are some locale dependent additional rules, for Turkish and Lithuania. And this depends on "some" global settings. http://perl11.org/blog/foldcase.html http://perl11.org/blog/foldcase.html
- mamcx 8y agoI think instead is better to title this as "maybe the fastest ascii string library"
- johnboyer 8y agoUnicode is supported with UTF-8, only different character types aren't supported because generics are messy in C. I thought of accepting void* to support wchar_t and others, but some performance penalties came with it so I decided against it.
- lifthrasiir 8y agoNo, having different character types (I believe you are referring to C11's `char16_t` and `char32_t`?) is not a requirement for Unicode support. At the very least you need to have a single function or two that... * Receives a string expected to be encoded in UTF-8, and an offset to it expected to be a UTF-8 sequence boundary. * Scans forward or backward for the next or previous UTF-8 sequence boundary. * Optionally returns the code point for the scanned UTF-8 sequence. * Has proper error handling for every imaginable cases: out of boundary, not a boundary, not a valid UTF-8 sequence. (OOB case needs to be handled because it will be the end condition of the iteration. Preferably should be distinct from other error conditions.) Every other functionality can build upon this little function, in particular the iteration and UTF-8 validation will be trivial. The full Unicode support including case mapping, folding, normalization and property lookup will of course require a not-so-small table but is not strictly necessary anyway. Björn Höhrmann's Flexible and Economical UTF-8 Decoder [1] will be handy for a concise implementation. [1] https://bjoern.hoehrmann.de/utf-8/decoder/dfa/ https://bjoern.hoehrmann.de/utf-8/decoder/dfa/
- flohofwoe 8y agoI don't see any functions in the OP's library that would require dedicated UTF-8 handling. The string length is given in bytes, not characters or codepoints. There's no functionality to give you the character at n-th location etc... you can easily implement all UNICODE-specific functionality in a separate library and use it together with the OPs library. IMHO that's even preferable.
- whyever 8y ago> But it's still just a membuf library, without any string support. Just like the strings in the C and C++ standard library.
- rurban 8y agoExactly. POSIX and the C/C++ standards don't support strings and utf-8 yet neither. coreutils and grep neither. Most computer languages neither. It's a mess and a big security risk. We are still in the stone age of string support.
- flohofwoe 8y agoAt least on Mac the command line utils support international text just fine, all text files are encoded as UTF-8 by convention, and (for instance) grep's case-insensitive search works for non-ASCII characters.
- burntsushi 8y agoIt's the same on Linux. GNU grep supports Unicode just fine, assuming, of course, your system's locale settings are configured correctly. :-)