6 ms·

Show HN: I built an embeddable Unicode library with MISRA C conformance

117 points by hgs3 2y ago

Hello, everyone. I built Unicorn: an embeddable MISRA C:2012 implementation of essential Unicode Algorithms. Unicorn is designed to be fully customizable: you can select which Unicode algorithms and character properties are included or excluded from compilation. You can also exclude Unicode character blocks wholesale for scripts your application does not support. It's perfect for resource constrained devices like microcontrollers and IoT devices. About me: I quit my Big Corp job a few years back to pursue my passion for software development and this is one of my first commercial releases.

sushidev 2y ago
Nice!
- sushidev 2y ago
  But not interesting for me in any way since it’s not open source.
  - hgs3 2y ago
    Unfortunately, the entire reason I didn't release Unicorn under an OSI approved license is because I see many (most?) FOSS projects are chronically underfunded. Now, I did not quit my job and build this to get rich or anything, but I do need to earn enough to sustain myself. If there's enough interest, I would consider crowdfunding a release under an OSI license.
    - kouteiheika 2y ago
      Why not dual license it under a commercial license and something like GPL?
      - hgs3 2y ago
        I went back and forth on this and in my uncertainty I decided it was better to start more "closed" first with the potential to become more "open" over time.
        h4ck_th3_pl4n3t 2y ago
        Therefore it will never be open source, and if so then only when you lost interest in the project. Got it.
- deleted 2y ago
  [deleted]
- hgs3 2y ago
  Thank you, let me know if you have any questions.
  - sushidev 2y ago
    Who is your main target audience?
    - hgs3 2y ago
      Primarily, companies developing for embedded systems or other resource constrained devices.
rubicks 2y ago
This is not a comment about open/closed-source software and/or licensing models. Projects like this never fail to impress me vis-a-vis source obfuscation. The 'generate.pyz' is an interesting twist on the usual practice.
- layer8 2y ago
  # You may not reverse engineer, decompile, disassemble, or otherwise attempt # to derive the source code or underlying structure of this script This prohibition is void in certain relevant jurisdictions, for any publicly available product.
chris_wot 2y ago
I don’t get the whole MISRA requirement that functions should only have one exit point. Honestly, nobody has been able to explain why this is important, other than it’s a historical anomaly inherited from FORTRAN. (Which was actually for a good reason)
- dark-star 2y ago
  one reason I can think of from the top of my head (although I never had to deal with MISRA C at all) is that if you have to add some cleanup code before your function returns, then there is exactly one place and one place only to do that. Otherwise this leads to duplication of cleanup code similar to allocate_something() .. if failed(foo) { deallocate_something() return FAILED; } .. deallocate_something() return SUCCESS;
  - samatman 2y ago
    This is, more than anything, an argument for a `defer` statement, of the sort you can enjoy in Zig right now. Or hopefully, eventually, in C, thanks to the tireless efforts of JeanHeyde Meneide: https://thephd.dev/just-put-raii-in-c-bro-please-bro-just-one-more-destructor-bro-cmon-im-good-for-it https://thephd.dev/just-put-raii-in-c-bro-please-bro-just-on...
    - layer8 2y ago
      MISRA C can’t mandate new language features though.
      - AlotOfReading 2y ago
        The MISRA people work with the C/C++ committees on upcoming language changes, which gives them loud voices to push things they want promoted.
    - eddd-ddde 2y ago
      > certain name manglings are not guaranteed to be 1:1 and can infact “demangle” into multiple different plausible entities. Now I'm really curious, doesn't that mean some valid C++ code would fail to link for having multiple definitions of the same symbol?? I would expect name mangling to be a bijection from function prototype to string.
kiritanpo 2y ago
This looks interesting. Most embedded project I know use ICU/libicu for their unicode needs. As a potential customer I would like to know how does it compare against ICU for performance and code size. Why should I switch?
- hgs3 2y ago
  > I would like to know how does it compare against ICU ICU is a large library, typically around ~40 MB depending on the platform, whereas Unicorn, with all features enabled, is only about 600 KB. ICU has a broader scope: it's not just a Unicode library, but also an internationalization library. Unicorn, on the other hand, is specifically focused on Unicode algorithms. ICU wasn't designed to be customized. It's also non-MISRA compliant and written in C++11. In contrast, Unicorn is written in C99, fully customizable, MISRA compliant, and only requires a few features from libc [1]. It's far more portable. [1] https://github.com/railgunlabs/unicorn/?tab=readme-ov-file#ultra-portable https://github.com/railgunlabs/unicorn/?tab=readme-ov-file#u...
  - ranger_danger 2y ago
    The most important difference to me, which is a deal-breaker, is that Unicorn is non-commercial. Note that I am not interested in actually using Unicorn commercially, but my understanding is that this restriction makes the library incompatible with FOSS licenses such as GPL.
rurban 2y ago
This is commercial only. Free and small is my safeclib, which does about half of it. ICU is not usable on small devices, and also pretty slow. It's much faster to use precomputed tables per algorithm, such as here or in safeclib. libunistring is also extremely slow. This was tried for grep and failed.
- hgs3 2y ago
  > This is commercial only. You can use Unicorn for non-commercial use [1], but yes, for commercial use you need to buy a license. > It's much faster to use precomputed tables per algorithm You're absolutely right about using precomputed tables per algorithm. That is the secret to the library's speed. > Free and small is my safeclib, which does about half of it. I like safeclib! It's nice to hear from the author. It's worth distinguishing that safeclib is a safer string library whereas Unicorn is a Unicode algorithms library, not a string library. [1] https://github.com/railgunlabs/unicorn/blob/master/LICENSE https://github.com/railgunlabs/unicorn/blob/master/LICENSE
  - rurban 2y ago
    Well, a string is unicode nowadays. And for sure not just a zero-terminated blob. That would be a buffer. Only the Linux kernel still holds this invalid view. So every string library needs at least a compare function to find strings, with all the variants of same graphemes. Which leads us to NFC normalization for a start. Upcase tables and wordlength tables are also needed.
- fao_ 2y ago
  > This is commercial only. Free and small is my safeclib Is it me or does this feel a bit weird? It seems like you're using the comments section here to self-advertise for exposure. I read it like — "businesses can't use this without paying the OP, however, if you're a business you can get 50% of the way there by using _my_ library, and you don't even have to pay me!". It comes off incredibly rude to try to undercut the OP like this.
  - josephcsible 2y ago
Someone 2y ago
On https://railgunlabs.com/unicorn/manual/misra-compliance/ https://railgunlabs.com/unicorn/manual/misra-compliance/, I think you will want to fix a typo in 1.2 Required Compliant (verified by compiling with Clang's -pdentic flag) ^^^^^^^^ Or am I too pedantic?
- hgs3 2y ago
  This is the most ironic typo I've ever made. Thanks for the catch. I've corrected it.
  - canucker2016 2y ago
    A couple more suggestions. List the platforms (& compilers) that you've tested on. Compare (pros/cons) against other Unicode libs (like others have done elsewhere in this thread, i.e. https://news.ycombinator.com/item?id=42424637 https://news.ycombinator.com/item?id=42424637 and https://news.ycombinator.com/item?id=42424638 https://news.ycombinator.com/item?id=42424638)
    - hgs3 2y ago
      Thanks for the suggestions. I think a comparison table would be useful, but I want to make sure I do it right since I'd be comparing my work to someone else's. As for the compilers, I’ve tested the library with GCC, Clang, and MSVC, and with the -pedantic flag like the GP mentioned. The library should build with any standard-compliant C99 compiler.
- deleted 2y ago
  [deleted]
  - deleted 2y ago
    [deleted]
__turbobrew__ 2y ago
There is not much to show if I can’t read the source code.
- hgs3 2y ago
  You can download a prebuilt amalgamation from GitHub to see the amalgamated C code [1]. The GitHub repo contains the code that generates the amalgamation. [1] https://github.com/railgunlabs/unicorn/releases/ https://github.com/railgunlabs/unicorn/releases/
tocariimaa 2y ago
It uses a privative license if you're wondering.
biosboiii 2y ago
Since MISRA is targetted at Automotive, as a software dev in the automotive space I would suggest adding the note that this is able to run on POSIX compliant OSes like QNX :) If you would like to chat, hit me up.
deleted 2y ago
[deleted]
garganzol 2y ago
My comment is not directly related to the particular project which is impressive, but more to its presentation. If you go to the author's website, you will find neat to-the-point manuals and other useful information. This is what I call the real Web 3.0. Simple and to the point. Also the main company page is humorous in a good way, about the mad scientist etc.
- hgs3 2y ago
  I appreciate the kind words. When I was designing the website, I wanted to inject my personality into it. I wasn't sure how well the "playfulness" would go over, but I'm glad you enjoyed it.

117 points by hgs3 2y ago

sushidev 2y ago

Nice!

sushidev 2y ago
But not interesting for me in any way since it’s not open source.
- hgs3 2y ago
  Unfortunately, the entire reason I didn't release Unicorn under an OSI approved license is because I see many (most?) FOSS projects are chronically underfunded. Now, I did not quit my job and build this to get rich or anything, but I do need to earn enough to sustain myself. If there's enough interest, I would consider crowdfunding a release under an OSI license.
  - kouteiheika 2y ago
    Why not dual license it under a commercial license and something like GPL?
    - hgs3 2y ago
      I went back and forth on this and in my uncertainty I decided it was better to start more "closed" first with the potential to become more "open" over time.
      - h4ck_th3_pl4n3t 2y ago
        Therefore it will never be open source, and if so then only when you lost interest in the project. Got it.
deleted 2y ago
[deleted]
hgs3 2y ago
Thank you, let me know if you have any questions.
- sushidev 2y ago
  Who is your main target audience?
  - hgs3 2y ago
    Primarily, companies developing for embedded systems or other resource constrained devices.

rubicks 2y ago

This is not a comment about open/closed-source software and/or licensing models. Projects like this never fail to impress me vis-a-vis source obfuscation. The 'generate.pyz' is an interesting twist on the usual practice.

layer8 2y ago
# You may not reverse engineer, decompile, disassemble, or otherwise attempt # to derive the source code or underlying structure of this script This prohibition is void in certain relevant jurisdictions, for any publicly available product.

chris_wot 2y ago

I don’t get the whole MISRA requirement that functions should only have one exit point. Honestly, nobody has been able to explain why this is important, other than it’s a historical anomaly inherited from FORTRAN. (Which was actually for a good reason)

dark-star 2y ago
one reason I can think of from the top of my head (although I never had to deal with MISRA C at all) is that if you have to add some cleanup code before your function returns, then there is exactly one place and one place only to do that. Otherwise this leads to duplication of cleanup code similar to allocate_something() .. if failed(foo) { deallocate_something() return FAILED; } .. deallocate_something() return SUCCESS;
- samatman 2y ago
  This is, more than anything, an argument for a `defer` statement, of the sort you can enjoy in Zig right now. Or hopefully, eventually, in C, thanks to the tireless efforts of JeanHeyde Meneide: https://thephd.dev/just-put-raii-in-c-bro-please-bro-just-one-more-destructor-bro-cmon-im-good-for-it https://thephd.dev/just-put-raii-in-c-bro-please-bro-just-on...
  - layer8 2y ago
    MISRA C can’t mandate new language features though.
    - AlotOfReading 2y ago
      The MISRA people work with the C/C++ committees on upcoming language changes, which gives them loud voices to push things they want promoted.
  - eddd-ddde 2y ago
    > certain name manglings are not guaranteed to be 1:1 and can infact “demangle” into multiple different plausible entities. Now I'm really curious, doesn't that mean some valid C++ code would fail to link for having multiple definitions of the same symbol?? I would expect name mangling to be a bijection from function prototype to string.

kiritanpo 2y ago

This looks interesting. Most embedded project I know use ICU/libicu for their unicode needs. As a potential customer I would like to know how does it compare against ICU for performance and code size. Why should I switch?

hgs3 2y ago
> I would like to know how does it compare against ICU ICU is a large library, typically around ~40 MB depending on the platform, whereas Unicorn, with all features enabled, is only about 600 KB. ICU has a broader scope: it's not just a Unicode library, but also an internationalization library. Unicorn, on the other hand, is specifically focused on Unicode algorithms. ICU wasn't designed to be customized. It's also non-MISRA compliant and written in C++11. In contrast, Unicorn is written in C99, fully customizable, MISRA compliant, and only requires a few features from libc [1]. It's far more portable. [1] https://github.com/railgunlabs/unicorn/?tab=readme-ov-file#ultra-portable https://github.com/railgunlabs/unicorn/?tab=readme-ov-file#u...
- ranger_danger 2y ago
  The most important difference to me, which is a deal-breaker, is that Unicorn is non-commercial. Note that I am not interested in actually using Unicorn commercially, but my understanding is that this restriction makes the library incompatible with FOSS licenses such as GPL.

rurban 2y ago

This is commercial only. Free and small is my safeclib, which does about half of it. ICU is not usable on small devices, and also pretty slow. It's much faster to use precomputed tables per algorithm, such as here or in safeclib. libunistring is also extremely slow. This was tried for grep and failed.

hgs3 2y ago
> This is commercial only. You can use Unicorn for non-commercial use [1], but yes, for commercial use you need to buy a license. > It's much faster to use precomputed tables per algorithm You're absolutely right about using precomputed tables per algorithm. That is the secret to the library's speed. > Free and small is my safeclib, which does about half of it. I like safeclib! It's nice to hear from the author. It's worth distinguishing that safeclib is a safer string library whereas Unicorn is a Unicode algorithms library, not a string library. [1] https://github.com/railgunlabs/unicorn/blob/master/LICENSE https://github.com/railgunlabs/unicorn/blob/master/LICENSE
- rurban 2y ago
  Well, a string is unicode nowadays. And for sure not just a zero-terminated blob. That would be a buffer. Only the Linux kernel still holds this invalid view. So every string library needs at least a compare function to find strings, with all the variants of same graphemes. Which leads us to NFC normalization for a start. Upcase tables and wordlength tables are also needed.
fao_ 2y ago
> This is commercial only. Free and small is my safeclib Is it me or does this feel a bit weird? It seems like you're using the comments section here to self-advertise for exposure. I read it like — "businesses can't use this without paying the OP, however, if you're a business you can get 50% of the way there by using _my_ library, and you don't even have to pay me!". It comes off incredibly rude to try to undercut the OP like this.
- josephcsible 2y ago

Someone 2y ago

On https://railgunlabs.com/unicorn/manual/misra-compliance/ https://railgunlabs.com/unicorn/manual/misra-compliance/, I think you will want to fix a typo in 1.2 Required Compliant (verified by compiling with Clang's -pdentic flag) ^^^^^^^^ Or am I too pedantic?

hgs3 2y ago
This is the most ironic typo I've ever made. Thanks for the catch. I've corrected it.
- canucker2016 2y ago
  A couple more suggestions. List the platforms (& compilers) that you've tested on. Compare (pros/cons) against other Unicode libs (like others have done elsewhere in this thread, i.e. https://news.ycombinator.com/item?id=42424637 https://news.ycombinator.com/item?id=42424637 and https://news.ycombinator.com/item?id=42424638 https://news.ycombinator.com/item?id=42424638)
  - hgs3 2y ago
    Thanks for the suggestions. I think a comparison table would be useful, but I want to make sure I do it right since I'd be comparing my work to someone else's. As for the compilers, I’ve tested the library with GCC, Clang, and MSVC, and with the -pedantic flag like the GP mentioned. The library should build with any standard-compliant C99 compiler.
deleted 2y ago
[deleted]
- deleted 2y ago
  [deleted]

__turbobrew__ 2y ago

There is not much to show if I can’t read the source code.

hgs3 2y ago
You can download a prebuilt amalgamation from GitHub to see the amalgamated C code [1]. The GitHub repo contains the code that generates the amalgamation. [1] https://github.com/railgunlabs/unicorn/releases/ https://github.com/railgunlabs/unicorn/releases/

tocariimaa 2y ago

It uses a privative license if you're wondering.

biosboiii 2y ago

Since MISRA is targetted at Automotive, as a software dev in the automotive space I would suggest adding the note that this is able to run on POSIX compliant OSes like QNX :) If you would like to chat, hit me up.

deleted 2y ago

[deleted]

garganzol 2y ago

My comment is not directly related to the particular project which is impressive, but more to its presentation. If you go to the author's website, you will find neat to-the-point manuals and other useful information. This is what I call the real Web 3.0. Simple and to the point. Also the main company page is humorous in a good way, about the mad scientist etc.

hgs3 2y ago
I appreciate the kind words. When I was designing the website, I wanted to inject my personality into it. I wasn't sure how well the "playfulness" would go over, but I'm glad you enjoyed it.