4 ms·

It looks like Boa has Unicode tables compiled inside of itself: https://github.com/boa-dev/boa/tree/main/core/icu_provider https://github.com/boa-dev/boa/tree/m

by jerf 11mo ago

It looks like Boa has Unicode tables compiled inside of itself: https://github.com/boa-dev/boa/tree/main/core/icu_provider https://github.com/boa-dev/boa/tree/main/core/icu_provider Brimstone does not appear to. That covers the vast bulk of the difference. The ICU data is about 10.7MB in the source (boa/core/icu_provider) and may grow or shrink by some amount in the compiling. I'm not saying it's all the difference, just the bulk. There's a few reasons why svelte little executables with small library backings aren't possible anymore, and it isn't just ambient undefined "bloat". Unicode is a big one. Correct handling of unicode involves megabytes of tables and data that have to live somewhere, whether it's a linked library, compiled in, tables on disks, whatever. If a program touches text and it needs to handle it correctly rather than just passing it through, there's a minimum size for that now.

twoodfin 11mo ago
As well-defined as Unicode is, surprising that no one has tried to replace ICU with a better mousetrap. Not to say ICU isn’t a nice bit of engineering. The table builds in particular I recall having some great hacks.
- necovek 11mo ago
  POSIX systems actually have their own approach with "locales" and I it predates Unicode and ICU. Unfortunately, for a long time, POSIX system were uncommon on desktops, and most Unices do not provide a clean way to extend it from userland (though I believe GNU libc does).
rixed 11mo ago
I was currious to see what that data consisted of and aparently that's a lot of translations, like the name of all possible calendar formats in all possible languages, etc. This seems useless in the vast majority of use cases, including that of a JS interpreter. Looks to me like the typical output of a comitee that's looking too hard to extend its domain. Disclaimer: I never liked unicode specs.
- necovek 11mo ago
  Unicode is an attempt to encode the world's languages: there is not much to like or dislike about it, it only represents the reality. Sure, it has a number of weird details, butnif anything, it's due to the desire to simplify it (like Han unification or normal forms). Any language runtime wanting to provide date/time and string parsing functions needs access to the Unicode database (or something of comparable complexity and size). Saying "I don't like Unicode" is like saying "I don't like the linguistic diversity in the world": I mean sure, OK, but it's still there and it exists. Though note that date-time, currency, number, street etc. formatting is not "Unicode" even if provided by ICU: this is similarly defined by POSIX as "locales", anf GNU libc probably has the richest collection of locales outside of ICU. There are also many non-Unicode collation tables (think phonebook ordering that's different for each country and language): so no good sort() without those either.
  - deleted 11mo ago
    [deleted]
  - xeonmc 11mo ago
    Does that include emojis?
    - jcranmer 11mo ago
      Emojis are complicated from a font rendering perspective. But from a string processing perspective, they're generally going to be among the simplest characters: they don't have a lot of complex properties with a lot of variation between individual characters. Compare something like the basic Latin characters, where the mappings for precomposed characters are going to vary wildly from 'a' to 'b' to 'c', etc., whereas the list of precomposed characters for the emoji blocks amounts to "none."
      - necovek 11mo ago
        Agreed! FWIW, they are not even "complicated" from a font rendering perspective: they're simple non-combining characters and they are probably never used in ligatures either (though nothing really stops you; just like you can have locale-specific variants with locl tables). It's basically "draw whatever is in a font at this codepoint". Yes, if you want to call them out based on Unicode names, you need to have them in the database, and there are many of them, so a font needs to have them all, but really, they are the simplest of characters Unicode could have.
ambicapter 11mo ago
Unicode is everywhere though. You'd think there'd be much greater availability of those tables and data and that people wouldn't need to bundle it in their executables.
- nicoburns 11mo ago
  Unfortunately operating systems don't make the raw unicode data available (they only offer APIs to query it in various ways). Until they do we all have to ship it seperately.
  - pabs3 11mo ago
    Debian has a unicode-data package, so you can just depend on it.
  - lifthrasiir 11mo ago
    For some OSes like Windows, some relevant APIs can be indeed used to reconstruct those tables. I found that this is in fact viable for character encoding tables, only requiring a small table for fixes in most cases.
jancsika 11mo ago
If someone builds, say, a Korean website and needs sort(), does the ICU monolith handle 100% of the common cases? (Or substitute for Korean the language that has the largest amount of "stuff" in the ICU monolith.)
- adzm 11mo ago
  Yes, though it's easy to not use the ICU library properly or run into issues wrt normalization etc
HansHalverson 11mo ago
Brimstone does embed Unicode tables, but a smaller set than Boa embeds: https://github.com/Hans-Halverson/brimstone/tree/master/icu https://github.com/Hans-Halverson/brimstone/tree/master/icu. Brimstone does try to use the minimal set of Unicode data needed for the language itself. But I imagine much of the difference with Boa is because of Boa's support for the ECMA-402 Internationalization API (https://tc39.es/ecma402/ https://tc39.es/ecma402/).
- nekevss 11mo ago
  Yeah, the majority of the difference is from the Unicode data for Intl along with probably the timezone data for Temporal.
  - nicoburns 11mo ago
    Is it possible to build Boa without these APIs?
    - nekevss 11mo ago
      For the engine, the answer is yes, Intl and Temporal are feature flagged due to the dependencies. What I suspect they’re comparing above is the CLIs, which is completely different than the engine. I’d have to double check for the CLI. If I recall correctly, we include all features in the CLI by default.
miki123211 11mo ago
I just wish we could use system tables for that, instead of bloating every executable with their own outdated copy. I have no issue with my system using an extra 10mb for Ancient Egyptian capitalization to work correctly. Every single program including those rules is a lot more wasteful.

by jerf 11mo ago

twoodfin 11mo ago

As well-defined as Unicode is, surprising that no one has tried to replace ICU with a better mousetrap. Not to say ICU isn’t a nice bit of engineering. The table builds in particular I recall having some great hacks.

necovek 11mo ago
POSIX systems actually have their own approach with "locales" and I it predates Unicode and ICU. Unfortunately, for a long time, POSIX system were uncommon on desktops, and most Unices do not provide a clean way to extend it from userland (though I believe GNU libc does).

rixed 11mo ago

I was currious to see what that data consisted of and aparently that's a lot of translations, like the name of all possible calendar formats in all possible languages, etc. This seems useless in the vast majority of use cases, including that of a JS interpreter. Looks to me like the typical output of a comitee that's looking too hard to extend its domain. Disclaimer: I never liked unicode specs.

necovek 11mo ago
Unicode is an attempt to encode the world's languages: there is not much to like or dislike about it, it only represents the reality. Sure, it has a number of weird details, butnif anything, it's due to the desire to simplify it (like Han unification or normal forms). Any language runtime wanting to provide date/time and string parsing functions needs access to the Unicode database (or something of comparable complexity and size). Saying "I don't like Unicode" is like saying "I don't like the linguistic diversity in the world": I mean sure, OK, but it's still there and it exists. Though note that date-time, currency, number, street etc. formatting is not "Unicode" even if provided by ICU: this is similarly defined by POSIX as "locales", anf GNU libc probably has the richest collection of locales outside of ICU. There are also many non-Unicode collation tables (think phonebook ordering that's different for each country and language): so no good sort() without those either.
- deleted 11mo ago
  [deleted]
- xeonmc 11mo ago
  Does that include emojis?
  - jcranmer 11mo ago
    Emojis are complicated from a font rendering perspective. But from a string processing perspective, they're generally going to be among the simplest characters: they don't have a lot of complex properties with a lot of variation between individual characters. Compare something like the basic Latin characters, where the mappings for precomposed characters are going to vary wildly from 'a' to 'b' to 'c', etc., whereas the list of precomposed characters for the emoji blocks amounts to "none."
    - necovek 11mo ago
      Agreed! FWIW, they are not even "complicated" from a font rendering perspective: they're simple non-combining characters and they are probably never used in ligatures either (though nothing really stops you; just like you can have locale-specific variants with locl tables). It's basically "draw whatever is in a font at this codepoint". Yes, if you want to call them out based on Unicode names, you need to have them in the database, and there are many of them, so a font needs to have them all, but really, they are the simplest of characters Unicode could have.

ambicapter 11mo ago

Unicode is everywhere though. You'd think there'd be much greater availability of those tables and data and that people wouldn't need to bundle it in their executables.

nicoburns 11mo ago
Unfortunately operating systems don't make the raw unicode data available (they only offer APIs to query it in various ways). Until they do we all have to ship it seperately.
- pabs3 11mo ago
  Debian has a unicode-data package, so you can just depend on it.
- lifthrasiir 11mo ago
  For some OSes like Windows, some relevant APIs can be indeed used to reconstruct those tables. I found that this is in fact viable for character encoding tables, only requiring a small table for fixes in most cases.

jancsika 11mo ago

If someone builds, say, a Korean website and needs sort(), does the ICU monolith handle 100% of the common cases? (Or substitute for Korean the language that has the largest amount of "stuff" in the ICU monolith.)

adzm 11mo ago
Yes, though it's easy to not use the ICU library properly or run into issues wrt normalization etc

HansHalverson 11mo ago

Brimstone does embed Unicode tables, but a smaller set than Boa embeds: https://github.com/Hans-Halverson/brimstone/tree/master/icu https://github.com/Hans-Halverson/brimstone/tree/master/icu. Brimstone does try to use the minimal set of Unicode data needed for the language itself. But I imagine much of the difference with Boa is because of Boa's support for the ECMA-402 Internationalization API (https://tc39.es/ecma402/ https://tc39.es/ecma402/).

nekevss 11mo ago
Yeah, the majority of the difference is from the Unicode data for Intl along with probably the timezone data for Temporal.
- nicoburns 11mo ago
  Is it possible to build Boa without these APIs?
  - nekevss 11mo ago
    For the engine, the answer is yes, Intl and Temporal are feature flagged due to the dependencies. What I suspect they’re comparing above is the CLIs, which is completely different than the engine. I’d have to double check for the CLI. If I recall correctly, we include all features in the CLI by default.

miki123211 11mo ago

I just wish we could use system tables for that, instead of bloating every executable with their own outdated copy. I have no issue with my system using an extra 10mb for Ancient Egyptian capitalization to work correctly. Every single program including those rules is a lot more wasteful.