6 ms·

why is locale stuff even in sprintf? What things get localized? Dates? Line endings? I'm probably dumb but one of things that bugs me with various libraries is

why is locale stuff even in sprintf? What things get localized? Dates? Line endings? I'm probably dumb but one of things that bugs me with various libraries is when someone has made the decision to do something high level at a low level. For example, localizing inside sprintf, An another example might be an unzip library (read a zip file). Ideally the library should be small IMO. The simplest might you you pass it a bucket of bytes. If you want to make it flexible then you pass it some abstract interface (or 1-2 functions + void* userdata) so you can supply a "read(byteOffset, length)". You can then provide, outside of the library, streamed files, stream networking, etc... But, bad libraries (bad IMO) will instead provide like 12 overrides "unzip(void* bytes), unzip(const char* filename), unzip(socket), unzip(url)" and end up including the world in their library. This kind of "try to do everything" is extremely common in npm libraries :( I don't need your library to include command line parsing! If you want to make a tool, make a library, then make a separate tool that uses that library. Keep the 2 separated so users of the library don't need dependencies that only the tool needs. (probably the most common npm example but there are lots of others) Really surprised something as low-level as sprintf needs locale. Even streams I'd expect maybe a Date object would but not the stream itself.

kazinator 5y ago
locale stuff is even in strtod, so you can read numbers like 3,14159 in locales where comma is the decimal point.
cyral 5y ago
I get your point about bloated NPM libraries but I think it's also ironic that NPM is the only registry with some of the _smallest_ packages which are equally as annoying. left-pad, ansi-yellow, ansi-red (yes, packages for a single console color with over 200k monthly downloads), is-odd, is-even (which of course, depends on is-odd and inverts it), etc.
- haggy102 5y ago
  Don't forget about "is-thirteen" and its inverse lol https://www.npmjs.com/package/is-thirteen https://www.npmjs.com/package/is-thirteen https://www.npmjs.com/package/is-not-thirteen https://www.npmjs.com/package/is-not-thirteen
- esprehn 5y ago
  To be fair a lot of those super tiny packages like the ones you listed were created by the same person. They have over 800 repositories on GitHub and hundreds of (probably too) small published npm packages. There's been discussion of that person and their possible motivations on HN before.
MaulingMonkey 5y ago
> why is locale stuff even in sprintf? What things get localized Decimal places in floating point numbers is the primary standard thing (some locales use `.` as a decimal seperator and `,` as a thousands seperator... some locales swap interpretations!) Nonstandard extensions may also do things like accept unicode strings, wide unicode strings, etc. which may need to be re-encoded to whatever the ambient locale-specified narrow encoding is (UTF16 => windows-1251?). In the modern era of sprintf being exiled to some low level internal thing, I agree it makes little sense - causing more bugs than it fixes - but in the days of using it for a lot of heavy lifting of user-facing data, it was an understandable target for this kind of treatment.
- butlerm 5y ago
  The overhead is completely unnecessary in far too many cases. So what they should have done is kept sprintf to its traditional behavior and invented a new, slower alternative for those fairly unusual cases that need locale sensitivity. Locales tend to be all or nothing and extremely inefficient to make general use of anyway because too many things are controlled by the same setting. Standards committees should not carelessly sit down and devise things that destroy performance by a factor of ten or more, especially when it is entirely unnecessary and where they do not keep or provide simple, fast alternatives for the most common cases where the format is known and has nothing to do with a preferred locale. What they did instead was a massive energy and time wasting imposition of unhelpful semantics on functions that did not have or need any such thing instead of a tailored let's only do this in special cases with special functions approach.
  - jcelerier 5y ago
    C++ has thankfully locale- independent functions now, std::from_chars, std::to_chars.. but it took way too many years
    - gustavorg 5y ago
      I am very curious to know why you guys are forgetting std::to_string. It is available since c++11. This and then str.c_ctr() should make the trick in most cases.
WalterBright 5y ago
These are known as "kitchen sink" interfaces. Unfortunately, it takes experience for people to realize that they're not such a good idea. A better design is the component approach. As you suggest, a zip library should not be doing file I/O.
- Spivak 5y ago
  For all the memes Java’s IO BufferedReader nonsense is really really composable because of this design.
ReactiveJelly 5y ago
The worst libraries _only_ accept a file path or URL for unzip. I've seen audio libraries where you can only provide a file path for playback. Guess you're screwed if you wanted to get a Vorbis file out of a tarball, decode it silently faster than real-time, and send it over the network...
- loudmax 5y ago
  Selective untar into a FIFO pipe. Not optimal in this case but you can work around the limitation with some extra work. I'd rather do this than expose some 0-day due to odd combination of rarely used libraries.
  - gpderetta 5y ago
    Or memfd + mmap + /proc/<pid>/fd for those libraries that unconditionally want to seek.
    - saagarjha 5y ago
      This also works for annoying APIs like execve ;)
- alar44 5y ago
  That's likely the use case for 99% of people who zip things though or play audio. Your corner case is exactly what simple libraries should avoid.
  - pie_flavor 5y ago
    The only thing they're going to do is open it as an ifstream. Zero value is gained over taking an istream instead; initializing it is one bit of extra code for the user, in exchange for being able to pull in data from anywhere. Embedding it in the binary might be rare, but how about stdin? Or for that matter the Internet?
kingcharles 5y ago
Can you output Eastern-Arabic numerals (e.g. ٠١٢٣٤٥٦٧٨٩١٠) with sprintf? My gf sometimes uses them and its confusing as fuck for me. Especially as the numbers are then written RtL.
- microtherion 5y ago
  I believe the numbers are the one thing in Arabic that are NOT written RTL.
  - gumby 5y ago
    They are written RTL, it's simply that in Arabic numbers are written little endian. In case this comes off as sarcastic, it is not intended so.
    - thayne 5y ago
      In fact, the fact that arabic is written rtl is why our numbers are big endian. Because the European number system is based on the arabic system.
      - gumby 5y ago
        I don’t think that is true. Europe got decimal digits with their own characters via the arabs, true, but we use shapes derived from the original source, the Devanagari digits, not the shapes the Arabs developed due to their writing system/technology. South Asian languages other than Urdu are LTR. You can see the parallels here (clipped from Wikipedia): ० 0 १ 1 २ 2 ३ 3 ४ 4 ५ 5 ६ 6 ७ 7 ८ 8 ९ 9
    - microtherion 5y ago
      Ah, I get your point. They are _pronounced_ little endian, and consequently written that way RTL, which works out to big endian LTR, of course.
anonymoushn 5y ago
Recently I tried to make a Zig program that parses and formats some format containing numbers quickly, and I discovered that (after getting rid of all the places where one read function would repeatedly call another function that handles arbitrary-sized reads asking for 1 byte) I was spending a lot of time parsing UTF-8 strings to determine their length. This was surprising, because I didn't write any code that's concerned with that sort of thing! It turns out that any time you ask for a formatted value to be padded, the stdlib fmt routines will helpfully format the value, then determine its number of UTF-8 codepoints, then pad it with whatever padding byte you supplied until the number of padding bytes plus the number of UTF-8 codepoints in the formatted value is at least the width you asked for. It will do this even if you were trying to pad a number with zeroes. So now we're compiling against our own stdlib with one line changed to not parse UTF-8 strings when you ask it to format numbers. Luckily none of this stuff cares about locales.
- pjmlp 5y ago
  Lucky you, in Europe I have plenty of fun working in projects where the files being parsed for ETL are written in the local language, or not, while the host OS might be English, or not.
haberman 5y ago
The worst part is that there is no way to turn off the locale sensitivity. This is bad for not only performance, but correctness. If you are writing a JSON serializer and you use sprintf() to format float/double values, someone can make you produce incorrect output by setting the locale to something that uses "," as a decimal separator. And there is no way to turn this off! This is one moment where the difference between an application language and a system language seems clear. An application language could maybe assume that string formatting is for showing to a user. A systems language should assume that you are implementing protocols, and that mere user preference should not change your output.
- codemac 5y ago
  My joke would be there's a difference between a programmer's laptop, and an application user's laptop. Install linux on your laptop, seeing the source code is worth it.
- tedunangst 5y ago
  The old way to turn it off would be setlocale() in your library before and after sprintf, but this is a disaster for threads, and so now there's the _l functions.
  - deleted 5y ago
    [deleted]
  - nwellnhof 5y ago
    glibc only supports a few _l functions. It doesn't support snprintf_l, for example. The solution here is uselocale.
- dhosek 5y ago
  Kind of like a bug in PHP where identifiers with i in them didn't work properly if the locale was Turkish since PHP managed its case insensitivity by uppercasing the identifiers, but used a locale-dependent format in which, if the locale specified Turkish, would uppercase i to İ rather than I.
- 5y ago
makeitdouble 5y ago
> bugs me with various libraries is when someone has made the decision to do something high level at a low level. For example, localizing inside sprintf I'm more surprised that sprintf is seen as low level. It formats and outputs strings, just by that token the problem space is enormous. Localization is just a small part. People are fast to forget the good old days where sending specific user strings to people would crash their browser or os. "Plain" text is not simple.
diego_sandoval 5y ago
Maybe I'm dumb, but locales make no sense to me. Why should the country I'm in and the language that I'm currently speaking determine if I want periods or commas as thousands separators? Or if I want dates to be presented in one of dozens of insane formats instead of RFC 3339/ISO 8061?
- hobs 5y ago
  Because that's what you grew up, learned, and have used forever? Let's ask it another way - why would the country you are in and the language you are currently speaking NOT decide those things (which are defined by said languages.) What are we doing right now to each other besides banging out some symbols that have a shared meaning?
  - diego_sandoval 5y ago
    I grew up with dd/mm/yy, comma as decimal separators, dot as thousands separator, but: 1. The fact that I grew up with those formats holds little weight for me. At this point, most of the literature I read and content I consume doesn't come from my own country. And why should it? when I have access to books, movies, websites from all over the world, and my country makes up for a tiny fraction of that. With the massification of international remote work this tendency will only increase. 2. Even when working with people from my own country, I can agree with them to use formats different from the ones we grew up with. In fact, everytime we type a floating point literal in any programming language, we do it without following the rules we grew up with. 3. In the 21st century, in the context of globalization, massification of the Internet and widespread access to computing, to keep doing these things differently by country/language makes no sense, specially when there are already good international standards that we can follow. It's not that hard to learn, either. We even accept English as the de facto language of programming and software development, and that's a full natural language that we have to spend years learning. 4. In my opinion, yyyy-mm-dd and dot as decimal separator are simply better for practical reasons. However, if we collectively decide that other formats are the "international standard" then I'd follow whatever rule we decide, as long as it's not too bad. Can you imagine if instead of using SI units, each country used their own special units of measurement? We are definitely better off with SI, standarization makes everything so much easier. I can talk to someone from Japan about meters and kilograms and they will understand without any ambiguity.

kazinator 5y ago

locale stuff is even in strtod, so you can read numbers like 3,14159 in locales where comma is the decimal point.

cyral 5y ago

I get your point about bloated NPM libraries but I think it's also ironic that NPM is the only registry with some of the _smallest_ packages which are equally as annoying. left-pad, ansi-yellow, ansi-red (yes, packages for a single console color with over 200k monthly downloads), is-odd, is-even (which of course, depends on is-odd and inverts it), etc.

haggy102 5y ago
Don't forget about "is-thirteen" and its inverse lol https://www.npmjs.com/package/is-thirteen https://www.npmjs.com/package/is-thirteen https://www.npmjs.com/package/is-not-thirteen https://www.npmjs.com/package/is-not-thirteen
esprehn 5y ago
To be fair a lot of those super tiny packages like the ones you listed were created by the same person. They have over 800 repositories on GitHub and hundreds of (probably too) small published npm packages. There's been discussion of that person and their possible motivations on HN before.

MaulingMonkey 5y ago

> why is locale stuff even in sprintf? What things get localized Decimal places in floating point numbers is the primary standard thing (some locales use `.` as a decimal seperator and `,` as a thousands seperator... some locales swap interpretations!) Nonstandard extensions may also do things like accept unicode strings, wide unicode strings, etc. which may need to be re-encoded to whatever the ambient locale-specified narrow encoding is (UTF16 => windows-1251?). In the modern era of sprintf being exiled to some low level internal thing, I agree it makes little sense - causing more bugs than it fixes - but in the days of using it for a lot of heavy lifting of user-facing data, it was an understandable target for this kind of treatment.

butlerm 5y ago
The overhead is completely unnecessary in far too many cases. So what they should have done is kept sprintf to its traditional behavior and invented a new, slower alternative for those fairly unusual cases that need locale sensitivity. Locales tend to be all or nothing and extremely inefficient to make general use of anyway because too many things are controlled by the same setting. Standards committees should not carelessly sit down and devise things that destroy performance by a factor of ten or more, especially when it is entirely unnecessary and where they do not keep or provide simple, fast alternatives for the most common cases where the format is known and has nothing to do with a preferred locale. What they did instead was a massive energy and time wasting imposition of unhelpful semantics on functions that did not have or need any such thing instead of a tailored let's only do this in special cases with special functions approach.
- jcelerier 5y ago
  C++ has thankfully locale- independent functions now, std::from_chars, std::to_chars.. but it took way too many years
  - gustavorg 5y ago
    I am very curious to know why you guys are forgetting std::to_string. It is available since c++11. This and then str.c_ctr() should make the trick in most cases.

WalterBright 5y ago

These are known as "kitchen sink" interfaces. Unfortunately, it takes experience for people to realize that they're not such a good idea. A better design is the component approach. As you suggest, a zip library should not be doing file I/O.

Spivak 5y ago
For all the memes Java’s IO BufferedReader nonsense is really really composable because of this design.

ReactiveJelly 5y ago

The worst libraries _only_ accept a file path or URL for unzip. I've seen audio libraries where you can only provide a file path for playback. Guess you're screwed if you wanted to get a Vorbis file out of a tarball, decode it silently faster than real-time, and send it over the network...

loudmax 5y ago
Selective untar into a FIFO pipe. Not optimal in this case but you can work around the limitation with some extra work. I'd rather do this than expose some 0-day due to odd combination of rarely used libraries.
- gpderetta 5y ago
  Or memfd + mmap + /proc/<pid>/fd for those libraries that unconditionally want to seek.
  - saagarjha 5y ago
    This also works for annoying APIs like execve ;)
alar44 5y ago
That's likely the use case for 99% of people who zip things though or play audio. Your corner case is exactly what simple libraries should avoid.
- pie_flavor 5y ago
  The only thing they're going to do is open it as an ifstream. Zero value is gained over taking an istream instead; initializing it is one bit of extra code for the user, in exchange for being able to pull in data from anywhere. Embedding it in the binary might be rare, but how about stdin? Or for that matter the Internet?

kingcharles 5y ago

Can you output Eastern-Arabic numerals (e.g. ٠١٢٣٤٥٦٧٨٩١٠) with sprintf? My gf sometimes uses them and its confusing as fuck for me. Especially as the numbers are then written RtL.

microtherion 5y ago
I believe the numbers are the one thing in Arabic that are NOT written RTL.
- gumby 5y ago
  They are written RTL, it's simply that in Arabic numbers are written little endian. In case this comes off as sarcastic, it is not intended so.
  - thayne 5y ago
    In fact, the fact that arabic is written rtl is why our numbers are big endian. Because the European number system is based on the arabic system.
    - gumby 5y ago
      I don’t think that is true. Europe got decimal digits with their own characters via the arabs, true, but we use shapes derived from the original source, the Devanagari digits, not the shapes the Arabs developed due to their writing system/technology. South Asian languages other than Urdu are LTR. You can see the parallels here (clipped from Wikipedia): ० 0 १ 1 २ 2 ३ 3 ४ 4 ५ 5 ६ 6 ७ 7 ८ 8 ९ 9
  - microtherion 5y ago
    Ah, I get your point. They are _pronounced_ little endian, and consequently written that way RTL, which works out to big endian LTR, of course.

anonymoushn 5y ago

Recently I tried to make a Zig program that parses and formats some format containing numbers quickly, and I discovered that (after getting rid of all the places where one read function would repeatedly call another function that handles arbitrary-sized reads asking for 1 byte) I was spending a lot of time parsing UTF-8 strings to determine their length. This was surprising, because I didn't write any code that's concerned with that sort of thing! It turns out that any time you ask for a formatted value to be padded, the stdlib fmt routines will helpfully format the value, then determine its number of UTF-8 codepoints, then pad it with whatever padding byte you supplied until the number of padding bytes plus the number of UTF-8 codepoints in the formatted value is at least the width you asked for. It will do this even if you were trying to pad a number with zeroes. So now we're compiling against our own stdlib with one line changed to not parse UTF-8 strings when you ask it to format numbers. Luckily none of this stuff cares about locales.

pjmlp 5y ago
Lucky you, in Europe I have plenty of fun working in projects where the files being parsed for ETL are written in the local language, or not, while the host OS might be English, or not.

haberman 5y ago

The worst part is that there is no way to turn off the locale sensitivity. This is bad for not only performance, but correctness. If you are writing a JSON serializer and you use sprintf() to format float/double values, someone can make you produce incorrect output by setting the locale to something that uses "," as a decimal separator. And there is no way to turn this off! This is one moment where the difference between an application language and a system language seems clear. An application language could maybe assume that string formatting is for showing to a user. A systems language should assume that you are implementing protocols, and that mere user preference should not change your output.

codemac 5y ago
My joke would be there's a difference between a programmer's laptop, and an application user's laptop. Install linux on your laptop, seeing the source code is worth it.
tedunangst 5y ago
The old way to turn it off would be setlocale() in your library before and after sprintf, but this is a disaster for threads, and so now there's the _l functions.
- deleted 5y ago
  [deleted]
- nwellnhof 5y ago
  glibc only supports a few _l functions. It doesn't support snprintf_l, for example. The solution here is uselocale.
dhosek 5y ago
Kind of like a bug in PHP where identifiers with i in them didn't work properly if the locale was Turkish since PHP managed its case insensitivity by uppercasing the identifiers, but used a locale-dependent format in which, if the locale specified Turkish, would uppercase i to İ rather than I.
5y ago

makeitdouble 5y ago

> bugs me with various libraries is when someone has made the decision to do something high level at a low level. For example, localizing inside sprintf I'm more surprised that sprintf is seen as low level. It formats and outputs strings, just by that token the problem space is enormous. Localization is just a small part. People are fast to forget the good old days where sending specific user strings to people would crash their browser or os. "Plain" text is not simple.

diego_sandoval 5y ago

Maybe I'm dumb, but locales make no sense to me. Why should the country I'm in and the language that I'm currently speaking determine if I want periods or commas as thousands separators? Or if I want dates to be presented in one of dozens of insane formats instead of RFC 3339/ISO 8061?

hobs 5y ago
Because that's what you grew up, learned, and have used forever? Let's ask it another way - why would the country you are in and the language you are currently speaking NOT decide those things (which are defined by said languages.) What are we doing right now to each other besides banging out some symbols that have a shared meaning?
- diego_sandoval 5y ago
  I grew up with dd/mm/yy, comma as decimal separators, dot as thousands separator, but: 1. The fact that I grew up with those formats holds little weight for me. At this point, most of the literature I read and content I consume doesn't come from my own country. And why should it? when I have access to books, movies, websites from all over the world, and my country makes up for a tiny fraction of that. With the massification of international remote work this tendency will only increase. 2. Even when working with people from my own country, I can agree with them to use formats different from the ones we grew up with. In fact, everytime we type a floating point literal in any programming language, we do it without following the rules we grew up with. 3. In the 21st century, in the context of globalization, massification of the Internet and widespread access to computing, to keep doing these things differently by country/language makes no sense, specially when there are already good international standards that we can follow. It's not that hard to learn, either. We even accept English as the de facto language of programming and software development, and that's a full natural language that we have to spend years learning. 4. In my opinion, yyyy-mm-dd and dot as decimal separator are simply better for practical reasons. However, if we collectively decide that other formats are the "international standard" then I'd follow whatever rule we decide, as long as it's not too bad. Can you imagine if instead of using SI units, each country used their own special units of measurement? We are definitely better off with SI, standarization makes everything so much easier. I can talk to someone from Japan about meters and kilograms and they will understand without any ambiguity.