5 ms·

> N2888 - Exact-width Integer Types May Exceed (u)intmax_t This creates a problem. How do you portably printf() an integer type that you don’t (and can’t) kno

by teddyh 4y ago

> N2888 - Exact-width Integer Types May Exceed (u)intmax_t This creates a problem. How do you portably printf() an integer type that you don’t (and can’t) know the size of, like, say, a uid_t? Before, when intmax_t was guaranteed to be the largest type, you could reliably and portably cast a uid_t userid to an intmax_t and printf() that: printf(PRIdMAX "\n", (intmax_t)userid); But now, when any type might be larger than an intmax_t? What do you do? (The same problem exists when reading an integer string; how do you read, say, a port number, when you don’t know what type in_port_t is? Previously, you could call strtoimax() on the string, detect overflow, and then cast the resulting intmax_t to in_port_t and finally compare for equality to make sure it wasn’t truncated.) > There is active work in this area to allow us to transition to a better ABI and let these two types live up to their promises Yes please. Introducing new wider integer types in your CPU, but pretending you haven’t created a new architecture with an associated new ABI, is silly.

formerly_proven 4y ago
> But now, when any type might be larger than an uintmax_t? What do you do? Complain to POSIX so they add printf macro defines for all their implementation-sized integer types by the year 2100.
- teddyh 4y ago
  For printf(), that would work. For sscanf(), POSIX could add macros there, too, but sscanf() does not detect integer overflow. That’s why the only previous solution was strtoimax() which does detect overflow.
  - rwmj 4y ago
    scanf not detecting integer overflow is arguably another problem. It's usually the easiest way to parse numbers out of simple strings, but the lack of overflow detection is a problem. (Actually we use sscanf for this and ignore the overflow problem, as long as it doesn't cause any security or correctness concern.) IIRC glibc actually does overflow detection, so there's not anything inherently problematic with adding it.
orlp 4y ago
To the first problem you can use a classic mod 10 divide by 10 loop to convert into a string digit by digit. Similarly for the second problem you can read digit by digit in reverse order and create the number using x = 10x + d while detecting overflow. I'm not saying that this is a great thing to need to do, but the mentioned problems aren't particularly fundamental and the solutions are CS101 stuff.
- teddyh 4y ago
  You may be right that it’s the best solution. It’s horrible to contemplate, but it might be so.
  - orlp 4y ago
    Oh I agree, it's bad.
  - jcranmer 4y ago
    The best solution is to say that uid_t et al are probably no larger than uint64_t, and introducing a static_assert that makes sure that the compiler will fail if it's not. Sure, someone might come up with an architecture where that's not true. But even in an intmax_t-is-really-the-largest-integer-type world [which hasn't been true for the major C compilers for the past decade or so!], I can almost assuredly come up with some compilation modes that still conform with whichever C version you use that still causes your code to fail horribly. I don't think there's any architecture where uid_t et al would exceed uint64_t, and until such an architecture comes to exist, it's quite frankly not worth caring about.
- csmpltn 4y ago
  > "To the first problem you can use a classic mod 10 divide by 10 loop to convert into a string digit by digit." But the "physical" size of the number as represented in memory can change across platforms, no?
pdw 4y ago
I think it's best to regard intmax_t as a failed experiment. E.g. GCC has long supported __int128, clang has _ExtInt, other compilers have similar extensions, but they refuse to increase intmax_t due to ABI concerns.
- qalmakka 4y ago
  If people use intmax_t and then they expect the ABI to stay stable, than they are at fault, not libc, LLVM or GCC for changing it when the actual "max int" changes.
  - teddyh 4y ago
    I feel the fault instead lies in the operating system for wanting to run old binaries so much that the operating system pretends that the system architecture (and associated ABI) has not changed, even when there exists larger integers than what the previous architecture had set as intmax_t.
    - afiori 4y ago
      The problem is not old binaries, it is old code that assumes an upper bound to the size of intmax_t.
      - teddyh 4y ago
        I’m not sure what you mean. I agree that the problem is not the binaries, but I believe the problem to be the operating system. The source code may not have assumed a specific upper size limit on intmax_t, but the binaries were compiled in good faith to a specific machine architecture, which had a specific width on intmax_t, namely the largest integer type which that architecture could support. If then new CPU models come out with larger integer types, but the operating systems still runs that same binary straight up (without any emulation layer), this exposes the old binary to larger integer types, breaking the binary. The binary is not at fault, it’s the operating system which committed the sin of running the binary on what is effectively a different architecture without any considerations on what architecture the binary was actually compiled for.
csmpltn 4y ago
> "How do you portably printf() an integer type that you don’t (and can’t) know the size of" "Portably" across what? You need to narrow down the problem space and better define it. If it's portably across a myriad of processor architectures, hardware profiles and operating systems then you would need to take the intermediate representation route (like Java). There's very little else that can be done in the face of infinite, non-compatible ABIs at every layer.
- teddyh 4y ago
  I mean portable over all platforms which claims to support standard C. > you would need to take the intermediate representation route (like Java). There's very little else that can be done in the face of infinite, non-compatible ABIs at every layer. No, I don’t think so. There’s an obvious solution, dictated by the architectual choices made by C (i.e. every architecture which introduces a new larger integer type, by necessity redefines intmax_t, and creates a new ABI), but the community (for reasons of their own) did not want to do it that way, and instead broke the promise of intmax_t always being the largest type.
Aardwolf 4y ago
Maybe C just needs to provide a way to print/format any built in type without specifying it. They got stdc_popcount to work with any integer type (looks like function overloading in C through macros), so can same be done for printing integers?
- teddyh 4y ago
  Don’t forget the other side of the coin; we also need to read a string into any integer type (while also detecting overflows).
__phantomderp 4y ago
I mentioned this in another article of mine. Robert Seacord closed the loop on "how do I print numbers the non-shit way?" in C23 with N2680: https://thephd.dev/c-the-improvements-june-september-virtual-c-meeting#n2680---specific-width-length-modifiers https://thephd.dev/c-the-improvements-june-september-virtual... https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2680.pdf https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2680.pdf
- teddyh 4y ago
  Great, thanks! But that only solves printing. How do I do the reverse operation? I.e. parsing a string to an integer type of an unknown size, while checking for overflow?
  - __phantomderp 4y ago
    ¯\_(ツ)_/¯ Someone's got to write a paper to get it in, hopefully based on some sort of existing practice.
  - Someone 4y ago
    I think this would work: first, determine the range. For unsigned variants, we have whatever_max = (whatever)(-1), if it isn’t already defined. That makes parsing the unsigned variant relatively simple. Undefined behavior would be end-of-game, but unsigned addition and multiplication will not invoke that (if you need he,p, look at your favorite multiple-precision library for ideas about detecting wraparound) For parsing signed integers, derive the min and max values of the signed variant from the max value of the unsigned one ((whatever_max - 1) / 2 or something like that Then parse your input to the unsigned variant first, then check whether it fits in the unsigned version.

by teddyh 4y ago

formerly_proven 4y ago

> But now, when any type might be larger than an uintmax_t? What do you do? Complain to POSIX so they add printf macro defines for all their implementation-sized integer types by the year 2100.

teddyh 4y ago
For printf(), that would work. For sscanf(), POSIX could add macros there, too, but sscanf() does not detect integer overflow. That’s why the only previous solution was strtoimax() which does detect overflow.
- rwmj 4y ago
  scanf not detecting integer overflow is arguably another problem. It's usually the easiest way to parse numbers out of simple strings, but the lack of overflow detection is a problem. (Actually we use sscanf for this and ignore the overflow problem, as long as it doesn't cause any security or correctness concern.) IIRC glibc actually does overflow detection, so there's not anything inherently problematic with adding it.

orlp 4y ago

To the first problem you can use a classic mod 10 divide by 10 loop to convert into a string digit by digit. Similarly for the second problem you can read digit by digit in reverse order and create the number using x = 10x + d while detecting overflow. I'm not saying that this is a great thing to need to do, but the mentioned problems aren't particularly fundamental and the solutions are CS101 stuff.

teddyh 4y ago
You may be right that it’s the best solution. It’s horrible to contemplate, but it might be so.
- orlp 4y ago
  Oh I agree, it's bad.
- jcranmer 4y ago
  The best solution is to say that uid_t et al are probably no larger than uint64_t, and introducing a static_assert that makes sure that the compiler will fail if it's not. Sure, someone might come up with an architecture where that's not true. But even in an intmax_t-is-really-the-largest-integer-type world [which hasn't been true for the major C compilers for the past decade or so!], I can almost assuredly come up with some compilation modes that still conform with whichever C version you use that still causes your code to fail horribly. I don't think there's any architecture where uid_t et al would exceed uint64_t, and until such an architecture comes to exist, it's quite frankly not worth caring about.
csmpltn 4y ago
> "To the first problem you can use a classic mod 10 divide by 10 loop to convert into a string digit by digit." But the "physical" size of the number as represented in memory can change across platforms, no?

pdw 4y ago

I think it's best to regard intmax_t as a failed experiment. E.g. GCC has long supported __int128, clang has _ExtInt, other compilers have similar extensions, but they refuse to increase intmax_t due to ABI concerns.

qalmakka 4y ago
If people use intmax_t and then they expect the ABI to stay stable, than they are at fault, not libc, LLVM or GCC for changing it when the actual "max int" changes.
- teddyh 4y ago
  I feel the fault instead lies in the operating system for wanting to run old binaries so much that the operating system pretends that the system architecture (and associated ABI) has not changed, even when there exists larger integers than what the previous architecture had set as intmax_t.
  - afiori 4y ago
    The problem is not old binaries, it is old code that assumes an upper bound to the size of intmax_t.
    - teddyh 4y ago
      I’m not sure what you mean. I agree that the problem is not the binaries, but I believe the problem to be the operating system. The source code may not have assumed a specific upper size limit on intmax_t, but the binaries were compiled in good faith to a specific machine architecture, which had a specific width on intmax_t, namely the largest integer type which that architecture could support. If then new CPU models come out with larger integer types, but the operating systems still runs that same binary straight up (without any emulation layer), this exposes the old binary to larger integer types, breaking the binary. The binary is not at fault, it’s the operating system which committed the sin of running the binary on what is effectively a different architecture without any considerations on what architecture the binary was actually compiled for.

csmpltn 4y ago

> "How do you portably printf() an integer type that you don’t (and can’t) know the size of" "Portably" across what? You need to narrow down the problem space and better define it. If it's portably across a myriad of processor architectures, hardware profiles and operating systems then you would need to take the intermediate representation route (like Java). There's very little else that can be done in the face of infinite, non-compatible ABIs at every layer.

teddyh 4y ago
I mean portable over all platforms which claims to support standard C. > you would need to take the intermediate representation route (like Java). There's very little else that can be done in the face of infinite, non-compatible ABIs at every layer. No, I don’t think so. There’s an obvious solution, dictated by the architectual choices made by C (i.e. every architecture which introduces a new larger integer type, by necessity redefines intmax_t, and creates a new ABI), but the community (for reasons of their own) did not want to do it that way, and instead broke the promise of intmax_t always being the largest type.

Aardwolf 4y ago

Maybe C just needs to provide a way to print/format any built in type without specifying it. They got stdc_popcount to work with any integer type (looks like function overloading in C through macros), so can same be done for printing integers?

teddyh 4y ago
Don’t forget the other side of the coin; we also need to read a string into any integer type (while also detecting overflows).

__phantomderp 4y ago

I mentioned this in another article of mine. Robert Seacord closed the loop on "how do I print numbers the non-shit way?" in C23 with N2680: https://thephd.dev/c-the-improvements-june-september-virtual-c-meeting#n2680---specific-width-length-modifiers https://thephd.dev/c-the-improvements-june-september-virtual... https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2680.pdf https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2680.pdf

teddyh 4y ago
Great, thanks! But that only solves printing. How do I do the reverse operation? I.e. parsing a string to an integer type of an unknown size, while checking for overflow?
- __phantomderp 4y ago
  ¯\_(ツ)_/¯ Someone's got to write a paper to get it in, hopefully based on some sort of existing practice.
- Someone 4y ago
  I think this would work: first, determine the range. For unsigned variants, we have whatever_max = (whatever)(-1), if it isn’t already defined. That makes parsing the unsigned variant relatively simple. Undefined behavior would be end-of-game, but unsigned addition and multiplication will not invoke that (if you need he,p, look at your favorite multiple-precision library for ideas about detecting wraparound) For parsing signed integers, derive the min and max values of the signed variant from the max value of the unsigned one ((whatever_max - 1) / 2 or something like that Then parse your input to the unsigned variant first, then check whether it fits in the unsigned version.