6 ms·

Beautiful idea: a number representation where the distribution of accuracy aligns with the distribution of typical use. >Real numbers can’t be perfectly repres

Beautiful idea: a number representation where the distribution of accuracy aligns with the distribution of typical use. >Real numbers can’t be perfectly represented in hardware simply because there are infinitely many of them. To fit into a designated number of bits, many real numbers have to be rounded. The advantage of posits comes from the way the numbers they represent exactly are distributed along the number line. In the middle of the number line, around 1 and -1, there are more posit representations than floating point. And at the wings, going out to large negative and positive numbers, posit accuracy falls off more gracefully than floating point. >“It’s a better match for the natural distribution of numbers in a calculation,” says Gustafson. “It’s the right dynamic range, and it’s the right accuracy where you need more accuracy. There’s an awful lot of bit patterns in floating-point arithmetic no one ever uses. And that’s waste.”

boberoni 4y ago
I like this idea of choosing a representation for numbers based on the use case. Like, "ints" and "floats" but even more specialized. For example, what would be the most efficient binary representation of probabilities values between 0 and 1? In the future, I can imagine hardware + software that is specialized for examples like this.
- dr_dshiv 4y ago
  Makes me wonder about quantum computing possibilities.
- js8 4y ago
  I feel like probabilities are best stored as inverse logistic function, because the closer the probability is to 0 or 1, the less you care about significant digits, because the events are going (not) to happen anyway. (That's why I am proponent of LNS.)
  - jbay808 4y ago
    I'd agree with this suggestion. Furthermore this does not need a new number format, as IEEE floating point is already pretty ideally distributed. https://en.wikipedia.org/wiki/Logit https://en.wikipedia.org/wiki/Logit
  - planede 4y ago
    I feel that it's actually the opposite. There is a world of difference between 0.0...01 and 0.0...001 , where ... represents the same number of 0s in the two cases. Same applies for the other end. edit: Do I really want more resolution between 0.41 and 0.42 than between 0.01 and 0.02?
    - daniel-cussen 4y ago
      You're dead on. Metals are sold by how fine they are, so .999%, .9995% (should end in a 7 numeral not a 5, whole industry is fucking up right there, but at any rate) .9999%, more nines more purity better experimental results. When they talk about 4 nines or 5 nines the actual measurement is of the f(x)= - log(1 - x). That is the purity. Negative of the log of the complement. And that's why 4.5 nines should be .99997% not fucking .99995%, so stupid (it's almost exactly .99997%). Mining engineers, come on! Such impressive degrees! PhD's all over the place! Professors even! I'm sorry, don't mind me with my math. Some experiments like with liquid crystals early on required purities that what chemical company was it, Merck I think, around the year 1900, complained like they felt insulted. Told the inventor of liquid crystals because it was an absurd amount of purity required to get them actually working. But look at them go! Right in front of your very eyes!
  - teo_zero 4y ago
    Isn't it the other way out? The logistic curve becomes denser close to 0 and 1. Which makes sense: you will want to tell apart 1 defect per million from 0.01 dpm, and 5-sigma process (99.977%) from 6-sigma (99.99966%), much more than tell apart 30% from 30.001%.
Animats 4y ago
Ah. So it's because the machine learning people mostly work in the -1 .. 1 range, but it's possible for their numbers to go outside that range. So they need a compact representation which uses most of the possible values for the range of interest. If you're down to 8 bit numbers, something like this makes sense for custom machine learning hardware. I wonder if it has graphics applications for high dynamic range images. Probably not.
- threatripper 4y ago
  The machine learning space probably has the easiest time to adopt a new number format. The players are big enough to define their own silicon and training/inference don't even need the same number format. As long as it works on their new specialized training hardware it's good to go. The real challenge is probably in the power efficiency. As far as I know right now power consumption is the biggest cost factor when training new models. Everything else seems like a minor trade-off but when accounting for the power consumption there is real money involved.
  - totoglazer 4y ago
    Hardware is not cheap either at this scale. Reallly rough math follows, please anyone correct me if I err. A100 cards are >$10,000 each. A node is typically 8 and a cluster might be 64 A100s. So about a million in hardware for the medium cluster of 64 A100s. (Cross checking, a machine from lambda labs is apparently ~175k for 8 gpus, so 1.4m for the cluster above) Those A100 use 300watts each, plus let’s say 500watts for the rest. (8gpus300+500)8servers ≈ 25Kw * 8760 hours = 219k KwH / year. So if your costs are $0.15/KwH that’s only on the order of $32k/year.
    - mirker 4y ago
      It’s still more complicated because you have to amortize costs over several years and then also keep in mind that big players will design their own hardware to avoid the A100 cost and energy. Disks and other devices will also consume power. And you datacenter cooling is now more expensive, etc.
philipov 4y ago
>Real numbers can’t be perfectly represented in hardware simply because there are infinitely many of them. There are also infinitely many integers, but we can represent them just fine inside a finite bound. The problem with reals (and rationals to a lesser extent) is that, within any range we are interested in, a single real number can be infinitely 'long'.
- LittlePeter 4y ago
  That is a nice insight.
- kgwgk 4y ago
  We could even say that within any range we are interested in there are infinitely many of them.
- visarga 4y ago
  A single real number can contain all the information ever produced by humanity.
  - dr_dshiv 4y ago
    Oh, wow! Which one?
    - philipov 4y ago
      I would tell you, but it's too big to fit in the margin.
- LordDragonfang 4y ago
  Similarly put, the infinite number of reals is also a bigger infinity than the infinite number of integers (aleph-null).
ouid 4y ago
real numbers can't be represented in hardware because most of them are incomputable. It's not that there are too many of them, it's that any one of them is unlikly to be representable as the output of any program.

boberoni 4y ago

I like this idea of choosing a representation for numbers based on the use case. Like, "ints" and "floats" but even more specialized. For example, what would be the most efficient binary representation of probabilities values between 0 and 1? In the future, I can imagine hardware + software that is specialized for examples like this.

dr_dshiv 4y ago
Makes me wonder about quantum computing possibilities.
js8 4y ago
I feel like probabilities are best stored as inverse logistic function, because the closer the probability is to 0 or 1, the less you care about significant digits, because the events are going (not) to happen anyway. (That's why I am proponent of LNS.)
- jbay808 4y ago
  I'd agree with this suggestion. Furthermore this does not need a new number format, as IEEE floating point is already pretty ideally distributed. https://en.wikipedia.org/wiki/Logit https://en.wikipedia.org/wiki/Logit
- planede 4y ago
  I feel that it's actually the opposite. There is a world of difference between 0.0...01 and 0.0...001 , where ... represents the same number of 0s in the two cases. Same applies for the other end. edit: Do I really want more resolution between 0.41 and 0.42 than between 0.01 and 0.02?
  - daniel-cussen 4y ago
    You're dead on. Metals are sold by how fine they are, so .999%, .9995% (should end in a 7 numeral not a 5, whole industry is fucking up right there, but at any rate) .9999%, more nines more purity better experimental results. When they talk about 4 nines or 5 nines the actual measurement is of the f(x)= - log(1 - x). That is the purity. Negative of the log of the complement. And that's why 4.5 nines should be .99997% not fucking .99995%, so stupid (it's almost exactly .99997%). Mining engineers, come on! Such impressive degrees! PhD's all over the place! Professors even! I'm sorry, don't mind me with my math. Some experiments like with liquid crystals early on required purities that what chemical company was it, Merck I think, around the year 1900, complained like they felt insulted. Told the inventor of liquid crystals because it was an absurd amount of purity required to get them actually working. But look at them go! Right in front of your very eyes!
- teo_zero 4y ago
  Isn't it the other way out? The logistic curve becomes denser close to 0 and 1. Which makes sense: you will want to tell apart 1 defect per million from 0.01 dpm, and 5-sigma process (99.977%) from 6-sigma (99.99966%), much more than tell apart 30% from 30.001%.

Animats 4y ago

Ah. So it's because the machine learning people mostly work in the -1 .. 1 range, but it's possible for their numbers to go outside that range. So they need a compact representation which uses most of the possible values for the range of interest. If you're down to 8 bit numbers, something like this makes sense for custom machine learning hardware. I wonder if it has graphics applications for high dynamic range images. Probably not.

threatripper 4y ago
The machine learning space probably has the easiest time to adopt a new number format. The players are big enough to define their own silicon and training/inference don't even need the same number format. As long as it works on their new specialized training hardware it's good to go. The real challenge is probably in the power efficiency. As far as I know right now power consumption is the biggest cost factor when training new models. Everything else seems like a minor trade-off but when accounting for the power consumption there is real money involved.
- totoglazer 4y ago
  Hardware is not cheap either at this scale. Reallly rough math follows, please anyone correct me if I err. A100 cards are >$10,000 each. A node is typically 8 and a cluster might be 64 A100s. So about a million in hardware for the medium cluster of 64 A100s. (Cross checking, a machine from lambda labs is apparently ~175k for 8 gpus, so 1.4m for the cluster above) Those A100 use 300watts each, plus let’s say 500watts for the rest. (8gpus300+500)8servers ≈ 25Kw * 8760 hours = 219k KwH / year. So if your costs are $0.15/KwH that’s only on the order of $32k/year.
  - mirker 4y ago
    It’s still more complicated because you have to amortize costs over several years and then also keep in mind that big players will design their own hardware to avoid the A100 cost and energy. Disks and other devices will also consume power. And you datacenter cooling is now more expensive, etc.

philipov 4y ago

>Real numbers can’t be perfectly represented in hardware simply because there are infinitely many of them. There are also infinitely many integers, but we can represent them just fine inside a finite bound. The problem with reals (and rationals to a lesser extent) is that, within any range we are interested in, a single real number can be infinitely 'long'.

LittlePeter 4y ago
That is a nice insight.
kgwgk 4y ago
We could even say that within any range we are interested in there are infinitely many of them.
visarga 4y ago
A single real number can contain all the information ever produced by humanity.
- dr_dshiv 4y ago
  Oh, wow! Which one?
  - philipov 4y ago
    I would tell you, but it's too big to fit in the margin.
LordDragonfang 4y ago
Similarly put, the infinite number of reals is also a bigger infinity than the infinite number of integers (aleph-null).

ouid 4y ago

real numbers can't be represented in hardware because most of them are incomputable. It's not that there are too many of them, it's that any one of them is unlikly to be representable as the output of any program.