4 ms·

A logarithmic representation is great for multiplication, but adding numbers becomes an expensive operation.

• js8 4y ago
  Today, both integer addition and multiplication is IMHO done in similar speeds in silicon, although multiplication circuits are way larger. In the logarithmic representation, multiplication becomes addition. I don't think that just by doing everything in logarithms you really increase the overall complexity, rather you transfer it from multiplication onto addition, so I don't think adding logarithms should be more expensive than integer multiplication.. Should it then matter for neural networks, which IIRC require similar amount of additions and multiplications? Even floating-point is actually trading off the simplicity of addition in order to make multiplication easier, because they are quasi-logarithmic representation. However, I wonder if there is a numeric representation where the circuits for addition and multiplication are of similar complexity. Something like half-logarithm, which if applied twice, you get the logarithm.
  • NohatCoder 4y ago
    Addition in logarithmic space is expensive, we don't have any neat ways of doing it. Options include: * Converting back and forth to linear space, using exp and log functions, this is much slower than a regular multiplication. * Evaluating a polynomial that approximates the logarithmic space add, this takes several multiplications, so also much slower. In general we tend to use more additions than multiplications, so trading addition speed for multiplication speed is rarely a good idea, even 1 for 1. If we need a lot of exponentiation keeping some values in logarithmic space may be beneficial, but it has to be those rare cases only.