4 ms·

You have to be a lot clearer about what you mean by continuous here. An LLM technically does not produce a continuous function on the reals because its inputs a

You have to be a lot clearer about what you mean by continuous here. An LLM technically does not produce a continuous function on the reals because its inputs are floating point numbers with finite precision. Any function on discrete inputs like this has an extension to the reals which is continuous, just imagine joining all the discrete points up with lines. So then your claim wouldn't be about the limits of LLMs themselves, but on the limits of systems that do not take continuous inputs. The question then is do you think that humans take in continuous input?? Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.

lanstin 3y ago
It is quantum which isn’t exactly discrete. You receive one photon or two, not 1.5, sure, but the energy of that photon, its frequency is a real. At least I am not aware of a quantum mechanics over Q. I think the math would be hard because you loose all the convergence properties. Maybe there is some Hilbert space over computable reals, Google is not finding it but I no longer find that to be indicative of anything.
tsimionescu 3y ago
I mean continuous in the calculus sense, as in it doesn't have discontinuities, not as in continuous VS discrete. > Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure. This is a misunderstanding of quantum mechanics. Only certain specific quantities come in discrete quanta (spin, charge, certain energy levels, etc). Other physical quantities are very much continuous - notably time and space. In fact, much of the mathematics of QM is not discretizable, it won't work if you try to make time or space discrete.
- adroniser 3y ago
  But continuous depends on the topology of the space you are working over. If the topology is discrete any function is continuous. I've found some papers that argue that simply from an error corrections standpoint the signals in the brain need to be discrete.
kaba0 3y ago
The whitepaper operates on mathematical numbers, though. They didn’t calculate it with floating points of n-bits.
- adroniser 3y ago
  Given a fixed level of precision for input and output, and a function on this discrete space, we can construct a continuous extension of this function to the reals. Now using the paper we know that there is a neural network with continuous weights that approximates this continuous extension to arbitrary precision. If we imagine rounding the continuous input and output to the fixed precision specified, then because of continuity of the neural network, and the fact there is a finite number of weights, we can choose a tolerance by which we can change each of the weights such that the output does not change by more than the precision of the output value. Thus we can pick a level of precision for the weights where both the weights and inputs and outputs are discrete.

lanstin 3y ago

It is quantum which isn’t exactly discrete. You receive one photon or two, not 1.5, sure, but the energy of that photon, its frequency is a real. At least I am not aware of a quantum mechanics over Q. I think the math would be hard because you loose all the convergence properties. Maybe there is some Hilbert space over computable reals, Google is not finding it but I no longer find that to be indicative of anything.

tsimionescu 3y ago

I mean continuous in the calculus sense, as in it doesn't have discontinuities, not as in continuous VS discrete. > Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure. This is a misunderstanding of quantum mechanics. Only certain specific quantities come in discrete quanta (spin, charge, certain energy levels, etc). Other physical quantities are very much continuous - notably time and space. In fact, much of the mathematics of QM is not discretizable, it won't work if you try to make time or space discrete.

adroniser 3y ago
But continuous depends on the topology of the space you are working over. If the topology is discrete any function is continuous. I've found some papers that argue that simply from an error corrections standpoint the signals in the brain need to be discrete.

kaba0 3y ago

The whitepaper operates on mathematical numbers, though. They didn’t calculate it with floating points of n-bits.

adroniser 3y ago
Given a fixed level of precision for input and output, and a function on this discrete space, we can construct a continuous extension of this function to the reals. Now using the paper we know that there is a neural network with continuous weights that approximates this continuous extension to arbitrary precision. If we imagine rounding the continuous input and output to the fixed precision specified, then because of continuity of the neural network, and the fact there is a finite number of weights, we can choose a tolerance by which we can change each of the weights such that the output does not change by more than the precision of the output value. Thus we can pick a level of precision for the weights where both the weights and inputs and outputs are discrete.