3 ms·

Skimming papers like this make me think maybe we spend too much time computing stuff in discrete spaces vs continuous ones. This textbook covers CS theory usin

Skimming papers like this make me think maybe we spend too much time computing stuff in discrete spaces vs continuous ones. This textbook covers CS theory using real numbers instead of integers. https://www.amazon.com/Complexity-Real-Computation-Lenore-Blum/dp/0387982817/ref=sr_1_2?keywords=real+computation&qid=1582431445&sr=8-2 https://www.amazon.com/Complexity-Real-Computation-Lenore-Bl...

gumby 7y ago
> Skimming papers like this make me think maybe we spend too much time computing stuff in discrete spaces vs continuous ones. I would go one step further and argue that we shouldn't teach kids discrete math first, but rather continuous math instead. Sure, you have discrete digits and toys, but Piaget (and his student Papert) observe that kids begin pouring water between different containers in the bath before they can do integer counting and from that develop understanding that objects of different shape can have the same volume and concepts of partial filling, ratios etc. The human scale world is continuous more than it is discrete.
- eru 7y ago
  Humans have some intuitions for both discrete and continuous domains. (Euclidean) geometry is an interesting case. It has discrete arrangements that you can vary continuously. Of course, there's also areas of math without anything resembling numbers or the discrete vs continuous distinction in it.
- threatofrain 7y ago
  Current K-12 curriculum is, from a perspective, all about preparing kids for 3 years of calculus. From this pedagogical perspective, discrete narratives are there to be a stepping stone into continuous narratives. Some people like Gilbert Strang believe that there's way too much emphasis on calculus and not enough on algebra.
nestorD 7y ago
With the strong caveat that floating point numbers are technicaly discrete. Anecdotal consequence: 16bits floating point numbers have so much non-linearity in their round-off error that you can use them to build neural network with no activation functions (which are traditionally needed to introduces non-linearity).
- GregarianChild 7y ago
  This is very interesting. I have been wondering about this for years. I have asked neural network people whether it is possible to build a neural network from the non-linearity induced by rounding. I have never received a convincing answer. Would you be able to point me towards a write-up of this fact? And why is this not used in practice? I imagine that how well this works also strongly depends on the kinds of rounding. I imagine that stochastic rounding, or the rounding used in Google's bfloat16 are different in this regard in comparison with standard IEEE floating point rounding.
  - unlikelymordant 7y ago
    https://openai.com/blog/nonlinear-computation-in-linear-networks/ https://openai.com/blog/nonlinear-computation-in-linear-netw...
  - nestorD 7y ago
    Someone else gave a link to an openai blog post, I personally first heard about it in a post by facebook (on their experiments with small precision). I believe it is not used because it requires 16bits precision which, nowadays, you only get on GPU. People usually train on GPU but then evaluate on CPU (in production) where the discontinuity would be much smaller (as you would use 32 bits precision). Furthermore I don't know if, in practice, that type of discontinuity trains as well as a classical activation function (the gradient propagation might be hindered by the limited precision).
  - formalsystem 7y ago
    See https://en.wikipedia.org/wiki/Analog_computer https://en.wikipedia.org/wiki/Analog_computer AFAIK analog computers are still the standard in radars for example and it sound like neural networks would benefit from similar hardware.
versteegen 7y ago
I would like to suggest "Concrete Mathematics: A Foundation for Computer Science", by Ronald Graham, Donald Knuth, and Oren Patashnik, 1994. "A blend of CONtinuous and disCRETE mathematics." "A textbook that is widely used in computer-science departments as a substantive but light-hearted treatment of the analysis of algorithms" --Wikipedia

gumby 7y ago

> Skimming papers like this make me think maybe we spend too much time computing stuff in discrete spaces vs continuous ones. I would go one step further and argue that we shouldn't teach kids discrete math first, but rather continuous math instead. Sure, you have discrete digits and toys, but Piaget (and his student Papert) observe that kids begin pouring water between different containers in the bath before they can do integer counting and from that develop understanding that objects of different shape can have the same volume and concepts of partial filling, ratios etc. The human scale world is continuous more than it is discrete.

eru 7y ago
Humans have some intuitions for both discrete and continuous domains. (Euclidean) geometry is an interesting case. It has discrete arrangements that you can vary continuously. Of course, there's also areas of math without anything resembling numbers or the discrete vs continuous distinction in it.
threatofrain 7y ago
Current K-12 curriculum is, from a perspective, all about preparing kids for 3 years of calculus. From this pedagogical perspective, discrete narratives are there to be a stepping stone into continuous narratives. Some people like Gilbert Strang believe that there's way too much emphasis on calculus and not enough on algebra.

nestorD 7y ago

With the strong caveat that floating point numbers are technicaly discrete. Anecdotal consequence: 16bits floating point numbers have so much non-linearity in their round-off error that you can use them to build neural network with no activation functions (which are traditionally needed to introduces non-linearity).

GregarianChild 7y ago
This is very interesting. I have been wondering about this for years. I have asked neural network people whether it is possible to build a neural network from the non-linearity induced by rounding. I have never received a convincing answer. Would you be able to point me towards a write-up of this fact? And why is this not used in practice? I imagine that how well this works also strongly depends on the kinds of rounding. I imagine that stochastic rounding, or the rounding used in Google's bfloat16 are different in this regard in comparison with standard IEEE floating point rounding.
- unlikelymordant 7y ago
  https://openai.com/blog/nonlinear-computation-in-linear-networks/ https://openai.com/blog/nonlinear-computation-in-linear-netw...
- nestorD 7y ago
  Someone else gave a link to an openai blog post, I personally first heard about it in a post by facebook (on their experiments with small precision). I believe it is not used because it requires 16bits precision which, nowadays, you only get on GPU. People usually train on GPU but then evaluate on CPU (in production) where the discontinuity would be much smaller (as you would use 32 bits precision). Furthermore I don't know if, in practice, that type of discontinuity trains as well as a classical activation function (the gradient propagation might be hindered by the limited precision).
- formalsystem 7y ago
  See https://en.wikipedia.org/wiki/Analog_computer https://en.wikipedia.org/wiki/Analog_computer AFAIK analog computers are still the standard in radars for example and it sound like neural networks would benefit from similar hardware.

versteegen 7y ago

I would like to suggest "Concrete Mathematics: A Foundation for Computer Science", by Ronald Graham, Donald Knuth, and Oren Patashnik, 1994. "A blend of CONtinuous and disCRETE mathematics." "A textbook that is widely used in computer-science departments as a substantive but light-hearted treatment of the analysis of algorithms" --Wikipedia