3 ms·

I'm curious about this claim: A set of all the possible states of the program’s memory, can be modeled by a finite dimensional real vector space V ≡ Rn. Isn't

I'm curious about this claim: A set of all the possible states of the program’s memory, can be modeled by a finite dimensional real vector space V ≡ Rn. Isn't the set of possible states of a computer's memory countable? How can one reasonably model this using a vector space over the reals? The paper claims any field will do. Well Z_p isn't a Banach space. It seems to me if they want a notion of differentiability of a program that one would work in a field of characteristic 2 and use the module of Kahler differentials or use discrete differential forms. I certainly don't know enough about computer science but using a real vector space seems like it wouldn't be helpful.

cs702 10y ago
yequalsx: yes, almost by definition, memory made up of bits should be modeled with a field of characteristic 2. That seems the most "natural" way to do it. However, the authors of this paper are thinking about "computers" in a broader sense. Take a look at these two papers: * "Neural Turing Machines:" https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 * "Hybrid computing using a neural network with dynamic external memory:" http://www.nature.com/nature/journal/v538/n7626/full/nature20101.html http://www.nature.com/nature/journal/v538/n7626/full/nature2... and a related blog post at https://deepmind.com/blog/differentiable-neural-computers https://deepmind.com/blog/differentiable-neural-computers The memory space of these "neural" computers is formally modeled as R^n (n being the total number of parameters and memory addresses), and implemented approximately with 32-bit floating point numbers in practice (i.e., with a memory space of (2^32)^n bits). As you read the paper linked to by the OP, think about these neural computers instead of the one atop your desk. EDITS: removed sloppy math and lightly edited some sentences.
- cs702 10y ago
  Correction: on the second to last sentence above, the (2^32)^n figure is incorrect. I make this sort of mistake all the time... sorry! The point still stands, however :-)
- yequalsx 10y ago
  Thanks for the links and insight!
  - cs702 10y ago
    Glad to be helpful. And sorry for the sloppy math -- my mind is always on a million other things at the same time! :-)

cs702 10y ago

yequalsx: yes, almost by definition, memory made up of bits should be modeled with a field of characteristic 2. That seems the most "natural" way to do it. However, the authors of this paper are thinking about "computers" in a broader sense. Take a look at these two papers: * "Neural Turing Machines:" https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 * "Hybrid computing using a neural network with dynamic external memory:" http://www.nature.com/nature/journal/v538/n7626/full/nature20101.html http://www.nature.com/nature/journal/v538/n7626/full/nature2... and a related blog post at https://deepmind.com/blog/differentiable-neural-computers https://deepmind.com/blog/differentiable-neural-computers The memory space of these "neural" computers is formally modeled as R^n (n being the total number of parameters and memory addresses), and implemented approximately with 32-bit floating point numbers in practice (i.e., with a memory space of (2^32)^n bits). As you read the paper linked to by the OP, think about these neural computers instead of the one atop your desk. EDITS: removed sloppy math and lightly edited some sentences.

cs702 10y ago
Correction: on the second to last sentence above, the (2^32)^n figure is incorrect. I make this sort of mistake all the time... sorry! The point still stands, however :-)
yequalsx 10y ago
Thanks for the links and insight!
- cs702 10y ago
  Glad to be helpful. And sorry for the sloppy math -- my mind is always on a million other things at the same time! :-)