4 ms·

Catch seems to be no detailed implementation/code as it's a private company and so likely a proprietary technology. Haven't read the whole paper and would be d

Catch seems to be no detailed implementation/code as it's a private company and so likely a proprietary technology. Haven't read the whole paper and would be delighted to be wrong.

gnramires 4y ago
It seems this is using actual circuits, as in logic gates. Which is interesting, and offers the benefits discussed, but probably not ideal for things like microcontrollers and other embedded devices. I wonder what the results would be for a more general program search, or a more tractable operation graph involving commonly available instructions like *,+,=,>>,<<, etc. (I guess you could use AST symbols directly? I unfortunately know very little about compilers). This would make them a more general version of neural networks. Since this uses genetic algorithms, which can in principle tackle any kind of structure, I think Turing-completeness (i.e. recurrence and memory) of the programs could yield significantly greater inference capabilities. They do mention flip-flops in the article (which I haven't read completely), I wonder if there is significant recurrence or its just gate buffers. Turing completeness of course opens the gate (no pun intended) for more strange effects and bugs of course (but that's kind of expected of any similar algorithm like neural nets?). The benefits for constrained environments are interesting, I'd love to try it for something like making a tiny insect-like robot and other fun applications :)
- PaulHoule 4y ago
  The future of machine learning, I think, is systems that are a lot more efficient. It’s just nuts that we are encoding linguistic information as float32 (how many of those bits are you really using?) and not much better to be using float16 or float8. I think we might be seeing the beginning of the end of GPU progress but for inference particularly this kind of few-order-of-magnitude.
  - gnramires 4y ago
    I agree on f32, although I still think the differential approach will remain dominant for large systems. Formally, that's likely because of training efficiency of gradient-based approaches. More speculatively, I think beliefs in general benefit from having associated values (strongly believing something, versus weakly believing something) which to me suggest an inherent benefit of numeric values inside the networks (see fuzzy logic[1]). Of course, sometimes you're dealing with rules that are less gradual, s.t. boolean logic (or maybe something like fp4) may suffice. To me the non-differential approach will shine on very constrained environments and open quite a few applications. In the near future we might have very efficient and effective code generation from LLMs which changes the landscape as well.

Catch seems to be no detailed implementation/code as it's a private company and so likely a proprietary technology. Haven't read the whole paper and would be delighted to be wrong.

gnramires 4y ago

It seems this is using actual circuits, as in logic gates. Which is interesting, and offers the benefits discussed, but probably not ideal for things like microcontrollers and other embedded devices. I wonder what the results would be for a more general program search, or a more tractable operation graph involving commonly available instructions like *,+,=,>>,<<, etc. (I guess you could use AST symbols directly? I unfortunately know very little about compilers). This would make them a more general version of neural networks. Since this uses genetic algorithms, which can in principle tackle any kind of structure, I think Turing-completeness (i.e. recurrence and memory) of the programs could yield significantly greater inference capabilities. They do mention flip-flops in the article (which I haven't read completely), I wonder if there is significant recurrence or its just gate buffers. Turing completeness of course opens the gate (no pun intended) for more strange effects and bugs of course (but that's kind of expected of any similar algorithm like neural nets?). The benefits for constrained environments are interesting, I'd love to try it for something like making a tiny insect-like robot and other fun applications :)

PaulHoule 4y ago
The future of machine learning, I think, is systems that are a lot more efficient. It’s just nuts that we are encoding linguistic information as float32 (how many of those bits are you really using?) and not much better to be using float16 or float8. I think we might be seeing the beginning of the end of GPU progress but for inference particularly this kind of few-order-of-magnitude.
- gnramires 4y ago
  I agree on f32, although I still think the differential approach will remain dominant for large systems. Formally, that's likely because of training efficiency of gradient-based approaches. More speculatively, I think beliefs in general benefit from having associated values (strongly believing something, versus weakly believing something) which to me suggest an inherent benefit of numeric values inside the networks (see fuzzy logic[1]). Of course, sometimes you're dealing with rules that are less gradual, s.t. boolean logic (or maybe something like fp4) may suffice. To me the non-differential approach will shine on very constrained environments and open quite a few applications. In the near future we might have very efficient and effective code generation from LLMs which changes the landscape as well.