4 ms·
The problem is that unless you perfectly specify your tests to provide a complete and formal definition of the interface, GP can produce results that might appe
by zzalpha 10y ago
The problem is that unless you perfectly specify your tests to provide a complete and formal definition of the interface, GP can produce results that might appear correct but is actually subtly (or grossly) broken.
That's an extremely difficult task for any but the most trivial problems. It shares many of the same issues with formal verification, which has also failed to catch on beyond the odd niche project.
- ethbro 10y agoAs an example of why this is difficult: when integrating with legacy systems you often have (1) no API (the concept didn't exist / contractor didn't bother to isolate one), (2) no comprehensive documentation of the legacy system's internals (often, not even source), & (3) no human expert available with experience on the legacy system. Turning any kind of genetic / ML system loose on that? How would you even write test cases, short of backfeeding production data? And expecting the result to pass muster with auditors?
- halflings 10y agoYup, in ML you would call this the generalization problem: How are you going to build your algorithm build something useful instead of just memorizing the inputs in some way? Differentiable Neural Computers [1] are getting us closer (by incorporating an explicit I/O operations on a memory) but it's uncertain how much test data you would need to build complex code routines. https://deepmind.com/blog/differentiable-neural-computers/ https://deepmind.com/blog/differentiable-neural-computers/