5 ms·

I think to some extent, this kind of difficulty occurs with any numerical programming. With typical software engineering the bugs are usually in logic. All but

by blt 10y ago

I think to some extent, this kind of difficulty occurs with any numerical programming. With typical software engineering the bugs are usually in logic. All but the hardest problems can be diagnosed with a visual debugger. With numerical code, though, you usually can't look at the state of a few discrete-valued variables and a stack trace to figure out what went wrong. "Why is this matrix singular?" can take days to answer. You spend a lot of time staring at the code, comparing it to the math on paper, trying to visualize high-dimensional intermediate data, etc. Continuous math can be a lot harder to reason about than discrete.

simonbarker87 10y ago
This was it for me. I don't have a CS background but with PhD in electronics/semiconductors I can usually brute force my way through things and get a good grasp. I got half way through Andrew Ng's ML course and felt I fully understood the concepts but spent all of my time battling with matrices and the foundation mathematics rather than building on ML understanding. If I had a better grasp of the maths then I think I'd get on better with it.
nabla9 10y ago
It's even worse than that. Talking from personal experience in the private sector: You go trough the whole process and present your results for a customer. They make the required changes, see few percent improvement in the bottom line and are happy. Year later you pick up the same code for another project and discover small error in data collection script that completely invalidates everything. Garbage in, garbage out errors can have internal consistency that survive cross validation and provide similar results with many different algorithms and models. Random changes in the real word can produce actual gains. Standard machine learning datasets rarely suffer from this problem because results and semantics of the problem are known beforehand. If you have a original black-box problem, it's possible to do random search and improve by accident.
boulos 10y ago
tl;dr: Agreed, but the problems and models ML folks play with are more complex than most physics-like systems. Having spent a lot of time doing Monte Carlo rendering, I can say that "why is this pixel messed up" can be a painful affair (and that domain is at least visual!). Bad input geometry, NaNs or Infs because you forgot to handle some 1/r^2 term, etc. occur in all numerical computing. Some of this is actually obviated in TensorFlow the same way it is in much of numerical computing: frameworks that have been battle hardened through years of bug fixes. This lets you at least focus on "Did I describe my model correctly?" rather than "Did I implement gradient descent correctly too?". Again, rendering (and all physical simulation) has the benefit of producing a picture and the inputs are effectively "physical". The challenge for machine learning is that you're trying to optimize for figuring out the underlying model rather than relying on the laws of physics for your problem. While this is just as true in any numerical modeling problem ("Should this be a 4th degree polynomial or just cubic?"), the models that ML folks put together are effectively quite complex and the input datasets are both very large and usually unstructured. The visualization of the network being trained in the TF Playground (http://playground.tensorflow.org http://playground.tensorflow.org) helps with the model but not the original space.

by blt 10y ago

simonbarker87 10y ago

This was it for me. I don't have a CS background but with PhD in electronics/semiconductors I can usually brute force my way through things and get a good grasp. I got half way through Andrew Ng's ML course and felt I fully understood the concepts but spent all of my time battling with matrices and the foundation mathematics rather than building on ML understanding. If I had a better grasp of the maths then I think I'd get on better with it.

nabla9 10y ago

It's even worse than that. Talking from personal experience in the private sector: You go trough the whole process and present your results for a customer. They make the required changes, see few percent improvement in the bottom line and are happy. Year later you pick up the same code for another project and discover small error in data collection script that completely invalidates everything. Garbage in, garbage out errors can have internal consistency that survive cross validation and provide similar results with many different algorithms and models. Random changes in the real word can produce actual gains. Standard machine learning datasets rarely suffer from this problem because results and semantics of the problem are known beforehand. If you have a original black-box problem, it's possible to do random search and improve by accident.

boulos 10y ago

tl;dr: Agreed, but the problems and models ML folks play with are more complex than most physics-like systems. Having spent a lot of time doing Monte Carlo rendering, I can say that "why is this pixel messed up" can be a painful affair (and that domain is at least visual!). Bad input geometry, NaNs or Infs because you forgot to handle some 1/r^2 term, etc. occur in all numerical computing. Some of this is actually obviated in TensorFlow the same way it is in much of numerical computing: frameworks that have been battle hardened through years of bug fixes. This lets you at least focus on "Did I describe my model correctly?" rather than "Did I implement gradient descent correctly too?". Again, rendering (and all physical simulation) has the benefit of producing a picture and the inputs are effectively "physical". The challenge for machine learning is that you're trying to optimize for figuring out the underlying model rather than relying on the laws of physics for your problem. While this is just as true in any numerical modeling problem ("Should this be a 4th degree polynomial or just cubic?"), the models that ML folks put together are effectively quite complex and the input datasets are both very large and usually unstructured. The visualization of the network being trained in the TF Playground (http://playground.tensorflow.org http://playground.tensorflow.org) helps with the model but not the original space.