3 ms·
"I wanted to write something that is concise, easy to understand, easy to extend, and reasonably efficient. There is a subtle trade-off between conciseness and
by kiyoto 12y ago
"I wanted to write something that is concise, easy to understand, easy to extend, and reasonably efficient. There is a subtle trade-off between conciseness and extensibility: If we use a very high level language that already has a "neural_network_train" function, we can write very concise code but we lose the ability to change the training algorithm. If we use a very low level language that only provides primitive arithmetic operations, all the algorithm details are exposed and modifiable but the code is bulky and difficult to understand. For a happy medium, the code should reflect the level at which I think of the problem"
The tradeoff is not between concision and extensibility, but high- and low-level computations.
Even if the language natively implements a "neural_network_train" function, as long as the language also offers low-level primitives to implement all the necessary parts of the neural_network function, the language is no less extensible than the OP's suggested alternative. For example, almost 100% of R users use "lm" to run linear regressions, but R has all the necessary pieces to implement the linear regression calculation (either by inverting matrices or running iterative gradient descent algorithms)
The OP conflates the library-level abstraction and language-level abstraction. I am with him in that there is a trade off between concision and extensibility w/r/t language-level abstraction. The library-level abstraction is pragmatically important (i.e., you would not use OCaml to run websites) but theoretically uninteresting (Ocaml can certainly express all the needed computation for a web server).
- Houshalter 12y agoAs a practical example, brain.js is a very limited neural network library. You can only provide a dataset and train on that dataset. You can't implement any variations like dropout, stochastic gradient descent, momentum, etc. Whereas a good neural network library like Torch lets you work at a much lower level of abstraction. You can put together individual layers, and it gives you the internal code for doing forward and backward passes, and chaining them together.
- kiyoto 12y agoThat's exactly what I am talking about. Your anti-example is a case of library-level abstraction. Now, if you are saying that JavaScript doesn't lend itself to writing extensible machine learning routines whereas Lua does, it's a different story.
- agibsonccc 12y agoI don't think there has to be. Many libraries for example overkill it and make everything a layer when it should be a layer with a hyper parameter. A good example of this is drop out. Why not make it a knob rather than a layer? I think people should be given high level primitives like "layers", allow them to make their own where necessary, but allow the defaults to be: Layer with x (dropout,momentum,..) trained by optimization algo: LBFGS,Hessian Free,.. This allows people to experiment with different configurations without having to dive deep to achieve some basic problems. Relevant to julia: it's a great language and what I wish production code could look like (while being fast!) Like rust, it's in a pretty alpha state right now. I'm watching the language heavily though.