4 ms·
Thanks for this. My approach to speeding up an autodiff system like this was to write it in terms of nd-arrays rather than scalars, using numpy/cupy [1]. But it
by montebicyclelo 3y ago
Thanks for this. My approach to speeding up an autodiff system like this was to write it in terms of nd-arrays rather than scalars, using numpy/cupy [1]. But it's still slower than deep learning frameworks that compile / fuse operations. Wondering how it compares to the approach in this post. (I might try to benchmark.)
[1] https://github.com/sradc/SmallPebble https://github.com/sradc/SmallPebble
- deleted 3y ago[deleted]
- tekknolagi 3y agoPlease let us know the results for this network!
- tekknolagi 3y agoRe: Small Pebble: what is this Lazy thing? I've seen it crop up a couple places.
- montebicyclelo 3y agoThe underlying autodiff engine in smallpebble runs computations immediately, as they are defined, for the given inputs. But when modelling, you want to define a model, and then use it again and again, on different inputs. The laziness is a way to do that; you can define a graph of autodiff operations (using "sp.Lazy", and "sp.Placeholder"), without running it, and then pass in values and run the graph on those operations. We end up with something similar to how models are defined in tensorflow [1]. [1] https://www.tensorflow.org/js/guide/models_and_layers#the_functional_model https://www.tensorflow.org/js/guide/models_and_layers#the_fu...
- tekknolagi 3y agoAha! Okay, thank you. And how do you pass in the inputs differently?
- montebicyclelo 3y agoIn smallpebble it's currently done by setting the value of the Placeholders, e.g. # define some graph x = sp.Placeholder() y = sp.Lazy(sp.square)(x) # and run the graph later on x.assign_value(sp.Variable(np.array([1, 2, 3]))) y_val = y.run() # the result, y_val is an sp.Variable gradients = sp.get_gradients(y_val) # run same graph with different values x.assign_value(sp.Variable(np.array([5, 6, 7]))) y_val = y.run() gradients = sp.get_gradients(y_val) (There's some more examples of this being used in training loops in the readme.)
- tekknolagi 3y agoOh, oh, okay, these placeholders are like my `input` Values. Neat! Thank you.