3 ms·
To directly quote the source: # these are the llops your accelerator must implement, along with toCpu UnaryOps = Enum("UnaryOps", ["NOOP", "NEG", "RELU
by WithinReason 4y ago
To directly quote the source:
# these are the llops your accelerator must implement, along with toCpu
UnaryOps = Enum("UnaryOps", ["NOOP", "NEG", "RELU", "EXP", "LOG", "SIGN", "RECIPROCAL"])
BinaryOps = Enum("BinaryOps", ["ADD", "SUB", "MUL", "DIV", "POW", "CMPEQ"])
ReduceOps = Enum("ReduceOps", ["SUM", "MAX"])
MovementOps = Enum("MovementOps", ["RESHAPE", "PERMUTE", "EXPAND", "FLIP", "STRIDED", "PAD", "SHRINK"])
ProcessingOps = Enum("ProcessingOps", ["CONV"])
https://github.com/geohot/tinygrad/blob/caea34c52996cde2ed464d740ee9b00d50ddccf5/tinygrad/ops.py#L10 https://github.com/geohot/tinygrad/blob/caea34c52996cde2ed46...
There is a MAX but not a MIN? Is that because max(x,y) = -min(-x,-y)? But then why is there a SUB? Why is there a RELU if it's only max(0,x)? Maybe MIN is just too rare to be worth implementing?
- georgehotz 4y agoMin is an HLOP. From: https://github.com/geohot/tinygrad/blob/master/tinygrad/tensor.py#L222 https://github.com/geohot/tinygrad/blob/master/tinygrad/tens... def min(self, axis=None, keepdim=False): return -((-self).max(axis=axis, keepdim=keepdim)) All folded together, no slower than MAX.
- WithinReason 4y agoBut then SUB, DIV and RELU could be an HLOP as well, no?
- georgehotz 4y agoWe could have NEG instead of SUB, but with the constant folding it's a wash. DIV is already an HLOP with reciprocal (used to use POW, but that was slower. And what would you implement RELU in terms of?
- WithinReason 4y agomax(0,x)
- georgehotz 4y agoThat's a ReduceOp right now, more annoying to reason about than a UnaryOp. But in the limit, yea. Or add an elementwise BinaryOp for max. Submit a PR if you can improve something!