4 ms·

Dive into deep learning compilers

142 points by sebg 5y ago

aghilmort 5y ago
cool! is this your work?
37ef_ced3 5y ago
Related: https://NN-512.com https://NN-512.com is a compiler that generates C99 code for neural net inference. It takes as input a simple text description of a convolutional neural net inference graph. It produces as output a human-readable, stand-alone (no dependencies) C99 implementation of that graph. The generated C99 code uses AVX-512 vector instructions to perform inference.
- theK 5y ago
  Tangential question, why not just host a git repo instead of hosting the files without history?
  - 37ef_ced3 5y ago
    The compiler generates and serves its own website with no dependencies. For personal projects (including NN-512) I use Mercurial for version control, not git. But in any case I don't want the compiler (or the website it generates) to depend on git or any other version control software.
    - smaudet 5y ago
      What do you do for version control? I think you have this backwards, anybody could check in the contents of your tarball into a vcs, so not sure the point...
- jhgb 5y ago
  That's cool for anyone who has AVX-512 enabled CPUs. Is there anything that works on CPUs supporting only up to AVX2?
dr_zoidberg 5y ago
For those surprised at the dot product benchmark on 3.1 [0] where np.dot() is substantially faster when using floats over ints, the things is simply that numpy can use BLAS libraries for floats, but not for ints (because BLAS libs don't implement integer data types). [0] https://tvm.d2l.ai/chapter_cpu_schedules/arch.html https://tvm.d2l.ai/chapter_cpu_schedules/arch.html [1] https://stackoverflow.com/questions/11856293/numpy-dot-product-very-slow-using-ints https://stackoverflow.com/questions/11856293/numpy-dot-produ...
- momojo 5y ago
  There's a great explanation on algorithmica as well: https://en.algorithmica.org/hpc/complexity/languages/ https://en.algorithmica.org/hpc/complexity/languages/
srvmshr 5y ago
Dive into DL (d2l) is becoming an incredibly good one stop reference for theory, code and now machine / architecture internals.
adamnemecek 5y ago
Seems like automatic differentiation should be covered.
forgotmyoldacc 5y ago
Hasn't been updated since 2020: https://github.com/d2l-ai/d2l-tvm https://github.com/d2l-ai/d2l-tvm, may be severely outdated, since deep learning compilers is a rapidly moving field.
mrintellectual 5y ago
This goes over network construction very well, but I'd like to know about more about how modern compilers go about doing autograd.
smaudet 5y ago
Legit question, what on earth is this (for)? I tried reading the book a bit and it seems like a library for turning ml models into C code? But its really not clear what the point of this book is about...Is this written by space aliens or is this a technical manual for chemistry? I have coming up on a decade of Python, Java, JS experience, I know what programming is, I don't know what this...thing...is for.

142 points by sebg 5y ago

cool! is this your work?

Related: https://NN-512.com https://NN-512.com is a compiler that generates C99 code for neural net inference. It takes as input a simple text description of a convolutional neural net inference graph. It produces as output a human-readable, stand-alone (no dependencies) C99 implementation of that graph. The generated C99 code uses AVX-512 vector instructions to perform inference.

theK 5y ago
Tangential question, why not just host a git repo instead of hosting the files without history?
- 37ef_ced3 5y ago
  The compiler generates and serves its own website with no dependencies. For personal projects (including NN-512) I use Mercurial for version control, not git. But in any case I don't want the compiler (or the website it generates) to depend on git or any other version control software.
  - smaudet 5y ago
    What do you do for version control? I think you have this backwards, anybody could check in the contents of your tarball into a vcs, so not sure the point...
jhgb 5y ago
That's cool for anyone who has AVX-512 enabled CPUs. Is there anything that works on CPUs supporting only up to AVX2?

dr_zoidberg 5y ago

For those surprised at the dot product benchmark on 3.1 [0] where np.dot() is substantially faster when using floats over ints, the things is simply that numpy can use BLAS libraries for floats, but not for ints (because BLAS libs don't implement integer data types). [0] https://tvm.d2l.ai/chapter_cpu_schedules/arch.html https://tvm.d2l.ai/chapter_cpu_schedules/arch.html [1] https://stackoverflow.com/questions/11856293/numpy-dot-product-very-slow-using-ints https://stackoverflow.com/questions/11856293/numpy-dot-produ...

momojo 5y ago
There's a great explanation on algorithmica as well: https://en.algorithmica.org/hpc/complexity/languages/ https://en.algorithmica.org/hpc/complexity/languages/

srvmshr 5y ago

Dive into DL (d2l) is becoming an incredibly good one stop reference for theory, code and now machine / architecture internals.

adamnemecek 5y ago

Seems like automatic differentiation should be covered.

forgotmyoldacc 5y ago

Hasn't been updated since 2020: https://github.com/d2l-ai/d2l-tvm https://github.com/d2l-ai/d2l-tvm, may be severely outdated, since deep learning compilers is a rapidly moving field.

mrintellectual 5y ago

This goes over network construction very well, but I'd like to know about more about how modern compilers go about doing autograd.

smaudet 5y ago

Legit question, what on earth is this (for)? I tried reading the book a bit and it seems like a library for turning ml models into C code? But its really not clear what the point of this book is about...Is this written by space aliens or is this a technical manual for chemistry? I have coming up on a decade of Python, Java, JS experience, I know what programming is, I don't know what this...thing...is for.