4 ms·

NN-512 (https://NN-512.com https://NN-512.com) generates stand-alone C code for AVX-512 neural nets The idea was that user companies (or even the hardware comp

NN-512 (https://NN-512.com https://NN-512.com) generates stand-alone C code for AVX-512 neural nets The idea was that user companies (or even the hardware companies: Intel, AMD) would pay me as a contractor to implement deconvolutions, transformers, etc., as extensions to the free/open core I released (which includes very sophisticated support for all forms of dense convolution, fully-connected layers, pooling, batch norm, activations, etc.) The free/open core supplies all the basic operators, but many users would need extensions of the core to support their particular networks (or even just software to feed their parameters into the system and get results out) and hardware companies benefit from the software's existence because they sell chips and NN-512 makes the new vector units on their chips more useful But it didn't work out that way. NN-512 did generate offers to work on other projects (a generous offer from Intel, for example) but nothing that extends NN-512 Why? There are numerous free end-to-end deep learning tools developed by large, company-sponsored teams (Nvidia, Facebook, Google, Intel, etc.) so a specialized tool that requires integration effort is unattractive. That's my guess. So beware, if you're working along similar lines

staunch 6y ago
I saw this in one of your other posts and it seems super interesting to me. 1. Improve the web site slightly (add a contact method and pricing info) and make it clear that it's a company offering an enterprise software tool. Programmers will be viewing the site, so it doesn't need much more than is there already. 2. Charge a flat $7,500/year for a commercial usage license and increase it as the functionality increases. Offer email support and updates. Many programmers can buy a $7,500/year piece of software without their management even asking a follow up question. 3. Write some blog posts showing how to use it and why it's good. Some benchmarks (LOC, performance, etc) vs alternatives would be a good source of posts. Real world examples that people can copy/paste to start with is probably the most useful. 4. Submit the blog posts to HN and some subreddits. 5. Profit (maybe)
- 37ef_ced3 6y ago
  Thank you for this thoughtful reply!
- Aeolun 6y ago
  > Many programmers can buy a $7,500/year piece of software without their management even asking a follow up question. I’d like to know where you work. No place I’ve ever worked would approve that without a lot of questions.
  - unixhero 6y ago
    > He is not wrong
d_burfoot 6y ago
How about a tool that generates C (or Java or X) code from TensorFlow or PyTorch models? That way you can train a model offline, and then export it to your preferred language for serving/production. For bonus points, figure out how to get the generated code to use the GPU properly.
- 37ef_ced3 6y ago
  Addressing your last sentence, I am actually a GPU programmer by trade. GPUs are my specialty, and much of my professional work has been in Nvidia's CUDA But AVX-512 is a real step forward, if you look at the price/performance ratio for real-world workloads. So AVX-512 is the focus of NN-512 GPU cloud compute is almost unbelievably expensive. Even Linode charges $1000 per month, or $1.50 per hour (look at the GPU plans: https://www.linode.com/pricing/#row--compute https://www.linode.com/pricing/#row--compute). It's really hard to keep that GPU saturated, which is what you need to do to get your money's worth An AVX-512 Skylake-X cloud compute instance costs $10 per CPU-core per month, or 2 cents per hour, at Vultr (https://www.vultr.com/products/cloud-compute/ https://www.vultr.com/products/cloud-compute/). It's easy to saturate that core Think about it this way: a GPU is really just a cluster of SIMD units with fast memory. A GPU thread variable is just like a SIMD lane, and a GPU warp variable is just like a SIMD vector register Nvidia's SIMD pseudo-instructions are called PTX and they are like a 1024-bit version of AVX-512. An AVX-512 core is like a general purpose CPU with a 512-bit GPU core built in So paying for a single AVX-512 core is like paying for part of a GPU (16 threads), plus the general purpose compute you need to keep that GPU part supplied with work
  - bravura 6y ago
    I've been interested in your project. I don't think you're realizing the potential of the parent's suggestion. You motivate the work through bottom-up construction, but I don't think you have a clear explanation for who the users are. They must be: 1) Sure they are only going to use models that are supported by NN-512. This is a big jump for me and many people who want to do transformers, LSTMs, etc. and not just what NN-512 costs. 2) They have to be price sensitive. But if you want larger companies, they will rarely just have one use-case that NN-512 supports; they will probably have a ton of use-cases that NN-512 supports. 3) They want to target AVX-512. If they are a medium or larger company they probably want to have tooling around this, to support model development and later deployment. Those are my thoughts in general. I currently need to be able to target and deploy on PC and Mac CPUs from the past 10 years. (I know, this is specific.) I think parent is saying that individual devs are more likely to try NN-512 if they don't have to learn a new API, since most of them want to have the flexibility to fool around before committing to a new tech or bringing it in-house. I guess my suggestion is really try to specifically define who the user is for NN-512 and what NN problems they are trying to solve, and let that shape the roadmap.
reitzensteinm 6y ago
Your project is seriously cool, I'm sorry it didn't work out for you.
- 37ef_ced3 6y ago
  Thanks dude
  - person_of_color 6y ago
    Where did you land?
Smerity 6y ago
I work in the space and was impressed with NN-512 as there's a painful gap in inference cost between CPU and GPU that doesn't have to exist. Intel and AMD are really missing a boat here, most other companies have enough cash they just go to GPUs, academics rarely sling low level code even in CUDA let alone AVX-512, and other than Fabrice Bellard's work few I've seen few go that low level. My suggestion would be to focus on an initial use case where a very limited low cost / high efficiency CPU model can provide massive advantage. NN-512 should be the framework that expands from that Redis like core. The limited use case tactic is what I'm focusing on[1], mainly as I have a particular application and have less technical brilliance than yourself so need to focus ;) An aged but still relevant example is the early word2vec work which was (and still is) frequently better to throw onto CPUs than GPUs. A well tuned implementation is not only advantageous on CPU but can win out in many scenarios where cost / latency / ... are important. Congrats on the project though! I'd be curious for your thoughts for the future if you ever want to chat =] [1]: Initial experiments written up as a tutorial with Rust and ISPC for a specific CPU based NN task - https://state.smerity.com/smerity/state/01E8RNH7HRRJT2A63NSX3N6SP1 https://state.smerity.com/smerity/state/01E8RNH7HRRJT2A63NSX...
- 37ef_ced3 6y ago
  In your [1], are your input/output arrays aligned the same for the fastest SIMD and ISPC runs?
  - Smerity 6y ago
    Not in that codebase as it was a tutorial / wanted to ensure it's callable from safe Rust code so stuck with `_mm256_loadu_ps`. That code was just playing with dot product like lookup over vectors on CPU. The code I'm more interested in is trying to cram models into ~L2/L3 cache such that a CPU optimized model can be trained on GPU to be deployed on CPU.
exged 6y ago
Just wondering, what makes NN-512 superior to Intel's oneAPI DNN library?
- 37ef_ced3 6y ago
  For example: try to find a deep learning framework that uses Intel's oneAPI DNN library and achieves 80% of the performance of this: https://nn-512.com/browse/DenseNet121 https://nn-512.com/browse/DenseNet121 Tell me if you can find one
overkalix 6y ago
I had a similar idea last year, but clearly I'm not talented nor experienced enough to do it. Congratulations dude, this rocks.

staunch 6y ago

I saw this in one of your other posts and it seems super interesting to me. 1. Improve the web site slightly (add a contact method and pricing info) and make it clear that it's a company offering an enterprise software tool. Programmers will be viewing the site, so it doesn't need much more than is there already. 2. Charge a flat $7,500/year for a commercial usage license and increase it as the functionality increases. Offer email support and updates. Many programmers can buy a $7,500/year piece of software without their management even asking a follow up question. 3. Write some blog posts showing how to use it and why it's good. Some benchmarks (LOC, performance, etc) vs alternatives would be a good source of posts. Real world examples that people can copy/paste to start with is probably the most useful. 4. Submit the blog posts to HN and some subreddits. 5. Profit (maybe)

37ef_ced3 6y ago
Thank you for this thoughtful reply!
Aeolun 6y ago
> Many programmers can buy a $7,500/year piece of software without their management even asking a follow up question. I’d like to know where you work. No place I’ve ever worked would approve that without a lot of questions.
- unixhero 6y ago
  > He is not wrong

d_burfoot 6y ago

How about a tool that generates C (or Java or X) code from TensorFlow or PyTorch models? That way you can train a model offline, and then export it to your preferred language for serving/production. For bonus points, figure out how to get the generated code to use the GPU properly.

37ef_ced3 6y ago
Addressing your last sentence, I am actually a GPU programmer by trade. GPUs are my specialty, and much of my professional work has been in Nvidia's CUDA But AVX-512 is a real step forward, if you look at the price/performance ratio for real-world workloads. So AVX-512 is the focus of NN-512 GPU cloud compute is almost unbelievably expensive. Even Linode charges $1000 per month, or $1.50 per hour (look at the GPU plans: https://www.linode.com/pricing/#row--compute https://www.linode.com/pricing/#row--compute). It's really hard to keep that GPU saturated, which is what you need to do to get your money's worth An AVX-512 Skylake-X cloud compute instance costs $10 per CPU-core per month, or 2 cents per hour, at Vultr (https://www.vultr.com/products/cloud-compute/ https://www.vultr.com/products/cloud-compute/). It's easy to saturate that core Think about it this way: a GPU is really just a cluster of SIMD units with fast memory. A GPU thread variable is just like a SIMD lane, and a GPU warp variable is just like a SIMD vector register Nvidia's SIMD pseudo-instructions are called PTX and they are like a 1024-bit version of AVX-512. An AVX-512 core is like a general purpose CPU with a 512-bit GPU core built in So paying for a single AVX-512 core is like paying for part of a GPU (16 threads), plus the general purpose compute you need to keep that GPU part supplied with work
- bravura 6y ago
  I've been interested in your project. I don't think you're realizing the potential of the parent's suggestion. You motivate the work through bottom-up construction, but I don't think you have a clear explanation for who the users are. They must be: 1) Sure they are only going to use models that are supported by NN-512. This is a big jump for me and many people who want to do transformers, LSTMs, etc. and not just what NN-512 costs. 2) They have to be price sensitive. But if you want larger companies, they will rarely just have one use-case that NN-512 supports; they will probably have a ton of use-cases that NN-512 supports. 3) They want to target AVX-512. If they are a medium or larger company they probably want to have tooling around this, to support model development and later deployment. Those are my thoughts in general. I currently need to be able to target and deploy on PC and Mac CPUs from the past 10 years. (I know, this is specific.) I think parent is saying that individual devs are more likely to try NN-512 if they don't have to learn a new API, since most of them want to have the flexibility to fool around before committing to a new tech or bringing it in-house. I guess my suggestion is really try to specifically define who the user is for NN-512 and what NN problems they are trying to solve, and let that shape the roadmap.

reitzensteinm 6y ago

Your project is seriously cool, I'm sorry it didn't work out for you.

37ef_ced3 6y ago
Thanks dude
- person_of_color 6y ago
  Where did you land?

Smerity 6y ago

I work in the space and was impressed with NN-512 as there's a painful gap in inference cost between CPU and GPU that doesn't have to exist. Intel and AMD are really missing a boat here, most other companies have enough cash they just go to GPUs, academics rarely sling low level code even in CUDA let alone AVX-512, and other than Fabrice Bellard's work few I've seen few go that low level. My suggestion would be to focus on an initial use case where a very limited low cost / high efficiency CPU model can provide massive advantage. NN-512 should be the framework that expands from that Redis like core. The limited use case tactic is what I'm focusing on[1], mainly as I have a particular application and have less technical brilliance than yourself so need to focus ;) An aged but still relevant example is the early word2vec work which was (and still is) frequently better to throw onto CPUs than GPUs. A well tuned implementation is not only advantageous on CPU but can win out in many scenarios where cost / latency / ... are important. Congrats on the project though! I'd be curious for your thoughts for the future if you ever want to chat =] [1]: Initial experiments written up as a tutorial with Rust and ISPC for a specific CPU based NN task - https://state.smerity.com/smerity/state/01E8RNH7HRRJT2A63NSX3N6SP1 https://state.smerity.com/smerity/state/01E8RNH7HRRJT2A63NSX...

37ef_ced3 6y ago
In your [1], are your input/output arrays aligned the same for the fastest SIMD and ISPC runs?
- Smerity 6y ago
  Not in that codebase as it was a tutorial / wanted to ensure it's callable from safe Rust code so stuck with `_mm256_loadu_ps`. That code was just playing with dot product like lookup over vectors on CPU. The code I'm more interested in is trying to cram models into ~L2/L3 cache such that a CPU optimized model can be trained on GPU to be deployed on CPU.

exged 6y ago

Just wondering, what makes NN-512 superior to Intel's oneAPI DNN library?

37ef_ced3 6y ago
For example: try to find a deep learning framework that uses Intel's oneAPI DNN library and achieves 80% of the performance of this: https://nn-512.com/browse/DenseNet121 https://nn-512.com/browse/DenseNet121 Tell me if you can find one

overkalix 6y ago

I had a similar idea last year, but clearly I'm not talented nor experienced enough to do it. Congratulations dude, this rocks.