3 ms·

> Get a book describing ML algorithms, and implement them yourself, first using plain C, then with MPI or CUDA, and finally using plain Numpy/MATLAB, or one of

by jamez1 10y ago

> Get a book describing ML algorithms, and implement them yourself, first using plain C, then with MPI or CUDA, and finally using plain Numpy/MATLAB, or one of the low-level ML frameworks (Theano or TensorFlow). You don't seem to understand what high-level and low-level mean, why on earth would you write something in C, and then re-write it in MATLAB? The point of the higher level languages is rapid development at the expense of performance. Re-writing code in a different language is quite a basic skill and not something to waste time perfecting. Instead the focus should be on understanding the language/systems themselves on a deeper level. No offense, but it sounds like you have a lot to learn before you should be giving out advice. Seriously you're talking about kaggle? Yet you don't make a single mention of computer science?

p1esk 10y ago
Let me give you an example. I first learned about neural networks from deeplearning.net tutorials. They use Theano, so when I got a convolutional network running, I was left wondering about two critical pieces: backpropagation, and the actual convolution operation. Theano completely hides them from a user. So I decided to implement a simple convnet using only Python and Numpy. That was when I realized that I had no idea how the backprop works for convnets. I'm guessing most people who only used Theano or similar framework to run convnets don't know how backprop works there. Next, I realized that the code I wrote was too slow for any practical purposes, and I decided to rewrite it in pure C. Well, actually I gave up half way and used Armadillo package (Numpy analog for C/C++), but the end result was my convnet became 5 times faster. It was still too slow, and I noticed that my code is using only one core out of 4 cores available on my computer, so I decided to parallelize it, using Cilk+. It was actually easier than I expected, and soon I got almost linear speedup using all 4 cores. The final exercise was to implement it with CUDA, and I got another factor of 5 speedup running it on my GPU. The benefit of this experience is enormous - I learned a lot about convnets, I learned a lot about tools and libraries useful for ML, and I learned a lot about making my code faster. p.s. I don't quite understand what you meant about Kaggle and CS. p.p.s. And yes, I do still have a lot to learn, but I wish someone gave me this advice when I was starting out.
- jamez1 10y ago
  You had suggested to learn an algorithm in a different order then what you've just written, that's why I suggested it was incorrect. (You put a MATLAB implementation after a C) Computer science is key to all of this, understanding how to write performance code isn't just in the programming skill, but it's also in understanding the complexity theory, graph theory and computer architecture. When going for a job at these places, (you did specify 'ML Engineer', this is different from a researcher), they will grill you to make sure you understand how to use a computer. They want you to know it inside out, not just having done some tutorials, but actually able to debug deep complex issues and performance tune. Getting familiarity with modifying the linux kernel is much more important than having done some silly competitions/tutorials. You have to remove the marketing material from reality. It's not about being able to "write something in C", it's about understanding the conception of every bit of code, and what is happening underneath, to engineer a quality solution.
  - p1esk 10y ago
    The exact order does not really matter. Some people prefer top down approach, others want to start from the bottom, then gradually abstract the details. My point is, you need to know how to work at each of these abstraction levels. I did mention a course on algorithms and data structures as a requirement, that's where you learn about complexity, graphs, and other things. I'm guessing you don't know what Kaggle is. It's not a silly competition. It's a place where you get exposed to real-world problems, using real-world data. This is a good alternative to an actual internship as a ML/Data engineer. You expect ML engineers to know how modify the Linux kernel? Do you also want them to know how to design a superscalar processor in Verilog? How about simulating circuits with SPICE? I don't think so.
    - jamez1 10y ago
      That's ridiculous, why would you need to do a MATLAB implementation after writing it in C? Do you even know what MATLAB is? Do you even understand what abstractions are? If I wrote something in C, I understand it on the MATLAB level. There is no gain from doing that. Kaggle is just a hobbyist competition. How many engineers do you think have used Kaggle to gain their employment? I get that it's hard to understand from the ivory tower of academia what is actually required in the real world. You seem to keep missing the point, an engineer should be comfortable with working on the low level, understanding performance tuning and they should have a solid understanding of architecture on the low level. Experimenting with the linux kernel is one way to get an intuitive feel for how things work, not a requirement. You also seem to keep misunderstanding the domain space, do you even know what an engineer does? What do you think they're doing all day? They're writing high performance code, not solving stupid riddles. They will almost never write anything MATLAB, they engineer things to specifications. I hope your university has an internship, after you complete it maybe you'll understand then.

by jamez1 10y ago

p1esk 10y ago

Let me give you an example. I first learned about neural networks from deeplearning.net tutorials. They use Theano, so when I got a convolutional network running, I was left wondering about two critical pieces: backpropagation, and the actual convolution operation. Theano completely hides them from a user. So I decided to implement a simple convnet using only Python and Numpy. That was when I realized that I had no idea how the backprop works for convnets. I'm guessing most people who only used Theano or similar framework to run convnets don't know how backprop works there. Next, I realized that the code I wrote was too slow for any practical purposes, and I decided to rewrite it in pure C. Well, actually I gave up half way and used Armadillo package (Numpy analog for C/C++), but the end result was my convnet became 5 times faster. It was still too slow, and I noticed that my code is using only one core out of 4 cores available on my computer, so I decided to parallelize it, using Cilk+. It was actually easier than I expected, and soon I got almost linear speedup using all 4 cores. The final exercise was to implement it with CUDA, and I got another factor of 5 speedup running it on my GPU. The benefit of this experience is enormous - I learned a lot about convnets, I learned a lot about tools and libraries useful for ML, and I learned a lot about making my code faster. p.s. I don't quite understand what you meant about Kaggle and CS. p.p.s. And yes, I do still have a lot to learn, but I wish someone gave me this advice when I was starting out.

jamez1 10y ago
You had suggested to learn an algorithm in a different order then what you've just written, that's why I suggested it was incorrect. (You put a MATLAB implementation after a C) Computer science is key to all of this, understanding how to write performance code isn't just in the programming skill, but it's also in understanding the complexity theory, graph theory and computer architecture. When going for a job at these places, (you did specify 'ML Engineer', this is different from a researcher), they will grill you to make sure you understand how to use a computer. They want you to know it inside out, not just having done some tutorials, but actually able to debug deep complex issues and performance tune. Getting familiarity with modifying the linux kernel is much more important than having done some silly competitions/tutorials. You have to remove the marketing material from reality. It's not about being able to "write something in C", it's about understanding the conception of every bit of code, and what is happening underneath, to engineer a quality solution.
- p1esk 10y ago
  The exact order does not really matter. Some people prefer top down approach, others want to start from the bottom, then gradually abstract the details. My point is, you need to know how to work at each of these abstraction levels. I did mention a course on algorithms and data structures as a requirement, that's where you learn about complexity, graphs, and other things. I'm guessing you don't know what Kaggle is. It's not a silly competition. It's a place where you get exposed to real-world problems, using real-world data. This is a good alternative to an actual internship as a ML/Data engineer. You expect ML engineers to know how modify the Linux kernel? Do you also want them to know how to design a superscalar processor in Verilog? How about simulating circuits with SPICE? I don't think so.
  - jamez1 10y ago
    That's ridiculous, why would you need to do a MATLAB implementation after writing it in C? Do you even know what MATLAB is? Do you even understand what abstractions are? If I wrote something in C, I understand it on the MATLAB level. There is no gain from doing that. Kaggle is just a hobbyist competition. How many engineers do you think have used Kaggle to gain their employment? I get that it's hard to understand from the ivory tower of academia what is actually required in the real world. You seem to keep missing the point, an engineer should be comfortable with working on the low level, understanding performance tuning and they should have a solid understanding of architecture on the low level. Experimenting with the linux kernel is one way to get an intuitive feel for how things work, not a requirement. You also seem to keep misunderstanding the domain space, do you even know what an engineer does? What do you think they're doing all day? They're writing high performance code, not solving stupid riddles. They will almost never write anything MATLAB, they engineer things to specifications. I hope your university has an internship, after you complete it maybe you'll understand then.