15 ms·

TensorFlow, Keras and deep learning, without a PhD

392 points by blopeur 6y ago

donquichotte 6y ago
Tangentially related (and also using the ubiquitous MNIST dataset), Sebastian Lague started a brilliant, but unfortunately unfinished video series on building neural networks from scratch. This video was an absolute eye-opener me [1] on what classification is, how it works and why a non-linear activation function is required. I probably learned more in the 5 minutes watching this than doing multiple Coursera courses on the subject. [1] https://www.youtube.com/watch?v=bVQUSndDllU https://www.youtube.com/watch?v=bVQUSndDllU
- pests 6y ago
  One "ah ha" about that required non-linear function I had was the fact that if you were just passing numbers through a series of linear functions they could by definition just be combined into one equation.
halflings 6y ago
The video version [1] is also pretty awesome, though its code itself is a bit outdated now. Explains a lot of very practical issues that you might not find in most academic textbooks, but you encounter every day in practice. [1] https://www.youtube.com/watch?v=vq2nnJ4g6N0 https://www.youtube.com/watch?v=vq2nnJ4g6N0
amrx101 6y ago
Have a simple rant here. All these BIG $ companies every now and then come out with statements and what not, that doing AI ML is very easy and every one including their cats should do AI, ML courses and training(preferably on their platform). Once that is done the job market is yours. Reality is far from this. - Today AI|ML does not have the capability marketed by these big companies. Incidentally marketing is targeted at governments, big non tech companies and gullible undergraduates. - Undergrads many a times take these training courses wherein they acquire the skill set to call these APIs and flood the job market wherein a data-entry or data-analyst job is tagged as AI|ML job. - High paying jobs in AI|ML still require a Masters or PhD or a mathematical background. In conclusion, the current hype around AI|ML is misguiding gullible undergrads and governments(I dont mind the government being cheated THOUGH).
- mpfundstein 6y ago
  i am doing quite some deep learning work as part of my consultancy practice. And its all hand made stuff , together with lots of trial and error while trying to replicate papers that might be relevant fkr the task at hand. so i totally agree with your statement. big corps overhype the shit out of it in order to sell and lots of n00bs fail for it. regarding your last points. even if its super easy nowadays to deploy YOLO, and everyone and his mother can do it, making actually something that works and provides business value is hardcore. and without scientific skills_> no chance
  - lordthirdeye 6y ago
    if you don't mind me asking, what do you mean by "hand made". I'm currently wworking on a information science degree and I'm trying to focus on machine learning and data science. Could you go into a little more detail on what gives something business value?
    - disgruntledphd2 6y ago
      Not the OP, but ultimately something that increases revenue or decreases costs by some measurable amount. The best thing you can do to make yourself good at this is to practice. Get some Kaggle data and try to fit a model. Realise your data is crap, clean data, repeat. Every useful system is hand made in the sense that there's a vast amount of set up and operational code. Mostly the model's the easy part (although it will take so much time to run).
    - mpfundstein 6y ago
      hand-make as in: - looking very carefully into the very specific challenge of your client - figuring out how (and if) ML can help - figuring if its still economically feasible (costs of research vs perceived(!) benefit) - deriving a solution. - tinkering tinkering tinkering. usually more with the data than with the models :-) All my A.I. projects are essentially outsourced R&D projects where we deliver the brain and computing power. So far, it never was as easy like installing YOLO or any other off the shelf product. Edit: You also need very often custom software to create custom datasets. AI models are often only tested on academic datasets but I observed empirically that their performance transfers badly to real world datasets. So you need to create your own datasets etc. This is often a non-trivial problem. So I wrote a lot of dataset creation tools in my AI practice.
wirrbel 6y ago
The whole AI/ML stuff has become so hyped up that its probably time to find another topic of interest in software engineering for me. Its a weird melange nowadays where frameworks and "academic credentials" are fused together by major tech companies and leaves me - who has deployed a dozen of classical ML models into production that are still running after couple of years - wondering what this is all about. Overall, working with people with different backgrounds, a ML-related PhD is usually not correlated nor anti-correlated to these people having a good understanding of the relationship between models and and their applications. I wish we could leave the framework and name-dropping behind and talk more about what it takes to evaluate predictions, how to cope with biases, etc.
- axegon_ 6y ago
  Really familiar territory. I think the hype has poisoned the minds of many and at this state "AI/ML" has turned into a simple buzzword. Much like "blockchain" 2 years ago. And while I'm still as fascinated about ml as I was 5 years ago, like many others, I've decided to stay in the shadows and do my own thing just for the fun of it. Especially since marketing and ego started playing a big role around those communities. It genuinely makes me sad but I think I always knew in the back of my mind that this would likely turn out to be a nail in the coffin of AI/ML, not robots taking over the world. The way I see it, ML/AI is nothing more than a marketing campaign for much of the industry and few people realize that it's often a small component and rarely a major selling point for anything. Like "ml-powered kitchen blender" or whatever. As you said, few people discuss evaluating predictions, tackling biases. I suspect because most people are a lot more interested in snatching a piece of the cake.
  - s1t5 6y ago
    > Much like "blockchain" 2 years ago. It's different. With the ML stuff there's a bunch of actually useful applications and interesting problems at the core with a lot of fluff and marketing piled on top of it. That's the reason why you're seeing the ML/AI hype last so much longer than blockchain (which was basically a quick cash grab with no substance).
echelon 6y ago
I'm doing deep learning without a data science background. Some of my current results are: * https://vo.codes https://vo.codes * https://trumped.com https://trumped.com The voices need better data curation and longer training, but some speakers such as David Attenborough are quite good. I've also built a real time streaming voice conversion system. I want to generalize it better so that it can be an actual product. I think it could be a killer app for Discord. Imagine talking to your friends as Ben Stein or Ninja. I've been watching TTS and VC evolve over the last few years, and it's incredible pace at which things are coming along. There are now singing neural networks that sound better than Vocaloid. If you follow researchers on Github (seriously, their social features are a killer app!), you'll see model after model get uploaded - complete with citations, results. It's super exciting, and it's the future I hoped research would become. If you're diving into this, I would recommend using PyTorch, not TensorFlow. PyTorch is much easier to use and has better library/language support. TorchScript / JIT is really fantastic, too. I even mean this if just you're poking around with someone else's model - find a PyTorch alternative if you can. It's much easier to wrap your head around. TensorFlow is just too obtuse for no good reason.
- amrrs 6y ago
  This is an amazing side project. Would you share some details about how you've hosted them? Also, if possible about the training details like how long did it take for you train them? Did you do it on your local GPU or some cloud provider or free service like Colab?
  - echelon 6y ago
    Thanks! I'm pretty serious, so I've put some money into it. And even more time. I've got a 2x1080Ti setup I used locally back in the day, but it's really slow. I still train stuff on it, but only things I know will train successfully for a long time (eg the Melgan model). I use rented V100 GPUs to train the speaker models. They're quick and allow me to refine the datasets and parameters much more quickly than if I was doing all of it on my own box. Colabs are great and I could probably get along with them if I wasn't running so many experiments in parallel. I can get reasonable results in a few hours on an 8xV100. Once I hone in on a direction I like, I'll let it train for a few days. (The David Attenborough model is a result of this.) I still have a ton of refinement to do. I'm also working on singing models, and these should be ready by the weekend. I've thought about buying beefy GPUs at this point as I've proven to myself it's not just a temporary hobby. Cloud compute is expensive. The models are hosted on Rust microservices (a frontend proxy that fans out into multiple model servers), and this is deployed to a Kubernetes cluster. I'm planning to add more intelligence to the proxy and individual model containers so they independently scale.
totorovirus 6y ago
It is totally possible and easy to use tensorflow/torch if you havn't skipped linear algebra classes. Ph.D is needed if you are going for a job where you design a sophisticated model (not just adding layers, but experimenting with activation, attention etc).
- s1t5 6y ago
  A PhD isn't even a requirement for doing the more advanced stuff. Obviously you need a lot of math and ML specific knowledge but there's no reason why you can't have that knowledge with an undergraduate math degree (for example). Spending 3-6 years doing research in a very narrow and possibly unrelated branch of mathematics will give you a PhD, but the linear algebra and multivariable calculus that you actually need for the ML stuff are covered in a bunch of undergrad/masters courses in mathematics, computer science, engineering, physics etc.
  - lonesword 6y ago
    I second this. I have worked with a bunch of undergrads (I am pursuing a masters degree in CS) and they had a thorough grasp of the math and could really contribute to the research agenda of the group. When I did my undergrad (in 2011-15), I ended up taking a lot of electronics/hardware courses. Turns out undergrads these days just swap them with math/machine learning courses. Good for them.
- teruakohatu 6y ago
  I would say a PhD (or masters degree minimum) is required if you are developing new novel architectures, but not to iterate on existing architectures for a particular application. Applied Data Science does not require developing as yet unseen architectures, being able to read cutting edge papers and applying the research is more than likely to enough.
yumraj 6y ago
Does anyone know what is this site generated with?
DrNuke 6y ago
As usual with tools, it’s the use instead of the instrument. Domain knowledge still has advantages against generalism, in applied and critical fields. I mean, you can’t do medicine or materials yet with AI/ML without understanding the domain?
m0zg 6y ago
Mandatory plug: do consider using PyTorch instead. It's far easier to pick up and work with. Easy things are easy, hard things are possible.
- osipov 6y ago
  Agreed. I used TensorFlow 1.x before PyTorch matured. Today, all of my deep learning is PyTorch based because TensorFlow 2.x didn't fix the core TensorFlow 1.x issues (e.g., poor reproducibility, unintuitive API design) and PyTorch performs better than Keras w/TensorFlow backend.
- jiggunjer 6y ago
  Does it train just as fast on gpus?
  - comicjk 6y ago
    In my experience, not quite as fast for fully-tuned code, but the difference is small - and given the same project deadline, the PyTorch version will probably be faster.
  - m0zg 6y ago
    Properly tuned (DistributedDataParallel + mixed precision) it will train faster, and consume a lot less RAM, allowing you to use larger batches, and higher LR.
- ausjke 6y ago
  I read somewhere saying TF2 is good for production while Pytorch is good for research(and papers), is this true? I'm more interested in putting one of them into real products, esp standalone embedded devices.
  - m0zg 6y ago
    It used to be true, but nowadays there are several options to deploy Pytorch models: 1. PyTorch C++ API which can trivially load and execute your models exported via JIT 2. ONNX export and inference in TensorRT (highest performance inference option) 3. Or just deploy straight up PyTorch Python code - it'll run fine in "production". One place where PyTorch is weaker than TF is mobile. TFLite is a lot more mature and has all sorts of acceleration support (GPU, DSP). So if that's what you need, at this point there's really no other good choice IMO.
gas9S9zw3P9c 6y ago
As a researcher in the field I am not quite sure how I feel about these kind of resources. I am all for making research accessible to a wider audience and I believe that you don't need a PhD, or any degree, to do meaningful work. At the same time, the low barrier of entry and hype has resulted in a huge amount of people downloading Keras, copying a bunch of code, tuning a few parameters, and then putting their result on arXiv so they can put AI research on their resume. This has resulted in so much noise and low quality work that it really hurts the field. You don't need a degree, but I think you do need to spend some time to get a deep enough understanding of what's going on under the hood, which often includes some math and takes time. This can be made accessible and there are plenty of good resources for that. But all these "become an AI pro by looking at some visualizations and copying this code" is maybe hurting more than it helps because it gives the illusion of understanding when it's actually not there. I wouldn't want people learning (solely) from this touching my production systems, writing blogs, or putting papers on arXiv.
- throwgeorge 6y ago
  >putting their result on arXiv so they can put AI research on their resume are you kidding? it's resulted in a huge amount of researchers publishing results in journals so they can apply to funding agencies that are clueless. I collaborate with a national lab (in the us) and the number of LDRD calls that I'm on where classically trained scientists propose completely clueless machine learning projects is very high. so i think you're being quite naive (or disingenuous) about who the real culprits are in inflating arxiv. also you can't throw up stuff on arxiv without a prof vouching for you.
- s1t5 6y ago
  Do you ever feel like all the noise influences the way you think about your own career? I work as a data scientist and sometimes find the hype so off-putting that I think that I should look for a role that's related to solving some optimization problems outside of ML, or a software engineering role in some completely different domain and work as a backend developer or something similar.
  - gas9S9zw3P9c 6y ago
lumberjack 6y ago
Since everyone is talking about hype in ML, I wish there was some hype for good ole' conversional scientific computing. Yes, it's not so sexy, you have to build your own model yourself, and then the hard work is in finding and verifying a suitable numerical method and finally devising a solid implementation. It requires a vast number of different skills, anything from pure math to low level programming and it is definitely not trivial work, but it does not seem like it pays that well.
- LorenzGauge 6y ago
  So excited to start my Masters in Numerical/Scientific Computing for these reasons!
  - lc0_stein 6y ago
    What are some good universities for something like this?
- comicjk 6y ago
  It's also a lot of fun. For anyone that likes math, but also likes weird approximations, there's all kinds of juicy stuff in scientific computing. It will make you a better ML coder, too.
daiyanze 6y ago
Is this tutorial aiming at entry level? Those graphs are quite difficult. I guess it will take a lot of time to do homework on some fundamental curriculums.
robpal 6y ago
I attended a conference talk of a FB AI-engineer talking about her paper with backprop equations so obviously wrong my eyes hurt, and incorrect definitions of objects. It did not stop her from participating (btw. this is always unclear -- who did what) in state-of-the art research in object detection. PhD is overrated in the deep learning context. It is more about forging the intellectual resilience and ability to pursue ideas for months/years than learning useful things/tricks/theorems.
- Donthatme 6y ago
  > It is more about forging the intellectual resilience and ability to pursue ideas for months/years ... Isn't this what a PhD generically signals?
- sgillen 6y ago
  Don't underestimate useful things, tricks, and theorems though ;)
secondcoming 6y ago
It's a pity that most Tensorflow tutorials out there seem to deal with images. We tried to use it for real-time data classification (data -> [yes | no]). Every tuturial out there seems to assume you're using Python (which is probably not an invalid assumption). Here's my 2c when trying to use Tensorflow with C++: a) Loading SavedModels is a pain. I has to trawl the Tensorflow repo and Python wrappers to see how it worked. b) It's incredibly slow. It added ~250ms to our latency. We had to drop it. c) It has a C++ framework that doesn't work out-of-the-box, you have to use the C lib that wraps an old version of the C++ framework (confused? me too). d) It's locked to C++03. Tensorflow-Lite looked to fit the bill for us, but our model weren't convertible to it. We no longer use Tensorflow.
- osipov 6y ago
  I don't understand why you are getting downvoted. TensorFlow 1.x barely worked and people stuck with it because the alternatives were worse. I moved to PyTorch as soon as I could because it is better than TensorFlow 1.x, TensorFlow 2.x, or Keras w/TensorFlow backend. TensorFlow is designed by committee and is more of a brand now than a machine learning framework. Did you know there is TensorFlow Quantum?
  - deadliftpro 6y ago
    There were alternatives. IMO, Microsoft Cntk being one which was better than tensorflow. Tensorflow 'won' because of herd mentality and the perception that everything google does is cool.
- lostmsu 6y ago
  I am using Python's TensorFlow API from C# through my own binding, and I don't understand how you got 250ms latency on C API without screwing up on your side. I could effortlessly run a super real-time network playing a video game with soft actor-critic with my setup on 1.15.
  - secondcoming 6y ago
    We didn't understand either. One thing, our cloud instances had no GPU's. Our model wasn't exactly complex, just maybe 15 float and string scalar inputs.
seek3r00 6y ago
The title is obviously clickbait-y, but it’s fine: they’re trying to sell a product (Google Colab). IMO if you’re interested in AI research or ML engineering, you already know that — in order to avoid getting people killed - you have to understand how it works under-the-hood. You’re doing yourself, your employer and your fellow humans a favour. Just keep up the good work, and ignore the bullshit. If an AI winter comes, you’ll be well prepared to migrate to another engineering role.
polskibus 6y ago
I wonder if Google is using this resource to train their own staff without PhD and after that allow them to work as ML engineers? That would provide credibility to such program - instead it is more aimed to sell more ML computing power to the masses (who won't really understand how to use it to get meaningful results).
- nxpnsv 6y ago
  not bloody likely...
matlo 6y ago
This is a nice introduction, even though as most of the tutorials on ML it goes from 0 to 100 in 2 lessons. A couple years ago I started studying ML, and I have a design background, so I needed to digest all the math and concepts slowly in order to understand them properly. Now I think I understand most of the fundamental concepts, and I've been using it quite a lot for creative applications and teaching, and I have to say the best resource I've found for beginners, by far, is "Make Your Own Neural Network" by Tariq Rashid. It starts really from the beginning and it takes you through all the steps of building a NN from zero, with no previous knowledge. really good.
noetokyo 6y ago
It's always interesting to see the dunning kruger effect in hn comments.
sibmike 6y ago
I am constantly puzzled by people saying that AI is overhyped and fresh grads won't have enough jobs for them. Almost every real life industry: retail, logistics, construction, farming, heavy industries, mining, medicine have just recently started to try AI. The amount of manual and suboptimal tasks that have to be automated and optimized is enormous. I am pretty sure there is more the enough work for applied DSs with domain knowledge in mentioned industries.
- CabSauce 6y ago
  I agree with this totally. The amount of benefit we've seen in the last 5 years due to ML model improvements is staggering. There is huge potential to apply even pre-trained models in tons of industries. BERT + fine tuning solves domain-specific NLP problems far better than the cutting edge research of a couple years ago. That being said, the business / marketing use of terms like "AI" is way out of control.
- andrewnc 6y ago
  Agreed, I feel we are just barely scratching the surface. I'm fairly involved in the AI/ML world (research at FAANG) and the amount of hype definitely scares me, but ML/AI isn't going anywhere soon imo.
dekhn 6y ago
Twenty five years ago, this would have been "LINUX, UNIX, and serving, without a PhD" and Matt Welsh's Linux Installation And Getting Started was the intro (https://www.mdw.la/papers/linux-getting-started.pdf https://www.mdw.la/papers/linux-getting-started.pdf). I was one of many who adopted Linux early, using this book (later I read the BSD Unix Design and Implementation, which I would describe as senior undergrad/junior grad student material). Having those sorts of resources to introduce junior folks to advanced concepts are really great to me- my experience is that I learn a lot more by reading a good tutorial than a theory book, up until I need to do advanced work (this is particular to my style of learning; I can read code that implements math, but struggle to parse math symbology).
kadder 6y ago
Machine learning Academia, is very different from how its applied in a real world setting. The number of people who can launch a model and deal with the intricacies of serving a model in production are still very few. Resources like these are essential for people not familiar with ML. We need more people who can apply successfully at scale. A top down approach to learning ML in my opinion is more efficient than a bottom's up approach, that most people try to advocate
Guest42 6y ago
To me it seems the real goal of hyping these things so heavily is to increase their cloud revenues.
historyremade 6y ago
Pewwwwwwww....Soooo much for brute forcing...
arunoda 6y ago
This is something I did with fast.ai - https://deeplearningmantra.com/ https://deeplearningmantra.com/
mark_l_watson 6y ago
This is very well done, hitting on some pain points and explaining how to work around them. I have devoted close to 100% of my paid working time on deep learning for the last six years (most recently managing a deep learning team) and not only has the technology advanced rapidly but the online learning resources have kept pace. A personal issue: after seven years of not updating my Java AI book, I am taking advantage of free time at home to do a major update. New material on deep learning was the most difficult change because there are so many great resources, and there is only so much you can do in one long chapter. I ended up deciding to do just two DL4J examples and then spending most of the chapter just on advice. The field of deep learning is getting saturated. Recently I did a free mentoring session with someone with a very good educational background (PhD from MIT) and we were talking about needing specializations and specific skills, just using things like DL, cloud dev ops, etc. as necessary tools, but not always enough to base a career on. Definitely it can help peoples’ careers working through great online DL material, but great careers are usually made by having good expertise in two or three areas. Learn DL but combine that with other skills and domain knowledge.

392 points by blopeur 6y ago

Tangentially related (and also using the ubiquitous MNIST dataset), Sebastian Lague started a brilliant, but unfortunately unfinished video series on building neural networks from scratch. This video was an absolute eye-opener me [1] on what classification is, how it works and why a non-linear activation function is required. I probably learned more in the 5 minutes watching this than doing multiple Coursera courses on the subject. [1] https://www.youtube.com/watch?v=bVQUSndDllU https://www.youtube.com/watch?v=bVQUSndDllU

pests 6y ago
One "ah ha" about that required non-linear function I had was the fact that if you were just passing numbers through a series of linear functions they could by definition just be combined into one equation.

halflings 6y ago

The video version [1] is also pretty awesome, though its code itself is a bit outdated now. Explains a lot of very practical issues that you might not find in most academic textbooks, but you encounter every day in practice. [1] https://www.youtube.com/watch?v=vq2nnJ4g6N0 https://www.youtube.com/watch?v=vq2nnJ4g6N0

amrx101 6y ago

Have a simple rant here. All these BIG $ companies every now and then come out with statements and what not, that doing AI ML is very easy and every one including their cats should do AI, ML courses and training(preferably on their platform). Once that is done the job market is yours. Reality is far from this. - Today AI|ML does not have the capability marketed by these big companies. Incidentally marketing is targeted at governments, big non tech companies and gullible undergraduates. - Undergrads many a times take these training courses wherein they acquire the skill set to call these APIs and flood the job market wherein a data-entry or data-analyst job is tagged as AI|ML job. - High paying jobs in AI|ML still require a Masters or PhD or a mathematical background. In conclusion, the current hype around AI|ML is misguiding gullible undergrads and governments(I dont mind the government being cheated THOUGH).

mpfundstein 6y ago
i am doing quite some deep learning work as part of my consultancy practice. And its all hand made stuff , together with lots of trial and error while trying to replicate papers that might be relevant fkr the task at hand. so i totally agree with your statement. big corps overhype the shit out of it in order to sell and lots of n00bs fail for it. regarding your last points. even if its super easy nowadays to deploy YOLO, and everyone and his mother can do it, making actually something that works and provides business value is hardcore. and without scientific skills_> no chance
- lordthirdeye 6y ago
  if you don't mind me asking, what do you mean by "hand made". I'm currently wworking on a information science degree and I'm trying to focus on machine learning and data science. Could you go into a little more detail on what gives something business value?
  - disgruntledphd2 6y ago
    Not the OP, but ultimately something that increases revenue or decreases costs by some measurable amount. The best thing you can do to make yourself good at this is to practice. Get some Kaggle data and try to fit a model. Realise your data is crap, clean data, repeat. Every useful system is hand made in the sense that there's a vast amount of set up and operational code. Mostly the model's the easy part (although it will take so much time to run).
  - mpfundstein 6y ago
    hand-make as in: - looking very carefully into the very specific challenge of your client - figuring out how (and if) ML can help - figuring if its still economically feasible (costs of research vs perceived(!) benefit) - deriving a solution. - tinkering tinkering tinkering. usually more with the data than with the models :-) All my A.I. projects are essentially outsourced R&D projects where we deliver the brain and computing power. So far, it never was as easy like installing YOLO or any other off the shelf product. Edit: You also need very often custom software to create custom datasets. AI models are often only tested on academic datasets but I observed empirically that their performance transfers badly to real world datasets. So you need to create your own datasets etc. This is often a non-trivial problem. So I wrote a lot of dataset creation tools in my AI practice.

wirrbel 6y ago

The whole AI/ML stuff has become so hyped up that its probably time to find another topic of interest in software engineering for me. Its a weird melange nowadays where frameworks and "academic credentials" are fused together by major tech companies and leaves me - who has deployed a dozen of classical ML models into production that are still running after couple of years - wondering what this is all about. Overall, working with people with different backgrounds, a ML-related PhD is usually not correlated nor anti-correlated to these people having a good understanding of the relationship between models and and their applications. I wish we could leave the framework and name-dropping behind and talk more about what it takes to evaluate predictions, how to cope with biases, etc.

axegon_ 6y ago
Really familiar territory. I think the hype has poisoned the minds of many and at this state "AI/ML" has turned into a simple buzzword. Much like "blockchain" 2 years ago. And while I'm still as fascinated about ml as I was 5 years ago, like many others, I've decided to stay in the shadows and do my own thing just for the fun of it. Especially since marketing and ego started playing a big role around those communities. It genuinely makes me sad but I think I always knew in the back of my mind that this would likely turn out to be a nail in the coffin of AI/ML, not robots taking over the world. The way I see it, ML/AI is nothing more than a marketing campaign for much of the industry and few people realize that it's often a small component and rarely a major selling point for anything. Like "ml-powered kitchen blender" or whatever. As you said, few people discuss evaluating predictions, tackling biases. I suspect because most people are a lot more interested in snatching a piece of the cake.
- s1t5 6y ago
  > Much like "blockchain" 2 years ago. It's different. With the ML stuff there's a bunch of actually useful applications and interesting problems at the core with a lot of fluff and marketing piled on top of it. That's the reason why you're seeing the ML/AI hype last so much longer than blockchain (which was basically a quick cash grab with no substance).

echelon 6y ago

I'm doing deep learning without a data science background. Some of my current results are: * https://vo.codes https://vo.codes * https://trumped.com https://trumped.com The voices need better data curation and longer training, but some speakers such as David Attenborough are quite good. I've also built a real time streaming voice conversion system. I want to generalize it better so that it can be an actual product. I think it could be a killer app for Discord. Imagine talking to your friends as Ben Stein or Ninja. I've been watching TTS and VC evolve over the last few years, and it's incredible pace at which things are coming along. There are now singing neural networks that sound better than Vocaloid. If you follow researchers on Github (seriously, their social features are a killer app!), you'll see model after model get uploaded - complete with citations, results. It's super exciting, and it's the future I hoped research would become. If you're diving into this, I would recommend using PyTorch, not TensorFlow. PyTorch is much easier to use and has better library/language support. TorchScript / JIT is really fantastic, too. I even mean this if just you're poking around with someone else's model - find a PyTorch alternative if you can. It's much easier to wrap your head around. TensorFlow is just too obtuse for no good reason.

amrrs 6y ago
This is an amazing side project. Would you share some details about how you've hosted them? Also, if possible about the training details like how long did it take for you train them? Did you do it on your local GPU or some cloud provider or free service like Colab?
- echelon 6y ago
  Thanks! I'm pretty serious, so I've put some money into it. And even more time. I've got a 2x1080Ti setup I used locally back in the day, but it's really slow. I still train stuff on it, but only things I know will train successfully for a long time (eg the Melgan model). I use rented V100 GPUs to train the speaker models. They're quick and allow me to refine the datasets and parameters much more quickly than if I was doing all of it on my own box. Colabs are great and I could probably get along with them if I wasn't running so many experiments in parallel. I can get reasonable results in a few hours on an 8xV100. Once I hone in on a direction I like, I'll let it train for a few days. (The David Attenborough model is a result of this.) I still have a ton of refinement to do. I'm also working on singing models, and these should be ready by the weekend. I've thought about buying beefy GPUs at this point as I've proven to myself it's not just a temporary hobby. Cloud compute is expensive. The models are hosted on Rust microservices (a frontend proxy that fans out into multiple model servers), and this is deployed to a Kubernetes cluster. I'm planning to add more intelligence to the proxy and individual model containers so they independently scale.

totorovirus 6y ago

It is totally possible and easy to use tensorflow/torch if you havn't skipped linear algebra classes. Ph.D is needed if you are going for a job where you design a sophisticated model (not just adding layers, but experimenting with activation, attention etc).

s1t5 6y ago
A PhD isn't even a requirement for doing the more advanced stuff. Obviously you need a lot of math and ML specific knowledge but there's no reason why you can't have that knowledge with an undergraduate math degree (for example). Spending 3-6 years doing research in a very narrow and possibly unrelated branch of mathematics will give you a PhD, but the linear algebra and multivariable calculus that you actually need for the ML stuff are covered in a bunch of undergrad/masters courses in mathematics, computer science, engineering, physics etc.
- lonesword 6y ago
  I second this. I have worked with a bunch of undergrads (I am pursuing a masters degree in CS) and they had a thorough grasp of the math and could really contribute to the research agenda of the group. When I did my undergrad (in 2011-15), I ended up taking a lot of electronics/hardware courses. Turns out undergrads these days just swap them with math/machine learning courses. Good for them.
teruakohatu 6y ago
I would say a PhD (or masters degree minimum) is required if you are developing new novel architectures, but not to iterate on existing architectures for a particular application. Applied Data Science does not require developing as yet unseen architectures, being able to read cutting edge papers and applying the research is more than likely to enough.

yumraj 6y ago

Does anyone know what is this site generated with?

DrNuke 6y ago

As usual with tools, it’s the use instead of the instrument. Domain knowledge still has advantages against generalism, in applied and critical fields. I mean, you can’t do medicine or materials yet with AI/ML without understanding the domain?

m0zg 6y ago

Mandatory plug: do consider using PyTorch instead. It's far easier to pick up and work with. Easy things are easy, hard things are possible.

osipov 6y ago
Agreed. I used TensorFlow 1.x before PyTorch matured. Today, all of my deep learning is PyTorch based because TensorFlow 2.x didn't fix the core TensorFlow 1.x issues (e.g., poor reproducibility, unintuitive API design) and PyTorch performs better than Keras w/TensorFlow backend.
jiggunjer 6y ago
Does it train just as fast on gpus?
- comicjk 6y ago
  In my experience, not quite as fast for fully-tuned code, but the difference is small - and given the same project deadline, the PyTorch version will probably be faster.
- m0zg 6y ago
  Properly tuned (DistributedDataParallel + mixed precision) it will train faster, and consume a lot less RAM, allowing you to use larger batches, and higher LR.
ausjke 6y ago
I read somewhere saying TF2 is good for production while Pytorch is good for research(and papers), is this true? I'm more interested in putting one of them into real products, esp standalone embedded devices.
- m0zg 6y ago
  It used to be true, but nowadays there are several options to deploy Pytorch models: 1. PyTorch C++ API which can trivially load and execute your models exported via JIT 2. ONNX export and inference in TensorRT (highest performance inference option) 3. Or just deploy straight up PyTorch Python code - it'll run fine in "production". One place where PyTorch is weaker than TF is mobile. TFLite is a lot more mature and has all sorts of acceleration support (GPU, DSP). So if that's what you need, at this point there's really no other good choice IMO.

gas9S9zw3P9c 6y ago

As a researcher in the field I am not quite sure how I feel about these kind of resources. I am all for making research accessible to a wider audience and I believe that you don't need a PhD, or any degree, to do meaningful work. At the same time, the low barrier of entry and hype has resulted in a huge amount of people downloading Keras, copying a bunch of code, tuning a few parameters, and then putting their result on arXiv so they can put AI research on their resume. This has resulted in so much noise and low quality work that it really hurts the field. You don't need a degree, but I think you do need to spend some time to get a deep enough understanding of what's going on under the hood, which often includes some math and takes time. This can be made accessible and there are plenty of good resources for that. But all these "become an AI pro by looking at some visualizations and copying this code" is maybe hurting more than it helps because it gives the illusion of understanding when it's actually not there. I wouldn't want people learning (solely) from this touching my production systems, writing blogs, or putting papers on arXiv.

throwgeorge 6y ago
>putting their result on arXiv so they can put AI research on their resume are you kidding? it's resulted in a huge amount of researchers publishing results in journals so they can apply to funding agencies that are clueless. I collaborate with a national lab (in the us) and the number of LDRD calls that I'm on where classically trained scientists propose completely clueless machine learning projects is very high. so i think you're being quite naive (or disingenuous) about who the real culprits are in inflating arxiv. also you can't throw up stuff on arxiv without a prof vouching for you.
s1t5 6y ago
Do you ever feel like all the noise influences the way you think about your own career? I work as a data scientist and sometimes find the hype so off-putting that I think that I should look for a role that's related to solving some optimization problems outside of ML, or a software engineering role in some completely different domain and work as a backend developer or something similar.
- gas9S9zw3P9c 6y ago

lumberjack 6y ago

Since everyone is talking about hype in ML, I wish there was some hype for good ole' conversional scientific computing. Yes, it's not so sexy, you have to build your own model yourself, and then the hard work is in finding and verifying a suitable numerical method and finally devising a solid implementation. It requires a vast number of different skills, anything from pure math to low level programming and it is definitely not trivial work, but it does not seem like it pays that well.

LorenzGauge 6y ago
So excited to start my Masters in Numerical/Scientific Computing for these reasons!
- lc0_stein 6y ago
  What are some good universities for something like this?
comicjk 6y ago
It's also a lot of fun. For anyone that likes math, but also likes weird approximations, there's all kinds of juicy stuff in scientific computing. It will make you a better ML coder, too.

daiyanze 6y ago

Is this tutorial aiming at entry level? Those graphs are quite difficult. I guess it will take a lot of time to do homework on some fundamental curriculums.

robpal 6y ago

I attended a conference talk of a FB AI-engineer talking about her paper with backprop equations so obviously wrong my eyes hurt, and incorrect definitions of objects. It did not stop her from participating (btw. this is always unclear -- who did what) in state-of-the art research in object detection. PhD is overrated in the deep learning context. It is more about forging the intellectual resilience and ability to pursue ideas for months/years than learning useful things/tricks/theorems.

Donthatme 6y ago
> It is more about forging the intellectual resilience and ability to pursue ideas for months/years ... Isn't this what a PhD generically signals?
sgillen 6y ago
Don't underestimate useful things, tricks, and theorems though ;)

secondcoming 6y ago

It's a pity that most Tensorflow tutorials out there seem to deal with images. We tried to use it for real-time data classification (data -> [yes | no]). Every tuturial out there seems to assume you're using Python (which is probably not an invalid assumption). Here's my 2c when trying to use Tensorflow with C++: a) Loading SavedModels is a pain. I has to trawl the Tensorflow repo and Python wrappers to see how it worked. b) It's incredibly slow. It added ~250ms to our latency. We had to drop it. c) It has a C++ framework that doesn't work out-of-the-box, you have to use the C lib that wraps an old version of the C++ framework (confused? me too). d) It's locked to C++03. Tensorflow-Lite looked to fit the bill for us, but our model weren't convertible to it. We no longer use Tensorflow.

osipov 6y ago
I don't understand why you are getting downvoted. TensorFlow 1.x barely worked and people stuck with it because the alternatives were worse. I moved to PyTorch as soon as I could because it is better than TensorFlow 1.x, TensorFlow 2.x, or Keras w/TensorFlow backend. TensorFlow is designed by committee and is more of a brand now than a machine learning framework. Did you know there is TensorFlow Quantum?
- deadliftpro 6y ago
  There were alternatives. IMO, Microsoft Cntk being one which was better than tensorflow. Tensorflow 'won' because of herd mentality and the perception that everything google does is cool.
lostmsu 6y ago
I am using Python's TensorFlow API from C# through my own binding, and I don't understand how you got 250ms latency on C API without screwing up on your side. I could effortlessly run a super real-time network playing a video game with soft actor-critic with my setup on 1.15.
- secondcoming 6y ago
  We didn't understand either. One thing, our cloud instances had no GPU's. Our model wasn't exactly complex, just maybe 15 float and string scalar inputs.

seek3r00 6y ago

The title is obviously clickbait-y, but it’s fine: they’re trying to sell a product (Google Colab). IMO if you’re interested in AI research or ML engineering, you already know that — in order to avoid getting people killed - you have to understand how it works under-the-hood. You’re doing yourself, your employer and your fellow humans a favour. Just keep up the good work, and ignore the bullshit. If an AI winter comes, you’ll be well prepared to migrate to another engineering role.

polskibus 6y ago

I wonder if Google is using this resource to train their own staff without PhD and after that allow them to work as ML engineers? That would provide credibility to such program - instead it is more aimed to sell more ML computing power to the masses (who won't really understand how to use it to get meaningful results).

nxpnsv 6y ago
not bloody likely...

matlo 6y ago

This is a nice introduction, even though as most of the tutorials on ML it goes from 0 to 100 in 2 lessons. A couple years ago I started studying ML, and I have a design background, so I needed to digest all the math and concepts slowly in order to understand them properly. Now I think I understand most of the fundamental concepts, and I've been using it quite a lot for creative applications and teaching, and I have to say the best resource I've found for beginners, by far, is "Make Your Own Neural Network" by Tariq Rashid. It starts really from the beginning and it takes you through all the steps of building a NN from zero, with no previous knowledge. really good.

noetokyo 6y ago

It's always interesting to see the dunning kruger effect in hn comments.

sibmike 6y ago

I am constantly puzzled by people saying that AI is overhyped and fresh grads won't have enough jobs for them. Almost every real life industry: retail, logistics, construction, farming, heavy industries, mining, medicine have just recently started to try AI. The amount of manual and suboptimal tasks that have to be automated and optimized is enormous. I am pretty sure there is more the enough work for applied DSs with domain knowledge in mentioned industries.

CabSauce 6y ago
I agree with this totally. The amount of benefit we've seen in the last 5 years due to ML model improvements is staggering. There is huge potential to apply even pre-trained models in tons of industries. BERT + fine tuning solves domain-specific NLP problems far better than the cutting edge research of a couple years ago. That being said, the business / marketing use of terms like "AI" is way out of control.
andrewnc 6y ago
Agreed, I feel we are just barely scratching the surface. I'm fairly involved in the AI/ML world (research at FAANG) and the amount of hype definitely scares me, but ML/AI isn't going anywhere soon imo.

dekhn 6y ago

Twenty five years ago, this would have been "LINUX, UNIX, and serving, without a PhD" and Matt Welsh's Linux Installation And Getting Started was the intro (https://www.mdw.la/papers/linux-getting-started.pdf https://www.mdw.la/papers/linux-getting-started.pdf). I was one of many who adopted Linux early, using this book (later I read the BSD Unix Design and Implementation, which I would describe as senior undergrad/junior grad student material). Having those sorts of resources to introduce junior folks to advanced concepts are really great to me- my experience is that I learn a lot more by reading a good tutorial than a theory book, up until I need to do advanced work (this is particular to my style of learning; I can read code that implements math, but struggle to parse math symbology).

kadder 6y ago

Machine learning Academia, is very different from how its applied in a real world setting. The number of people who can launch a model and deal with the intricacies of serving a model in production are still very few. Resources like these are essential for people not familiar with ML. We need more people who can apply successfully at scale. A top down approach to learning ML in my opinion is more efficient than a bottom's up approach, that most people try to advocate

Guest42 6y ago

To me it seems the real goal of hyping these things so heavily is to increase their cloud revenues.

historyremade 6y ago

Pewwwwwwww....Soooo much for brute forcing...

arunoda 6y ago

This is something I did with fast.ai - https://deeplearningmantra.com/ https://deeplearningmantra.com/

mark_l_watson 6y ago

This is very well done, hitting on some pain points and explaining how to work around them. I have devoted close to 100% of my paid working time on deep learning for the last six years (most recently managing a deep learning team) and not only has the technology advanced rapidly but the online learning resources have kept pace. A personal issue: after seven years of not updating my Java AI book, I am taking advantage of free time at home to do a major update. New material on deep learning was the most difficult change because there are so many great resources, and there is only so much you can do in one long chapter. I ended up deciding to do just two DL4J examples and then spending most of the chapter just on advice. The field of deep learning is getting saturated. Recently I did a free mentoring session with someone with a very good educational background (PhD from MIT) and we were talking about needing specializations and specific skills, just using things like DL, cloud dev ops, etc. as necessary tools, but not always enough to base a career on. Definitely it can help peoples’ careers working through great online DL material, but great careers are usually made by having good expertise in two or three areas. Learn DL but combine that with other skills and domain knowledge.