5 ms·

Congratulations on getting first rank on front page. Congratulations on getting your hands dirty and doing everything yourself like computing gradients manuall

Congratulations on getting first rank on front page. Congratulations on getting your hands dirty and doing everything yourself like computing gradients manually, badly shuffling (non Fisher-Yates), badly js transpose (double swapping), it is a great way to learn. Congratulations on completing a full pipeline, that's the hard part then it just swapping pieces for better pieces. I advise non-technical readers not to attach much value to the results of this neural network as it is probably inferior to the even simpler naive Bayes. The model of the neural network is simplistic : Concat(Word Vectors)-Dense(120,act=sigmoid)-Dense(60,act=sigmoid)-Dense(2,act=sigmoid) The Concat operation mean it is especially sensible to dropping or adding a word as it will offset the remaining words and give a totally different vector. Using word vectors mean it doesn't forgive any spelling mistake as a spelling mistake will usually correspond to <unknown> vector. Using a feed forward neural network means formulaic titles with a single word substitution from a good positive example from the training set will often work. It is trained by gradient descent using a squared error loss, on ~1000 examples one example at a time without cross-validation using a custom written neural network library. (Almost all these bad choices can be solved by using a framework). It seems to have successfully over-fit as it return Good ~1.0 for positive examples from the training set.

holbue 7y ago
Thank you for summarising and thx to OP for publishing his work. The combination of both was a nice opportunity for me (and others?) to learn something from mistakes. :-)
codesushi42 7y ago
This. Exactly this. No sophisticated tokenization. No interesting architecture using attention. And the author is completely clueless about overfitting... and even cross entropy loss. He could have gotten better results just using a bag of words approach. But this ends up on frontpage anyway. Welcome to HN.
- objektif 7y ago
  What tools would you use to detect overfitting in this case and in general?
  - codesushi42 7y ago
    My brain. You will overfit an NN trained on only 1000 examples. Also a simple train/test split will tell you that. But the author failed to take any time to learn the basics before spewing out this drivel.
    - dang 7y ago
      By now it's clear from your history of behaving aggressively in HN threads that you don't mean to use the site as intended. I've banned the account. https://news.ycombinator.com/item?id=21245546 https://news.ycombinator.com/item?id=21245546 I'm sad to do that because you are knowledgeable about a number of things. Many of us could learn from you if you would share what you know without putting other people down. But the aggression subtracts more than the knowledge adds. We can't have users behaving like this sort of asshole in comments, least of all in Show HN threads, where the idea is to teach people things and expressly not to shit on them and their work: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html. Other users here know things and are willing to talk about them without being mean. GistNoesis modeled this wonderfully in the GP comment. We'll learn what we can from them instead. But if you decide that you want to use HN in the intended spirit, as described in the site guidelines and especially the one that says Be kind, you're welcome to email hn@ycombinator.com and let us know. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
wodenokoto 7y ago
This really puts the work in perspective. Thank you for the summary!
deleted 7y ago
[deleted]
atum47 7y ago
I really like this comment. Thanks a lot for taking the time to going through the code and figure all that out. The way I learn things is by doing. Since I started college, the only way I can understand a mathematical equation is by turning it into code. The machine learning class I took was a basic one. We learned about knn, kmeans, adaline, perception, linear regression... the semester ended with a multilayer perceptron with only one hidden layer. This Dejavu NN you see in my code is like the fourth iteration of me trying to really understand NNs. Some of the things you said I already knew but didn't took the time to apply: like cross validation. Tht shuffle algorithm I didn't knew, thanks for that. Anyways, if all the criticism I got from my projects were like yours i'd be such much better now. When I learned about KNN, I made this project: https://github.com/victorqribeiro/budget https://github.com/victorqribeiro/budget When I learned about Kmeans, I made this one: https://github.com/victorqribeiro/groupImg https://github.com/victorqribeiro/groupImg When I learned about Perceptrons: https://github.com/victorqribeiro/carGamePerceptron https://github.com/victorqribeiro/carGamePerceptron (the training is done when you run the project) When I learned about MLP: https://github.com/victorqribeiro/jokenpo https://github.com/victorqribeiro/jokenpo When I learned about Neural Evolution: https://github.com/victorqribeiro/aimAndShoot https://github.com/victorqribeiro/aimAndShoot None of these projects are perfect, but helped me to better understand things. I did implemented a professional solution for a company using machine learning, but I used scikit-learn (back then). I have some experience with tensorflow also, but implementing things makes me fell like in control. But I know whena a solution is good enough for production and when it's not. I made clear on the README of this project that this is not at all good for production. I took the time to show every single step of the process and tell it how it is. I had limited time and limited resources, and with the little I had I made a "fun" experiment. I enjoy doing this kind of thing. I have to squish this kind of projects in my free time, cause I have a full time job and I'm finishing my final thesis (I'll defend it two week from now). Sorry if I turn this into a journal entry, but I kind got hurt yesterday when the other guy shted all over my project That's not at all what you did and I appreciate that. Thanks again.

holbue 7y ago

Thank you for summarising and thx to OP for publishing his work. The combination of both was a nice opportunity for me (and others?) to learn something from mistakes. :-)

codesushi42 7y ago

This. Exactly this. No sophisticated tokenization. No interesting architecture using attention. And the author is completely clueless about overfitting... and even cross entropy loss. He could have gotten better results just using a bag of words approach. But this ends up on frontpage anyway. Welcome to HN.

objektif 7y ago
What tools would you use to detect overfitting in this case and in general?
- codesushi42 7y ago
  My brain. You will overfit an NN trained on only 1000 examples. Also a simple train/test split will tell you that. But the author failed to take any time to learn the basics before spewing out this drivel.
  - dang 7y ago
    By now it's clear from your history of behaving aggressively in HN threads that you don't mean to use the site as intended. I've banned the account. https://news.ycombinator.com/item?id=21245546 https://news.ycombinator.com/item?id=21245546 I'm sad to do that because you are knowledgeable about a number of things. Many of us could learn from you if you would share what you know without putting other people down. But the aggression subtracts more than the knowledge adds. We can't have users behaving like this sort of asshole in comments, least of all in Show HN threads, where the idea is to teach people things and expressly not to shit on them and their work: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html. Other users here know things and are willing to talk about them without being mean. GistNoesis modeled this wonderfully in the GP comment. We'll learn what we can from them instead. But if you decide that you want to use HN in the intended spirit, as described in the site guidelines and especially the one that says Be kind, you're welcome to email hn@ycombinator.com and let us know. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

wodenokoto 7y ago

This really puts the work in perspective. Thank you for the summary!

deleted 7y ago

[deleted]

atum47 7y ago

I really like this comment. Thanks a lot for taking the time to going through the code and figure all that out. The way I learn things is by doing. Since I started college, the only way I can understand a mathematical equation is by turning it into code. The machine learning class I took was a basic one. We learned about knn, kmeans, adaline, perception, linear regression... the semester ended with a multilayer perceptron with only one hidden layer. This Dejavu NN you see in my code is like the fourth iteration of me trying to really understand NNs. Some of the things you said I already knew but didn't took the time to apply: like cross validation. Tht shuffle algorithm I didn't knew, thanks for that. Anyways, if all the criticism I got from my projects were like yours i'd be such much better now. When I learned about KNN, I made this project: https://github.com/victorqribeiro/budget https://github.com/victorqribeiro/budget When I learned about Kmeans, I made this one: https://github.com/victorqribeiro/groupImg https://github.com/victorqribeiro/groupImg When I learned about Perceptrons: https://github.com/victorqribeiro/carGamePerceptron https://github.com/victorqribeiro/carGamePerceptron (the training is done when you run the project) When I learned about MLP: https://github.com/victorqribeiro/jokenpo https://github.com/victorqribeiro/jokenpo When I learned about Neural Evolution: https://github.com/victorqribeiro/aimAndShoot https://github.com/victorqribeiro/aimAndShoot None of these projects are perfect, but helped me to better understand things. I did implemented a professional solution for a company using machine learning, but I used scikit-learn (back then). I have some experience with tensorflow also, but implementing things makes me fell like in control. But I know whena a solution is good enough for production and when it's not. I made clear on the README of this project that this is not at all good for production. I took the time to show every single step of the process and tell it how it is. I had limited time and limited resources, and with the little I had I made a "fun" experiment. I enjoy doing this kind of thing. I have to squish this kind of projects in my free time, cause I have a full time job and I'm finishing my final thesis (I'll defend it two week from now). Sorry if I turn this into a journal entry, but I kind got hurt yesterday when the other guy shted all over my project That's not at all what you did and I appreciate that. Thanks again.