15 ms·

The First Rule of Machine Learning: Start Without Machine Learning

764 points by 7d7n 5y ago

xnx 5y ago
I haven't read the article, but I really like the construction of the phrase. I would also propose: First rule of optimization: Don't optimize first. First rule of automation: Don't automate first.
- refactor_master 5y ago
  “This is just an agile PoC. It’s not meant to be readable, performant or documented”. The first rule of everything should be “it depends”.
  - AussieWog93 5y ago
    I propose instead the first rule of rules: Don't assume that general advice will be applicable to all circumstances. :P
elexhobby 5y ago
Furthermore, follow https://twitter.com/_brohrer_/status/1425770502321283073 https://twitter.com/_brohrer_/status/1425770502321283073 "When you have a problem, build two solutions - a deep Bayesian transformer running on multicloud Kubernetes and a SQL query built on a stack of egregiously oversimplifying assumptions. Put one on your resume, the other in production. Everyone goes home happy."
- arketyp 5y ago
  Furthermore in the article, yes.
- hughrr 5y ago
  This reminds me of an experience I had watching a company trying to replace a system with ML. First they marketed it heavily before even thinking. During test cycle they fed the entire data corpus in and ran some of the original test cases and found some business destroying results pop out. The entire system ended up a verbatim port of the VB6 crap which was a verbatim port of the original AS400 crap that actually worked. The marketing to this day says it’s ML based and everyone buys into the hype. It’s not. It was a complete failure. But the original system has 30 years of human experience codified in it.
  - ethbr0 5y ago
    If I had a nickel for every time I've seen "business rules engine" turned into "AI" in the last few years... But I guess if we complain that half of our colleagues and the media don't understand ML, why should we expect management to? When the command from C-level is "We need some AI projects to tell our shareholders about," we shouldn't be surprised when middle management suddenly has successful AI projects in their slide decks.
    - mjburgess 5y ago
      Well really-existing AI is just either "taking a mean()" or "programming a rule". So all of programming actually counts as (symbolic) AI.
    - cm2187 5y ago
      In the same way the terms “blockchain” being used for “digital signing” or “cloud” for a server...
    - i_am_proteus 5y ago
      If you have an existing rules-based decision-tree system, and you compare its performance with a bunch of other decision trees, and it does better, you are implementing a random forest that happens to be identical to your original system. Artificial Intelligence.
      - mumblemumble 5y ago
        If you talk enough all the models and hyperparameters you compared and suchlike that you experimented with, you can probably sufficiently impress people with the talk about the enormous deep learning model you spent several months developing that they won't even remember you mentioning the two-clause Boolean expression that you actually put into production. And of course it's AI. You used k-fold cross validation to select it.
  - pedrocr 5y ago
    The AI taxonomy includes the term "Expert Systems" for these kinds of things. On the one hand it's definitely not of the new wave of ML AI so hyping those things as innovative is off. On the other hand we should definitely give more attention to that kind of setup and understand how to build/maintain/test it properly. Otherwise often it ends up being ran by a few hundred Excel sheets and a few severely underpaid people and that's a disaster waiting to happen. The AS400->VB6->NewShiny path actually sounds like a success case given the messes that are out there.
- q-base 5y ago
  That quote is seriously brilliant! Thanks for sharing.
- DonHopkins 5y ago
  Just don't build one solution to your problem with regular expressions: then you have two problems.
- smichel17 5y ago
  TFA ends with that quote.
- bostonpete 5y ago
  Well, the article does conclude with that exact tweet...
- NumberCruncher 5y ago
  The "the right tool for the right job" applies for ML topics too. If the job involves "looking smart and innovative" for whatever reasons, people tend to err on the side of overly complex solutions. On the other hand if the advice "let's just go with an SQL query built on a stack of egregiously oversimplifying assumptions" comes from someone, who doesn't know how SQL and linear regression / logistic regression with binning/bucketing / simple decision trees work, I would ask for a second opinion. Because a huge part of the retail banking, non-life insurance and marketing business is running on this simple stack. Obviously profitable. If the same advice comes from someone, who knows when to use deep learning instead of XGBoost and why, I would go with his/her advice. And I would try to keep him happy and on my team.
- marcosdumay 5y ago
  My workplace has got all kinds of attention for building a blockchain based data collection system that encompasses an entire sector of the economy. It's "almost done", so we are right now starting a simple set of REST services that write into a badly normalized transactional database just in case it stays "almost done" for too long.
- arnaudsm 5y ago
  This isn't ironic, I've actually done that multiple times in a large company. No one noticed, everyone went home happy.
bongoman37 5y ago
A second point on that is, start with the simplest and most trivial models first, then add complexity as needed.
joeldo 5y ago
I wonder if this also applies to computer vision? There are certainly problem spaces where heuristics are well established, but many approaches around object detection/segmentation seem much easier/robust to implement with machine learning.
- godelski 5y ago
  There are some stuff that is more robust but the clarification is hard with classical methods or even small models. Though we're getting better at small models. There's different biases in the models too. But I wouldn't expect classical methods to do well on ImageNet. Though ImageNet has a lot of issues...
- crubier 5y ago
  Was going to answer this. I very much agree with the article, but deep learning is absolutely a game changer for computer vision. I myself tried several time to “not use ML” for some easy computer vision tasks where traditional CV methods are supposed to work. Well I always end up in situations where they don’t work well without fine parameter tuning, and tuning the parameter for a situation breaks the model in other situations, so you start adding layers of complexity to automatically tune the parameters, but the parameter tuning system also has its own parameters... While a simple neural net is trained easily and is much more robust, saving a lot of time and complexity. Another proof of that is that CV products only started meaningfully entering the market after ML became applicable to CV (after 2015 for complex tasks, or earlier for simpler stuff like MNIST).
  - bushbaba 5y ago
    To be fair. Doing that fine parameter tuning and complex layering of heuristics is to some extent creating a “ml model” from hand.
    - crubier 5y ago
      Exactly. This is why usually when you reach that point, a red light turns on in your brain saying “you are just reinventing ML at this point, stop”
arketyp 5y ago
I thought this was going to be about data preprocessing or domain transformation. The article does touch upon it. For instance, you can boost your image classifier by normalizing your images with simple statistics. Ironically, since neural networks are very good at finding basic (but non-trivial) feature correlations, the reverse is also true: for instance, you can boost your SVG classifier by adding to it the feature responses of a CNN pre-trained on Imagenet.
punnerud 5y ago
Most of the article is about the first of Google’s 43 rules about ML: “Don’t be afraid to launch a product without machine learning.” and this is the first part of the description: “ Machine learning is cool, but it requires data. Theoretically, you can take data from a different problem and then tweak the model for a new product, but this will likely underperform basic heuristics. If you think that machine learning will give you a 100% boost, then a heuristic will get you 50% of the way there. (..)” https://developers.google.com/machine-learning/guides/rules-of-ml https://developers.google.com/machine-learning/guides/rules-...
- arketyp 5y ago
  Yes, this is the third paragraph of the article.
  - fho 5y ago
    Meta: I feel like a lot of people (including me) just come to HN for the comments, which are often (subjectively) better than the article itself. Basically the heading becomes the random discussion topic that gets thrown in the room. Maybe there is an experimental social platform in that: (Re-)create a HN or reddit look-alike, but instead of user submitted links just pick random headings from news sites. Every ten minutes, post a new one without any context or link to be discussed and voted by the audience. No idea where this would take us.
    - punnerud 5y ago
      Isn’t that what «Ask HN:» is for? You can also just post a title without any link. So you are basically asking for a subset of HN? To avoid echo chamber I think the links is a good thing.
- otabdeveloper4 5y ago
  That makes it sound like the problem is lack of data, which isn't true. The problem is that the kind of ML that involves downloading a framework from github and tweaking features until the percent goes up is actually built on certain statistical models under the hood that people don't understand and that don't fit the process they're trying to model. When the statistical model is correct you don't need loads of data. E.g., you don't need more than a thousand respondents to make valid inferences about millions of people in a sociological survey.
cgufus 5y ago
I fully agree with the article. One thing not mentioned, however probably assumed to be given: domain knowledge. A domain expert using simple methods will probably beat any decent ML model because they are able to define strong features.
- jillesvangurp 5y ago
  That can happen indeed. Compensating for lack of system or domain understanding with ML can result in mediocre results. I've seen this repeatedly with ML teams struggling to get their models adjusted to what was fundamentally not so great data that needed a simple cleanup. Failing to understand the data was dirty, which was easy to address, led to a wild goose chase extracting this and that feature in attempts to make the magic work better. Once you have deep understanding of your domain and system, finding the places where ML truly adds value is a lot easier. Also, you'll have a basic understanding of how things are without it and you'll know whether it is working better or not and whether that's worth the trouble.
alkonaut 5y ago
But the point of ML to begin with is likely often to appeal not by a better product but by appealing to investors or managers. If you create a better product but it doesn’t have “AI” in it then it failed in that aspect. What’s needed is a set of things that can be sold as AI or ML but isn’t.
- xvector 5y ago
  Spend two weeks adding some hidden worthless token "feature" no one will ever need or use that relies on AI. Then you can say your product is powered by AI. Boom, done.
  - blitzar 5y ago
    My logger uses AI to generate a catchy 'message-of-the-day' to the console on first run. My project is powered by AI.
- deleted 5y ago
  [deleted]
- aitchnyu 5y ago
  MG, Chinese carmaker took full frontpage ad on Indian papers to saw their new car has AI, mostly meaning voice recognition commands and ADAS.
lysecret 5y ago
To me that is just an iteration on first you makes it run then you make it right. And to make it run you start by the simplest approach. And building your own model is generally not the simplest however, it can be. There are some areas where you should start with ml. Most importantly Vison and some NLP, whenever a pretrained model for your task exists.
- webspaceadam 5y ago
  This is correct. But i guess the article argues about not already solved problems. NLP is in the most cases so powerful and easy to implement, that i would argue it can be viewed as a more complex version of a heuristic. My thought comes from the idea, that you need to act up on the data NLP-Algorithms bring to you.
  - lysecret 5y ago
    Yea I get that but I have experience people working with all sorts of insanely complicated heuristic to get something like a NER system running when they could have much more easily used a Hugging face model. But I totally agree that the article holds true if you have to train your own model.
Humphrey 5y ago
Yes - and after many years, I'm yet to get past this first rule, and actually use ML. One day I hope to have a use case that's worth testing it out on!
quanto 5y ago
I recall attending a technical talk given by a team of senior ML scientists from a prestigious SV firm (that I shall not name here). The talk was given to an audience of scientists at a leading university. The problem was estimating an incoming train speed from an embedded microphone sensor near the train station. The ML scientists used the latest techniques in deep learning to process the acoustic time series. The talk session was two hours long. This project was their showcase. I guess no one in the prestigious ML team knew about the Doppler shift and its closed form expression. Typically taught in a highschool physics class. A simple formula that you can calculate by hand: no need for a GPU cluster.
- tomp 5y ago
  A sufficiently large RLCDNN would reinvent the Doppler effect from data, eight?
  - atoav 5y ago
    You could also let Tom the traindriver sit there and have him guesstimate the speed.
    - fho 5y ago
      Or just have two switches on the train tracks
      - pbhjpbhj 5y ago
        Radar exists too. The need might be for a sensor local to the platform as a back up to give warning for a train that's traveling too fast? In which case a sensor that mimics the old Cowboy film favourite of putting one's ear to the track seems like a reasonable thing to try.
      - Ekaros 5y ago
        Or some other type of sensor and minimal gear added to each locomotive...
      - 5y ago
Ikerso115 5y ago
Ns que es esto yo soyb español
___luigi 5y ago
ML can help reduce technical debt at logic layer, but it increases the technical debt at the infrastructure layer. It's a challenge for any company to deploy, manage and monitor models in production. If you can get away with a simple rule, that's a bigger win for the product (I'm not talking about research here). In the community, there is a trend that "complicated == better". imho, more is less in industrial ML. You need to deal with model management, worry about inference & latency when the model gets bigger. The author has another article where he argues that data scientists need to be full stack ninja. While I don't fully agree with that statement, I think it benefits the company in many many ways. Data scientists need to meet engineers in the middle, and all these challenges need to be considered from day 1. Another trend I see is that some data scientists are not driven by the question "Can we solve this problem for the company?", but rather "Can we solve this problem using ML/DL?". This will lead data scientists to use the shiny and trendy models, even if it is not suitable for the job. I would blame management here, in some environments, data scientists are evaluated based on "fancy" models they build, not solutions that they provide. Solutions can be simple (but not simpler) rules.
vletal 5y ago
In the business and corporate world this is so underrated. In the past I attended several meetings with customers where I was actively discouraged asking questions which would help us deliver a good meaningful solution as long as the customer would be happy "investing in a ML solution". And they were...
s_gourichon 5y ago
I can see both sides of the argument. On one side, using ML feels like huge overkill when a simple trick exists. Plus AI can freak out in some circumstances. On the other side, it may find other, less obvious cues giving something more robust. Rich Sutton's "bitter lesson" says the weight will move in time in favor of ML. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- rpmisms 5y ago
  From my limited experience, ML is good at massively multi-factor problems. If a human can understand the input, normal code will usually suffice. This is why ML is pretty much the only option for autonomous driving, but not for calculating credit scores.
  - GeneralMayhem 5y ago
    Credit scores aren't a great point of comparison because they have specific explainability requirements. If your goal is to predict defaults - for instance, if you work for a bank or a hedge fund dealing in bonds - then more sophisticated ML techniques might be appropriate. But credit scores are optimized for consistency, not accuracy. I know that was probably an offhand example, but it's illustrative of the kinds of non-functional requirements that can make ML solutions more or less viable as soon as the technology has contact with human society.
    - thinkharderdev 5y ago
      Was going to say the same. In any decision where the outcome affects a human being, "because the algorithm said so" is usually not a satisfactory answer either to the human being affected or to any regulators who have an interest.
MattGaiser 5y ago
Until a few weeks ago, I worked for a team trying to build AI driven products. A surprisingly challenging thing has been finding problems that aren't better solved without ML (as an ML company, we are supposed to be using it so those concepts get eliminated).
laichzeit0 5y ago
To offhand dismiss ML is also a cardinal sin. Control/treatment groups can show unambiguously when ML outperforms expert hand-crafted rules, pure random decisions or a simple model. The point is to measure, and not go 100% all in with one approach, but try many things and measure. I've done some process optimization with black-box methods, simple models, and SQL using domain expertise. In business you typically have budget and time constraints, so you go for the simplest and quickest solution first, show unambiguously that it works better, and then ask for more time and budget to build something more fancy. I ask myself "if this was my business, and my money, would I spend it doing this", if the answer is no, then you probably shouldn't.
- mjburgess 5y ago
  The thing is, it's almost always clear when ML will outperform and when it wont. It isn't magic. ML systems are just compressed aggregations of their input datasets. The question is then, (1) do we have datasets that are highly representative of the solutions to our problems? and (2) are our current systems sensitive to the relevant variations in these datasets? If (1) is NO, then ML is impossible. If (2) is YES, then it's unlikely to provide a big ROI.
  - antupis 5y ago
    I would add (3) can we leverage existing models.
  - HelloNurse 5y ago
    When ML replaces human decisions or very strict old software, there's also a more fundamental problem: are we enabling new mistakes that weren't possible before? How catastrophic? For example, processing images according to some trained model instead of fixed rules and formulas introduces the risk of mismatched models (e.g. landscape photographs treated as line art from anime). Cases like self-driving cars not seeing obstacles are more obvious and more tragic.
  - kumarvvr 5y ago
    >ML systems are just compressed aggregations of their input datasets I like to think of them as forgiving sieves of patterns in data. Overfitting a sieve will exclude a large number of almost positive cases, loose fitting will include a large number of mostly negative cases. And there is always a danger of falling into a local minima and not being able to come out of it.
DrNuke 5y ago
Don’t bash the tools… just bash the fools!
nikanj 5y ago
Starting with Machine Learning gets you funded, though.
artembugara 5y ago
So true, especially about RegEx. I love RegEx. You can do so many things with simple RegEx rules. For example, have you ever tried to autodetect a published datetime of a news article published online? In many cases, it will be in metadata, or in the time/datetime tag. However, there still many websites where published time is just written somewhere with no logic at all. Writing a RegEx script by hand can resolve a problem. But every time I speak about it with our clients/prospects, they ask about ML that we use to parse news content. Product: https://newscatcherapi.com/news-api https://newscatcherapi.com/news-api
unhammer 5y ago
Googe's Rule #2: > First, design and implement metrics. > Before formalizing what your machine learning system will do, track as much as possible in your current system. Do this for the following reasons: > * It is easier to gain permission from the system’s users earlier on. > * If you think that something might be a concern in the future, it is better to get historical data now. :-/
masswerk 5y ago
> Solve the problem manually, or with heuristics. This way, it will force you to become intimately familiar with the problem and the data, which is the most important first step. Back then, when I did social research at university, I found it helpful to just look at the raw data. This is immensely helpful for familiarizing yourself with the data and discerning patterns that high-level analysis wont reveal easily. (In this case, you may want to start with a subset for evident reasons.)
- jack_riminton 5y ago
  Relevant tweet: https://twitter.com/jsheltzer/status/1327256638420635648 https://twitter.com/jsheltzer/status/1327256638420635648
Iv 5y ago
I went into ML when I realized that this piece of advice is now wrong, at least in computer vision. It was a few years ago. I had to classify pictures of closed and opened hands. I thought surely I don't need ML for simple stuff like that: a hue filter, a blob detector, a perimeter/area ratio should give me a first prototype faster and given the little amount of data I had (about a hundred images of each), not worth the headache. I quickly had a simple detector with 80% success rate. Then as I was learning a new ML framework, I tried it too, thinking that would surely be overengineering for a poor result. I took the VGG16 cat-or-dog sample, replaced the training set with my poorly scaled, non-normalized one, ran training for a few hours and, yes, outperformed the simple detector that took me much longer to write. Now in computer vision, I think it makes sense to try ML first, and if you are doing common tasks like classification or localization of objects, setting up a prototype with pre-trained models has become ridiculously easy. Try that first, and then try to outperform that simple baseline. In most case, it will be hard and instead worth improving the ML way.
- ___luigi 5y ago
  I think the author was focusing more on general applications (given his research & industrial background). In computer vision & NLP, the field is a bit advanced and it's harder to come up with rules. The promise of Auto-ML is bigger in these two fields.
deleted 5y ago
[deleted]
lmilcin 5y ago
That's not how it works. People build ML solutions not because they went through rigorous analysis and figured out their problem needs ML solution. They just want to do ML and are looking for a problem that can be solved with it. Then they will likely ignore you when you say this problem has also neat traditional solution. This is further exacerbated by corporate actions like competitions for best AI (or Blockchain, etc.) project. Which you typically can't participate in if you have traditional solution even if it is way better.
- dolmen 5y ago
  The fallacy of ML/AI companies. Example: https://beta.openai.com/examples/default-translate https://beta.openai.com/examples/default-translate They even use flawed results in their marketing materials that they didn't validated with domain experts. ("Où est les toilettes ?" is not french).
OJFord 5y ago
I think a good rough guide is that if you consider it ML, if you're going to 'do ML', then.. it might be appropriate, but you're jumping to the solution and trying to make it fit (pun intended) the problem. If on the other hand you start from having some statistics to do on the data you have, then you might at some point find yourself doing the sexy subset of it that we call 'ML', and fine.
wanderingmind 5y ago
I come from a core engineering background. In my experience, ML especially DNN these days is a way for people to avoid doing critical thinking. The improvement even if it works is extremely marginal making the ROI useless. Further unlike social media, a failure of ML model will result in a loss of limb or life. Unfortunately most decision making C-suites are not engineers who fall for the marketing hype and burn through time and capital without tangible outcomes.
incrudible 5y ago
I disagree. If you have the data, try throwing ML at it. It's probably less work than trying to "understand it" and building a heuristic. If you don't have the data, how are you going to validate your heuristic anyway?
_wldu 5y ago
If you have not seen James Mickens (Harvard CS) USENIX Security keynote presentation from 2018, I highly recommend it. It's hilarious while clearly showing how reckless and dangerous ML is: https://www.youtube.com/watch?v=ajGX7odA87k https://www.youtube.com/watch?v=ajGX7odA87k
lincpa 5y ago
Explainable AI System use the law model and the Warehouse/Workshop Model (2021-04-30) https://github.com/linpengcheng/PurefunctionPipelineDataflow#Explainable-AI-System https://github.com/linpengcheng/PurefunctionPipelineDataflow...
lvl100 5y ago
ML really needs specification tests.
dataqa 5y ago
I have seen first hand at small and large companies how problems have been tackled with ML without trying a simple rule or heuristic first. And then, further down the line, the system has been compared to a few business rules put together, to find that the difference in performance did not explain the deployment of an ML system in the first place. It's true that if your rules grow in complexity, this might make it harder to maintain, but the good thing about rules is that they tend to be fully explainable, and they can be encoded by domain experts. So the maintenance of such a system does not need to be done exclusively by an ML engineer anymore. Here is where I insert my plug: I have developed a tool to create rules to solve NLP problems: https://github.com/dataqa/dataqa https://github.com/dataqa/dataqa
streamofdigits 5y ago
What people call "ML" is actually several bundled phenomena. Unbundling them is profitable exercise that can help prevent alot of heartburn * 1 -> the discovery of specific families of non-linear classification algorithms (with image and language patterns being examples succesful new domains). the domain where these approaches are productive might be significantly smaller than what all the hyperventilation and obfuscation suggests. * 2 -> the ability to deploy algorithms "at scale". this cannot be overemphasized. Statistics used to be dark art practiced by scienty types in white lab coats locked in ivory towers. With open source libraries, linux, etc to a large degree ML means "statistics as understood and practiced by recently graduated computer scientists" * 3 -> business models and regulatory environments that enabled the collection of massive amounts of personal data and the application of algorithms in "live" human contexts without much regard for consent, implications, risks etc. Compare that wild west with the hoops that medical, insurance or banking algorithms are supposed to pass Conclusion, ML is here to stay in some shape or form, but ML hype has an expiration date
deleted 5y ago
[deleted]
dataviz1000 5y ago
After months learning about machine learning for time series forecasting, several chapters in a book on deep learning techniques for time series analysis and forecasting, the author kindly pointed out that there are no papers published up to that point that prove deep learning (neural networks) can perform better than classical statistics. From the scikit-learn faqs: > Will you add GPU support? > No, or at least not in the near future. The main reason is that GPU support will introduce many software dependencies and introduce platform specific issues. scikit-learn is designed to be easy to install on a wide variety of platforms. Outside of neural networks, GPUs don’t play a large role in machine learning today, and much larger gains in speed can often be achieved by a careful choice of algorithms. Of course, there are libraries that can support GPU acceleration for numpy calculations using matrix transformations now. Nonetheless, they are not often necessary.
- gwbas1c 5y ago
  > the author kindly pointed out that there are no papers published up to that point that prove deep learning (neural networks) can perform better than classical statistics. Early in my career I moved to Silicon Valley to work for a large company. The project was a machine learning project. I was taking models defined in XML, grabbing data from a few different databases, and running it through a machine learning engine written in-house. After a year and a half, it came out that our machine-learning-based system couldn't beat the current system that used normal statistics. What rubbed me the wrong way was that the managers brought someone else in to run the data, manually, through the machine learning algorithm. More specifically, what bothered me was that we didn't attempt this kind of experiment early in the project. It felt like I was hired to work on a "solution in search of a problem." Career lesson: Ask a lot of questions early in a project's life. If you're working on something that uses machine learning, ask what system it's replacing, and make sure that someone (or you) runs it manually before spending the time to automate.
nabla9 5y ago
(for consulting in ML) The Second Rule of Machine Learning - Start Machine Learning with simple shallow models. 50% of the problems are solved with good data choice of data + some generalized linear model. 30% remaining problems solved with shallow models or old school ML models. Anything from support-vector machines, decision trees, nearest neighbors, very shallow neural networks. Remaining 20% require more work.
Dumblydorr 5y ago
You always start by looking at the data, not by busting out advanced statistical methods. Those methods are obscure and could easily hide how ugly and unclean your dataset is. You really do need to look at types, missingness, the data structure and ensuring the row ID is what you want it to be, eliminating duplicates, joining on other datasets; it's a massive list of steps. Even with a clean dataset, most clients will want basic arithmetic calculations: averages, counts, percentages, standard deviation, etc. Occasionally they'll want some basic logistic models, something slightly more causal. If they go straight to machine learning without these steps, do they actually understand their problem and what they want? Or are they reaching for the shiniest thing they've heard of?
oakfr 5y ago
There are domains where the use of ML is not only valid but the best viable option (e.g. recommendation systems, computer vision, etc.) A few thoughts on how to maximize your chances of winning in this case: https://medium.com/criteo-engineering/making-your-company-ml-centric-5266809fbd26 https://medium.com/criteo-engineering/making-your-company-ml...
- mrits 5y ago
  "You bought a BBQ grill, you must be interested in more BBQ grills". This is how Amazon ML engine seems to work for me
tikiman163 5y ago
Lately I've been thinking a lot about data cubes and how their use cases and methodologies for making them applicable are very similar to most machine learning algorithms. I don't mean how the output is generated or how things are programmed. What I mean is that they both tend to produce far more output than is practically useful. Additionally, it can be very easy to look at any small part of the output and draw incorrect conclusions. To clarify, when I talk about ML I'm primarily referring to classifier algorithms and approaches (including nlp). In the large part the ML is being used to generate classifier rules which generalize patterns, and data cubes are often used to look for aggregations and data sequences which generalize patterns. The problem is that random patterns happen all the time, and may even persist for a long time despite a lack of real correlation. Semantic analysis of data cube output is really important in order to find meaningful patterns. What I'm getting at is I often wonder why most ML projects try to treat it like it's magic. Human assisted learning has shown repeatedly to be the system which actually works in practical application. The classifier output needs to be pruned to remove rules that only held true in the sample data, or were merely coincidental, or simply have no practical value. Approaches like this are not cheap to set up and may in the end still only produce the same results as the existing entirely non-ML based system. What is the likely scale of work compared to the benefit is the first question I ask myself before working on anything. If I don't have objective data to answer that you have to do some research to find out. Never try to build a massive or complicated system you don't have objective reasons to expect will be worth the effort. That's precisely what people have been doing with ML constantly. It's little wonder most developers have such low opinions of ML projects.
charles_f 5y ago
Thanks for that! Some people I work with are constantly asking for ML, they invoke like its magic and will figure shit out by itself. Then when I push back asking how they would make the decisions themselves, their answers tend to be in the line of "it's ML, it should figure out by itself", and when I ask about the data to be used, "it sshould adapt itself and find the data". Getting to have a heuristic in the first place is so hard. Reminds me of the book "Everything is obvious", where they experimented a few times and showed that in complex systems, advanced prediction systems made on many available and seamingly relevant variables are only marginally better (2 to 4% in the experiments) than the simplest heuristics you can use. They interpreted that as a limit of predictability, because systems with sufficient complexity behave with a seemingly irreducible random part.
deleted 5y ago
[deleted]
dekhn 5y ago
I was very keen on machine learning for some time- I started working with ML in the mid 90s. The work I did definitely could have been replaced with a far less mathematically principled approach, but I wanted to learn ML because it was sexy and I assumed that at some point in the future we'd have a technological singularity due to ML research. I didn't really understand the technology (gradient descent) underlying the training, so I went to grad school and spent 7 years learning gradient descent and other optimization techniques. Didn't get any chances to work in ML after that because... well, ML had a terrible rep in all the structural biology fields and even the best models were at most 70% accurate. Not enough data, not enough training methods, not enough CPU time. Eventually I landed at Google in Ads and learned about their ML system, Smartass. I had to go back and learn a whole different approach to ML (Smartass is a weird system) and then wait years for Google to discover GPU-based machine learning (they have Vincent Vanhouke to thank- he sat near Jeff Dean and stuffed 8 GPUs into a workstation to prove that he could do training faster than thousands of CPUs in prod) and deep neural networks. Fast forward a few years, and I'm an expert in ML, and the only suggestion I have is that everybody should read and internalize: https://research.google/pubs/pub43146/ https://research.google/pubs/pub43146/ So little of success in ML comes from the sexy algorithms and so much just comes from ensuring a bunch of boring details get properly saved in the right place.
moedersmooiste 5y ago
I always have great success doing anomaly detection with basic standard deviation in some SQL queries...
rdevsrex 5y ago
So, I am a total ML noob. The thing I haven't found a straight answer to is, what is a model. I mean when it is in production? Is it just some random blob that you pipe data into and get data out?
- Mentlo 5y ago
  Depends on how it's put into production, but you can deploy a model as a RESTful API that has a defined interface and a defined output. What it does underneath is less important to you I guess. So for all intents and purposes, yes, a model in production is something you feed a predefined set of data points and it gives you a predefined format of output.
- jazzyjackson 5y ago
  So, you know how a straight line is defined as mx + b, where you just have two parameters: slope and intercept ? Your input value is X, you multiply it by your slope and add your intercept to get the output (the Y value on the line). The 'training' of an ML algo is really just finding the line-of-best-fit so that you can make predictions. So your line-of-best-fit is encoded in these two parameters, allowing you to make predictions about what the output would be for arbitrary input. The problems people are throwing at ML have many more parameters and dimensions, but the training is a matter of finding those parameters that come closest to predicting the outcome. The 'model' is this set of parameters that allows the function to make predictions. (disclaimer: also an ML noob, correct me if I'm wrong)
  - rdevsrex 5y ago
    thanks!
lobo_tuerto 5y ago
Seems like antirez (from Redis fame) doesn't agree with this: https://twitter.com/antirez/status/1440711992038158336 https://twitter.com/antirez/status/1440711992038158336
xyzzy21 5y ago
This is generally correct about ALL technologies. You should NEVER start with a solution and look for a problem except in a very general sense (e.g. looking for potential markets in the abstract). Taking a solution market without having the problem well defined and identified is absolutely Epic Fail. Once you think you have a market, you should see how it can be done FIRST without your "fancy miracle technology" because NOBODY buys a technology because it's sexy or trendy: they buy because it added value in terms of more capability or lower costs. And ALL problems have current solutions that almost certainly DO NOT use anything as complex as your technology solution so you have to trend very carefully and deliberately in a rational sense: what value are we REALLY adding? That starts with knowing your competition and the current solution to solving the problem first and then finding every reason why your technology won't work or will be problematic. You ONLY have market potential once you've exhausted those faults or have objective arguments for your value proposition that have been validated by actual customers. The actual prove is made when they are willing to write a PO to you for the solution. Until then, everything you are doing is unproven.
- samuel2 5y ago
  well said, thx for your comment
amts 5y ago
ML users may not be aware enough that the ecological validity of studies with peri- and intracellular recordings of individual neurons, which form the empirical basis for activation function(s), is very low in relation to general mind functions, which are supposed to be measured by a bit more, but still not 100% ecologically valid methods like fMRI etc and are thought to be simulated by ML.

764 points by 7d7n 5y ago

xnx 5y ago

I haven't read the article, but I really like the construction of the phrase. I would also propose: First rule of optimization: Don't optimize first. First rule of automation: Don't automate first.

refactor_master 5y ago
“This is just an agile PoC. It’s not meant to be readable, performant or documented”. The first rule of everything should be “it depends”.
- AussieWog93 5y ago
  I propose instead the first rule of rules: Don't assume that general advice will be applicable to all circumstances. :P

elexhobby 5y ago

Furthermore, follow https://twitter.com/_brohrer_/status/1425770502321283073 https://twitter.com/_brohrer_/status/1425770502321283073 "When you have a problem, build two solutions - a deep Bayesian transformer running on multicloud Kubernetes and a SQL query built on a stack of egregiously oversimplifying assumptions. Put one on your resume, the other in production. Everyone goes home happy."

arketyp 5y ago
Furthermore in the article, yes.
hughrr 5y ago
This reminds me of an experience I had watching a company trying to replace a system with ML. First they marketed it heavily before even thinking. During test cycle they fed the entire data corpus in and ran some of the original test cases and found some business destroying results pop out. The entire system ended up a verbatim port of the VB6 crap which was a verbatim port of the original AS400 crap that actually worked. The marketing to this day says it’s ML based and everyone buys into the hype. It’s not. It was a complete failure. But the original system has 30 years of human experience codified in it.
- ethbr0 5y ago
  If I had a nickel for every time I've seen "business rules engine" turned into "AI" in the last few years... But I guess if we complain that half of our colleagues and the media don't understand ML, why should we expect management to? When the command from C-level is "We need some AI projects to tell our shareholders about," we shouldn't be surprised when middle management suddenly has successful AI projects in their slide decks.
  - mjburgess 5y ago
    Well really-existing AI is just either "taking a mean()" or "programming a rule". So all of programming actually counts as (symbolic) AI.
  - cm2187 5y ago
    In the same way the terms “blockchain” being used for “digital signing” or “cloud” for a server...
  - i_am_proteus 5y ago
    If you have an existing rules-based decision-tree system, and you compare its performance with a bunch of other decision trees, and it does better, you are implementing a random forest that happens to be identical to your original system. Artificial Intelligence.
    - mumblemumble 5y ago
      If you talk enough all the models and hyperparameters you compared and suchlike that you experimented with, you can probably sufficiently impress people with the talk about the enormous deep learning model you spent several months developing that they won't even remember you mentioning the two-clause Boolean expression that you actually put into production. And of course it's AI. You used k-fold cross validation to select it.
- pedrocr 5y ago
  The AI taxonomy includes the term "Expert Systems" for these kinds of things. On the one hand it's definitely not of the new wave of ML AI so hyping those things as innovative is off. On the other hand we should definitely give more attention to that kind of setup and understand how to build/maintain/test it properly. Otherwise often it ends up being ran by a few hundred Excel sheets and a few severely underpaid people and that's a disaster waiting to happen. The AS400->VB6->NewShiny path actually sounds like a success case given the messes that are out there.
q-base 5y ago
That quote is seriously brilliant! Thanks for sharing.
DonHopkins 5y ago
Just don't build one solution to your problem with regular expressions: then you have two problems.
smichel17 5y ago
TFA ends with that quote.
bostonpete 5y ago
Well, the article does conclude with that exact tweet...
NumberCruncher 5y ago
The "the right tool for the right job" applies for ML topics too. If the job involves "looking smart and innovative" for whatever reasons, people tend to err on the side of overly complex solutions. On the other hand if the advice "let's just go with an SQL query built on a stack of egregiously oversimplifying assumptions" comes from someone, who doesn't know how SQL and linear regression / logistic regression with binning/bucketing / simple decision trees work, I would ask for a second opinion. Because a huge part of the retail banking, non-life insurance and marketing business is running on this simple stack. Obviously profitable. If the same advice comes from someone, who knows when to use deep learning instead of XGBoost and why, I would go with his/her advice. And I would try to keep him happy and on my team.
marcosdumay 5y ago
My workplace has got all kinds of attention for building a blockchain based data collection system that encompasses an entire sector of the economy. It's "almost done", so we are right now starting a simple set of REST services that write into a badly normalized transactional database just in case it stays "almost done" for too long.
arnaudsm 5y ago
This isn't ironic, I've actually done that multiple times in a large company. No one noticed, everyone went home happy.

bongoman37 5y ago

A second point on that is, start with the simplest and most trivial models first, then add complexity as needed.

joeldo 5y ago

I wonder if this also applies to computer vision? There are certainly problem spaces where heuristics are well established, but many approaches around object detection/segmentation seem much easier/robust to implement with machine learning.

godelski 5y ago
There are some stuff that is more robust but the clarification is hard with classical methods or even small models. Though we're getting better at small models. There's different biases in the models too. But I wouldn't expect classical methods to do well on ImageNet. Though ImageNet has a lot of issues...
crubier 5y ago
Was going to answer this. I very much agree with the article, but deep learning is absolutely a game changer for computer vision. I myself tried several time to “not use ML” for some easy computer vision tasks where traditional CV methods are supposed to work. Well I always end up in situations where they don’t work well without fine parameter tuning, and tuning the parameter for a situation breaks the model in other situations, so you start adding layers of complexity to automatically tune the parameters, but the parameter tuning system also has its own parameters... While a simple neural net is trained easily and is much more robust, saving a lot of time and complexity. Another proof of that is that CV products only started meaningfully entering the market after ML became applicable to CV (after 2015 for complex tasks, or earlier for simpler stuff like MNIST).
- bushbaba 5y ago
  To be fair. Doing that fine parameter tuning and complex layering of heuristics is to some extent creating a “ml model” from hand.
  - crubier 5y ago
    Exactly. This is why usually when you reach that point, a red light turns on in your brain saying “you are just reinventing ML at this point, stop”

arketyp 5y ago

I thought this was going to be about data preprocessing or domain transformation. The article does touch upon it. For instance, you can boost your image classifier by normalizing your images with simple statistics. Ironically, since neural networks are very good at finding basic (but non-trivial) feature correlations, the reverse is also true: for instance, you can boost your SVG classifier by adding to it the feature responses of a CNN pre-trained on Imagenet.

punnerud 5y ago

Most of the article is about the first of Google’s 43 rules about ML: “Don’t be afraid to launch a product without machine learning.” and this is the first part of the description: “ Machine learning is cool, but it requires data. Theoretically, you can take data from a different problem and then tweak the model for a new product, but this will likely underperform basic heuristics. If you think that machine learning will give you a 100% boost, then a heuristic will get you 50% of the way there. (..)” https://developers.google.com/machine-learning/guides/rules-of-ml https://developers.google.com/machine-learning/guides/rules-...

arketyp 5y ago
Yes, this is the third paragraph of the article.
- fho 5y ago
  Meta: I feel like a lot of people (including me) just come to HN for the comments, which are often (subjectively) better than the article itself. Basically the heading becomes the random discussion topic that gets thrown in the room. Maybe there is an experimental social platform in that: (Re-)create a HN or reddit look-alike, but instead of user submitted links just pick random headings from news sites. Every ten minutes, post a new one without any context or link to be discussed and voted by the audience. No idea where this would take us.
  - punnerud 5y ago
    Isn’t that what «Ask HN:» is for? You can also just post a title without any link. So you are basically asking for a subset of HN? To avoid echo chamber I think the links is a good thing.
otabdeveloper4 5y ago
That makes it sound like the problem is lack of data, which isn't true. The problem is that the kind of ML that involves downloading a framework from github and tweaking features until the percent goes up is actually built on certain statistical models under the hood that people don't understand and that don't fit the process they're trying to model. When the statistical model is correct you don't need loads of data. E.g., you don't need more than a thousand respondents to make valid inferences about millions of people in a sociological survey.

cgufus 5y ago

I fully agree with the article. One thing not mentioned, however probably assumed to be given: domain knowledge. A domain expert using simple methods will probably beat any decent ML model because they are able to define strong features.

jillesvangurp 5y ago
That can happen indeed. Compensating for lack of system or domain understanding with ML can result in mediocre results. I've seen this repeatedly with ML teams struggling to get their models adjusted to what was fundamentally not so great data that needed a simple cleanup. Failing to understand the data was dirty, which was easy to address, led to a wild goose chase extracting this and that feature in attempts to make the magic work better. Once you have deep understanding of your domain and system, finding the places where ML truly adds value is a lot easier. Also, you'll have a basic understanding of how things are without it and you'll know whether it is working better or not and whether that's worth the trouble.

alkonaut 5y ago

But the point of ML to begin with is likely often to appeal not by a better product but by appealing to investors or managers. If you create a better product but it doesn’t have “AI” in it then it failed in that aspect. What’s needed is a set of things that can be sold as AI or ML but isn’t.

xvector 5y ago
Spend two weeks adding some hidden worthless token "feature" no one will ever need or use that relies on AI. Then you can say your product is powered by AI. Boom, done.
- blitzar 5y ago
  My logger uses AI to generate a catchy 'message-of-the-day' to the console on first run. My project is powered by AI.
deleted 5y ago
[deleted]
aitchnyu 5y ago
MG, Chinese carmaker took full frontpage ad on Indian papers to saw their new car has AI, mostly meaning voice recognition commands and ADAS.

lysecret 5y ago

To me that is just an iteration on first you makes it run then you make it right. And to make it run you start by the simplest approach. And building your own model is generally not the simplest however, it can be. There are some areas where you should start with ml. Most importantly Vison and some NLP, whenever a pretrained model for your task exists.

webspaceadam 5y ago
This is correct. But i guess the article argues about not already solved problems. NLP is in the most cases so powerful and easy to implement, that i would argue it can be viewed as a more complex version of a heuristic. My thought comes from the idea, that you need to act up on the data NLP-Algorithms bring to you.
- lysecret 5y ago
  Yea I get that but I have experience people working with all sorts of insanely complicated heuristic to get something like a NER system running when they could have much more easily used a Hugging face model. But I totally agree that the article holds true if you have to train your own model.

Humphrey 5y ago

Yes - and after many years, I'm yet to get past this first rule, and actually use ML. One day I hope to have a use case that's worth testing it out on!

quanto 5y ago

I recall attending a technical talk given by a team of senior ML scientists from a prestigious SV firm (that I shall not name here). The talk was given to an audience of scientists at a leading university. The problem was estimating an incoming train speed from an embedded microphone sensor near the train station. The ML scientists used the latest techniques in deep learning to process the acoustic time series. The talk session was two hours long. This project was their showcase. I guess no one in the prestigious ML team knew about the Doppler shift and its closed form expression. Typically taught in a highschool physics class. A simple formula that you can calculate by hand: no need for a GPU cluster.

tomp 5y ago
A sufficiently large RLCDNN would reinvent the Doppler effect from data, eight?
- atoav 5y ago
  You could also let Tom the traindriver sit there and have him guesstimate the speed.
  - fho 5y ago
    Or just have two switches on the train tracks
    - pbhjpbhj 5y ago
      Radar exists too. The need might be for a sensor local to the platform as a back up to give warning for a train that's traveling too fast? In which case a sensor that mimics the old Cowboy film favourite of putting one's ear to the track seems like a reasonable thing to try.
    - Ekaros 5y ago
      Or some other type of sensor and minimal gear added to each locomotive...
    - 5y ago

Ikerso115 5y ago

Ns que es esto yo soyb español

___luigi 5y ago

ML can help reduce technical debt at logic layer, but it increases the technical debt at the infrastructure layer. It's a challenge for any company to deploy, manage and monitor models in production. If you can get away with a simple rule, that's a bigger win for the product (I'm not talking about research here). In the community, there is a trend that "complicated == better". imho, more is less in industrial ML. You need to deal with model management, worry about inference & latency when the model gets bigger. The author has another article where he argues that data scientists need to be full stack ninja. While I don't fully agree with that statement, I think it benefits the company in many many ways. Data scientists need to meet engineers in the middle, and all these challenges need to be considered from day 1. Another trend I see is that some data scientists are not driven by the question "Can we solve this problem for the company?", but rather "Can we solve this problem using ML/DL?". This will lead data scientists to use the shiny and trendy models, even if it is not suitable for the job. I would blame management here, in some environments, data scientists are evaluated based on "fancy" models they build, not solutions that they provide. Solutions can be simple (but not simpler) rules.

vletal 5y ago

In the business and corporate world this is so underrated. In the past I attended several meetings with customers where I was actively discouraged asking questions which would help us deliver a good meaningful solution as long as the customer would be happy "investing in a ML solution". And they were...

s_gourichon 5y ago

I can see both sides of the argument. On one side, using ML feels like huge overkill when a simple trick exists. Plus AI can freak out in some circumstances. On the other side, it may find other, less obvious cues giving something more robust. Rich Sutton's "bitter lesson" says the weight will move in time in favor of ML. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

rpmisms 5y ago
From my limited experience, ML is good at massively multi-factor problems. If a human can understand the input, normal code will usually suffice. This is why ML is pretty much the only option for autonomous driving, but not for calculating credit scores.
- GeneralMayhem 5y ago
  Credit scores aren't a great point of comparison because they have specific explainability requirements. If your goal is to predict defaults - for instance, if you work for a bank or a hedge fund dealing in bonds - then more sophisticated ML techniques might be appropriate. But credit scores are optimized for consistency, not accuracy. I know that was probably an offhand example, but it's illustrative of the kinds of non-functional requirements that can make ML solutions more or less viable as soon as the technology has contact with human society.
  - thinkharderdev 5y ago
    Was going to say the same. In any decision where the outcome affects a human being, "because the algorithm said so" is usually not a satisfactory answer either to the human being affected or to any regulators who have an interest.

MattGaiser 5y ago

Until a few weeks ago, I worked for a team trying to build AI driven products. A surprisingly challenging thing has been finding problems that aren't better solved without ML (as an ML company, we are supposed to be using it so those concepts get eliminated).

laichzeit0 5y ago

To offhand dismiss ML is also a cardinal sin. Control/treatment groups can show unambiguously when ML outperforms expert hand-crafted rules, pure random decisions or a simple model. The point is to measure, and not go 100% all in with one approach, but try many things and measure. I've done some process optimization with black-box methods, simple models, and SQL using domain expertise. In business you typically have budget and time constraints, so you go for the simplest and quickest solution first, show unambiguously that it works better, and then ask for more time and budget to build something more fancy. I ask myself "if this was my business, and my money, would I spend it doing this", if the answer is no, then you probably shouldn't.

mjburgess 5y ago
The thing is, it's almost always clear when ML will outperform and when it wont. It isn't magic. ML systems are just compressed aggregations of their input datasets. The question is then, (1) do we have datasets that are highly representative of the solutions to our problems? and (2) are our current systems sensitive to the relevant variations in these datasets? If (1) is NO, then ML is impossible. If (2) is YES, then it's unlikely to provide a big ROI.
- antupis 5y ago
  I would add (3) can we leverage existing models.
- HelloNurse 5y ago
  When ML replaces human decisions or very strict old software, there's also a more fundamental problem: are we enabling new mistakes that weren't possible before? How catastrophic? For example, processing images according to some trained model instead of fixed rules and formulas introduces the risk of mismatched models (e.g. landscape photographs treated as line art from anime). Cases like self-driving cars not seeing obstacles are more obvious and more tragic.
- kumarvvr 5y ago
  >ML systems are just compressed aggregations of their input datasets I like to think of them as forgiving sieves of patterns in data. Overfitting a sieve will exclude a large number of almost positive cases, loose fitting will include a large number of mostly negative cases. And there is always a danger of falling into a local minima and not being able to come out of it.

DrNuke 5y ago

Don’t bash the tools… just bash the fools!

nikanj 5y ago

Starting with Machine Learning gets you funded, though.

artembugara 5y ago

So true, especially about RegEx. I love RegEx. You can do so many things with simple RegEx rules. For example, have you ever tried to autodetect a published datetime of a news article published online? In many cases, it will be in metadata, or in the time/datetime tag. However, there still many websites where published time is just written somewhere with no logic at all. Writing a RegEx script by hand can resolve a problem. But every time I speak about it with our clients/prospects, they ask about ML that we use to parse news content. Product: https://newscatcherapi.com/news-api https://newscatcherapi.com/news-api

unhammer 5y ago

Googe's Rule #2: > First, design and implement metrics. > Before formalizing what your machine learning system will do, track as much as possible in your current system. Do this for the following reasons: > * It is easier to gain permission from the system’s users earlier on. > * If you think that something might be a concern in the future, it is better to get historical data now. :-/

masswerk 5y ago

> Solve the problem manually, or with heuristics. This way, it will force you to become intimately familiar with the problem and the data, which is the most important first step. Back then, when I did social research at university, I found it helpful to just look at the raw data. This is immensely helpful for familiarizing yourself with the data and discerning patterns that high-level analysis wont reveal easily. (In this case, you may want to start with a subset for evident reasons.)

jack_riminton 5y ago
Relevant tweet: https://twitter.com/jsheltzer/status/1327256638420635648 https://twitter.com/jsheltzer/status/1327256638420635648

Iv 5y ago

I went into ML when I realized that this piece of advice is now wrong, at least in computer vision. It was a few years ago. I had to classify pictures of closed and opened hands. I thought surely I don't need ML for simple stuff like that: a hue filter, a blob detector, a perimeter/area ratio should give me a first prototype faster and given the little amount of data I had (about a hundred images of each), not worth the headache. I quickly had a simple detector with 80% success rate. Then as I was learning a new ML framework, I tried it too, thinking that would surely be overengineering for a poor result. I took the VGG16 cat-or-dog sample, replaced the training set with my poorly scaled, non-normalized one, ran training for a few hours and, yes, outperformed the simple detector that took me much longer to write. Now in computer vision, I think it makes sense to try ML first, and if you are doing common tasks like classification or localization of objects, setting up a prototype with pre-trained models has become ridiculously easy. Try that first, and then try to outperform that simple baseline. In most case, it will be hard and instead worth improving the ML way.

___luigi 5y ago
I think the author was focusing more on general applications (given his research & industrial background). In computer vision & NLP, the field is a bit advanced and it's harder to come up with rules. The promise of Auto-ML is bigger in these two fields.

deleted 5y ago

[deleted]

lmilcin 5y ago

That's not how it works. People build ML solutions not because they went through rigorous analysis and figured out their problem needs ML solution. They just want to do ML and are looking for a problem that can be solved with it. Then they will likely ignore you when you say this problem has also neat traditional solution. This is further exacerbated by corporate actions like competitions for best AI (or Blockchain, etc.) project. Which you typically can't participate in if you have traditional solution even if it is way better.

dolmen 5y ago
The fallacy of ML/AI companies. Example: https://beta.openai.com/examples/default-translate https://beta.openai.com/examples/default-translate They even use flawed results in their marketing materials that they didn't validated with domain experts. ("Où est les toilettes ?" is not french).

OJFord 5y ago

I think a good rough guide is that if you consider it ML, if you're going to 'do ML', then.. it might be appropriate, but you're jumping to the solution and trying to make it fit (pun intended) the problem. If on the other hand you start from having some statistics to do on the data you have, then you might at some point find yourself doing the sexy subset of it that we call 'ML', and fine.

wanderingmind 5y ago

I come from a core engineering background. In my experience, ML especially DNN these days is a way for people to avoid doing critical thinking. The improvement even if it works is extremely marginal making the ROI useless. Further unlike social media, a failure of ML model will result in a loss of limb or life. Unfortunately most decision making C-suites are not engineers who fall for the marketing hype and burn through time and capital without tangible outcomes.

incrudible 5y ago

I disagree. If you have the data, try throwing ML at it. It's probably less work than trying to "understand it" and building a heuristic. If you don't have the data, how are you going to validate your heuristic anyway?

_wldu 5y ago

If you have not seen James Mickens (Harvard CS) USENIX Security keynote presentation from 2018, I highly recommend it. It's hilarious while clearly showing how reckless and dangerous ML is: https://www.youtube.com/watch?v=ajGX7odA87k https://www.youtube.com/watch?v=ajGX7odA87k

lincpa 5y ago

Explainable AI System use the law model and the Warehouse/Workshop Model (2021-04-30) https://github.com/linpengcheng/PurefunctionPipelineDataflow#Explainable-AI-System https://github.com/linpengcheng/PurefunctionPipelineDataflow...

lvl100 5y ago

ML really needs specification tests.

dataqa 5y ago

I have seen first hand at small and large companies how problems have been tackled with ML without trying a simple rule or heuristic first. And then, further down the line, the system has been compared to a few business rules put together, to find that the difference in performance did not explain the deployment of an ML system in the first place. It's true that if your rules grow in complexity, this might make it harder to maintain, but the good thing about rules is that they tend to be fully explainable, and they can be encoded by domain experts. So the maintenance of such a system does not need to be done exclusively by an ML engineer anymore. Here is where I insert my plug: I have developed a tool to create rules to solve NLP problems: https://github.com/dataqa/dataqa https://github.com/dataqa/dataqa

streamofdigits 5y ago

What people call "ML" is actually several bundled phenomena. Unbundling them is profitable exercise that can help prevent alot of heartburn * 1 -> the discovery of specific families of non-linear classification algorithms (with image and language patterns being examples succesful new domains). the domain where these approaches are productive might be significantly smaller than what all the hyperventilation and obfuscation suggests. * 2 -> the ability to deploy algorithms "at scale". this cannot be overemphasized. Statistics used to be dark art practiced by scienty types in white lab coats locked in ivory towers. With open source libraries, linux, etc to a large degree ML means "statistics as understood and practiced by recently graduated computer scientists" * 3 -> business models and regulatory environments that enabled the collection of massive amounts of personal data and the application of algorithms in "live" human contexts without much regard for consent, implications, risks etc. Compare that wild west with the hoops that medical, insurance or banking algorithms are supposed to pass Conclusion, ML is here to stay in some shape or form, but ML hype has an expiration date

deleted 5y ago

[deleted]

dataviz1000 5y ago

After months learning about machine learning for time series forecasting, several chapters in a book on deep learning techniques for time series analysis and forecasting, the author kindly pointed out that there are no papers published up to that point that prove deep learning (neural networks) can perform better than classical statistics. From the scikit-learn faqs: > Will you add GPU support? > No, or at least not in the near future. The main reason is that GPU support will introduce many software dependencies and introduce platform specific issues. scikit-learn is designed to be easy to install on a wide variety of platforms. Outside of neural networks, GPUs don’t play a large role in machine learning today, and much larger gains in speed can often be achieved by a careful choice of algorithms. Of course, there are libraries that can support GPU acceleration for numpy calculations using matrix transformations now. Nonetheless, they are not often necessary.

gwbas1c 5y ago
> the author kindly pointed out that there are no papers published up to that point that prove deep learning (neural networks) can perform better than classical statistics. Early in my career I moved to Silicon Valley to work for a large company. The project was a machine learning project. I was taking models defined in XML, grabbing data from a few different databases, and running it through a machine learning engine written in-house. After a year and a half, it came out that our machine-learning-based system couldn't beat the current system that used normal statistics. What rubbed me the wrong way was that the managers brought someone else in to run the data, manually, through the machine learning algorithm. More specifically, what bothered me was that we didn't attempt this kind of experiment early in the project. It felt like I was hired to work on a "solution in search of a problem." Career lesson: Ask a lot of questions early in a project's life. If you're working on something that uses machine learning, ask what system it's replacing, and make sure that someone (or you) runs it manually before spending the time to automate.

nabla9 5y ago

(for consulting in ML) The Second Rule of Machine Learning - Start Machine Learning with simple shallow models. 50% of the problems are solved with good data choice of data + some generalized linear model. 30% remaining problems solved with shallow models or old school ML models. Anything from support-vector machines, decision trees, nearest neighbors, very shallow neural networks. Remaining 20% require more work.

Dumblydorr 5y ago

You always start by looking at the data, not by busting out advanced statistical methods. Those methods are obscure and could easily hide how ugly and unclean your dataset is. You really do need to look at types, missingness, the data structure and ensuring the row ID is what you want it to be, eliminating duplicates, joining on other datasets; it's a massive list of steps. Even with a clean dataset, most clients will want basic arithmetic calculations: averages, counts, percentages, standard deviation, etc. Occasionally they'll want some basic logistic models, something slightly more causal. If they go straight to machine learning without these steps, do they actually understand their problem and what they want? Or are they reaching for the shiniest thing they've heard of?

oakfr 5y ago

There are domains where the use of ML is not only valid but the best viable option (e.g. recommendation systems, computer vision, etc.) A few thoughts on how to maximize your chances of winning in this case: https://medium.com/criteo-engineering/making-your-company-ml-centric-5266809fbd26 https://medium.com/criteo-engineering/making-your-company-ml...

mrits 5y ago
"You bought a BBQ grill, you must be interested in more BBQ grills". This is how Amazon ML engine seems to work for me

tikiman163 5y ago

Lately I've been thinking a lot about data cubes and how their use cases and methodologies for making them applicable are very similar to most machine learning algorithms. I don't mean how the output is generated or how things are programmed. What I mean is that they both tend to produce far more output than is practically useful. Additionally, it can be very easy to look at any small part of the output and draw incorrect conclusions. To clarify, when I talk about ML I'm primarily referring to classifier algorithms and approaches (including nlp). In the large part the ML is being used to generate classifier rules which generalize patterns, and data cubes are often used to look for aggregations and data sequences which generalize patterns. The problem is that random patterns happen all the time, and may even persist for a long time despite a lack of real correlation. Semantic analysis of data cube output is really important in order to find meaningful patterns. What I'm getting at is I often wonder why most ML projects try to treat it like it's magic. Human assisted learning has shown repeatedly to be the system which actually works in practical application. The classifier output needs to be pruned to remove rules that only held true in the sample data, or were merely coincidental, or simply have no practical value. Approaches like this are not cheap to set up and may in the end still only produce the same results as the existing entirely non-ML based system. What is the likely scale of work compared to the benefit is the first question I ask myself before working on anything. If I don't have objective data to answer that you have to do some research to find out. Never try to build a massive or complicated system you don't have objective reasons to expect will be worth the effort. That's precisely what people have been doing with ML constantly. It's little wonder most developers have such low opinions of ML projects.

charles_f 5y ago

Thanks for that! Some people I work with are constantly asking for ML, they invoke like its magic and will figure shit out by itself. Then when I push back asking how they would make the decisions themselves, their answers tend to be in the line of "it's ML, it should figure out by itself", and when I ask about the data to be used, "it sshould adapt itself and find the data". Getting to have a heuristic in the first place is so hard. Reminds me of the book "Everything is obvious", where they experimented a few times and showed that in complex systems, advanced prediction systems made on many available and seamingly relevant variables are only marginally better (2 to 4% in the experiments) than the simplest heuristics you can use. They interpreted that as a limit of predictability, because systems with sufficient complexity behave with a seemingly irreducible random part.

deleted 5y ago

[deleted]

dekhn 5y ago

I was very keen on machine learning for some time- I started working with ML in the mid 90s. The work I did definitely could have been replaced with a far less mathematically principled approach, but I wanted to learn ML because it was sexy and I assumed that at some point in the future we'd have a technological singularity due to ML research. I didn't really understand the technology (gradient descent) underlying the training, so I went to grad school and spent 7 years learning gradient descent and other optimization techniques. Didn't get any chances to work in ML after that because... well, ML had a terrible rep in all the structural biology fields and even the best models were at most 70% accurate. Not enough data, not enough training methods, not enough CPU time. Eventually I landed at Google in Ads and learned about their ML system, Smartass. I had to go back and learn a whole different approach to ML (Smartass is a weird system) and then wait years for Google to discover GPU-based machine learning (they have Vincent Vanhouke to thank- he sat near Jeff Dean and stuffed 8 GPUs into a workstation to prove that he could do training faster than thousands of CPUs in prod) and deep neural networks. Fast forward a few years, and I'm an expert in ML, and the only suggestion I have is that everybody should read and internalize: https://research.google/pubs/pub43146/ https://research.google/pubs/pub43146/ So little of success in ML comes from the sexy algorithms and so much just comes from ensuring a bunch of boring details get properly saved in the right place.

moedersmooiste 5y ago

I always have great success doing anomaly detection with basic standard deviation in some SQL queries...

rdevsrex 5y ago

So, I am a total ML noob. The thing I haven't found a straight answer to is, what is a model. I mean when it is in production? Is it just some random blob that you pipe data into and get data out?

Mentlo 5y ago
Depends on how it's put into production, but you can deploy a model as a RESTful API that has a defined interface and a defined output. What it does underneath is less important to you I guess. So for all intents and purposes, yes, a model in production is something you feed a predefined set of data points and it gives you a predefined format of output.
jazzyjackson 5y ago
So, you know how a straight line is defined as mx + b, where you just have two parameters: slope and intercept ? Your input value is X, you multiply it by your slope and add your intercept to get the output (the Y value on the line). The 'training' of an ML algo is really just finding the line-of-best-fit so that you can make predictions. So your line-of-best-fit is encoded in these two parameters, allowing you to make predictions about what the output would be for arbitrary input. The problems people are throwing at ML have many more parameters and dimensions, but the training is a matter of finding those parameters that come closest to predicting the outcome. The 'model' is this set of parameters that allows the function to make predictions. (disclaimer: also an ML noob, correct me if I'm wrong)
- rdevsrex 5y ago
  thanks!

lobo_tuerto 5y ago

Seems like antirez (from Redis fame) doesn't agree with this: https://twitter.com/antirez/status/1440711992038158336 https://twitter.com/antirez/status/1440711992038158336

xyzzy21 5y ago

This is generally correct about ALL technologies. You should NEVER start with a solution and look for a problem except in a very general sense (e.g. looking for potential markets in the abstract). Taking a solution market without having the problem well defined and identified is absolutely Epic Fail. Once you think you have a market, you should see how it can be done FIRST without your "fancy miracle technology" because NOBODY buys a technology because it's sexy or trendy: they buy because it added value in terms of more capability or lower costs. And ALL problems have current solutions that almost certainly DO NOT use anything as complex as your technology solution so you have to trend very carefully and deliberately in a rational sense: what value are we REALLY adding? That starts with knowing your competition and the current solution to solving the problem first and then finding every reason why your technology won't work or will be problematic. You ONLY have market potential once you've exhausted those faults or have objective arguments for your value proposition that have been validated by actual customers. The actual prove is made when they are willing to write a PO to you for the solution. Until then, everything you are doing is unproven.

samuel2 5y ago
well said, thx for your comment

amts 5y ago

ML users may not be aware enough that the ecological validity of studies with peri- and intracellular recordings of individual neurons, which form the empirical basis for activation function(s), is very low in relation to general mind functions, which are supposed to be measured by a bit more, but still not 100% ecologically valid methods like fMRI etc and are thought to be simulated by ML.