4 ms·
Yeah, One thing that I didn’t realise until sometime afterwards was just how challenging it is to handle issues like model decay, evaluation of models in produ
by joe_the_user 11y ago
Yeah,
One thing that I didn’t realise until sometime afterwards was just how challenging it is to handle issues like model decay, evaluation of models in production, dev-ops etc all by yourself.
Honestly, it seems like any standard machine learning application (based on train/test/deploy/etc) is going to be more subject to decay/bit-rot/etc than some equivalently complex application with a codified specification.
It seems like something over time is going to be a problem for any institution which relies on such applications.
- danso 11y agoReminds me of John Foreman (data scientist at Mailchimp) and his excellent blog, including this piece, Surviving Data Science at the Speed of Hype: http://www.john-foreman.com/blog/surviving-data-science-at-the-speed-of-hype http://www.john-foreman.com/blog/surviving-data-science-at-t... > Over the course of my career, I've built optimization models for a number of businesses, some large, like Royal Caribbean or Coke, some smaller, like MailChimp circa 2012. And the one thing I've learned about optimization models, for example, is that as soon as you've "finished" coding and deploying your model the business changes right under your nose, rendering your model fundamentally useless. And you have to change the optimization model to address the new process... > Process stability and "speed of BLAH" are not awesome bedfellows. Supervised AI models hate pivoting. When a business is changing a lot, that means processes get monkeyed with. Maybe customer support starts working in different shifts, maybe a new product gets released or prices are changed and that shifts demand from historical levels, or maybe your customer base changes to a younger demographic than your ML models have training data for targeting... Until the machines are literally thinking for themselves, it seems that being able to engineer a stable infrastructure for rapid debugging and sampling of feedback is as paramount as being able to do the math for the modeling.
- TeeWEE 11y agoWait? What has bit-rot todo with machine learning? I dont understand. Where can i learn about it?
- slv77 11y agoBit rot is the degradation of the performance of a ML model over time. When a model is trained it is learning a set of assumptions about the "universe" and over time the universe changes which can invalidate some of those assumptions which degrades the models performance. For example you create a model to classify tweets by sentiment and then people start referring to your company as sick and phat which your model doesn't understand. The bigger challenge is that most models attempt to predict human behavior but humans have the ability to learn and so simply deploying the model changes the universe. For example your FICO score is an attempt to predict your credit default risk but over time people learn how to trivially manipulate their scores which degrades the ability to use FICO scores to predict default risk.