25 ms·

Temporal quality degradation in AI models

325 points by santiviquez 4y ago

anjc 4y ago
Many comments are focussed on how this will affect models in deployment, but a more troubling implication for me is that evaluations and comparisons across models using past data are affected by this.
drKarl 4y ago
Would LLMs forget things, and would they be affected by the Mandela effect?
- Vecr 4y ago
  Not sure what they mean, if all the code and model data are the same, it's won't forget anything encoded in the model. Of course the actual post-training context window is really short, it won't remember anything at all that falls out of the context window.
nequo 4y ago
The published study (open access!) that the post writes about: https://www.nature.com/articles/s41598-022-15245-z https://www.nature.com/articles/s41598-022-15245-z
- noobermin 4y ago
  Generally sci rep is open access.
Tepix 4y ago
These models are so human-like, they even imitate aging! /s
danidiaz 4y ago
https://qntm.org/mmacevedo https://qntm.org/mmacevedo > However, even for these tasks, its performance has dropped measurably since the early 2060s and is now considered subpar compared to more recent uploads. This is primarily attributed to MMAcevedo's lack of understanding of the technological, social and political changes which have occurred in modern society since its creation in 2031. This phenomenon has also been observed in other uploads created after MMAcevedo, and is now referred to as context drift.
- noobermin 4y ago
  Holy shit, I read the article and assumed I heard of the term before since I kinda of remembered the definition, but I didn't make the connection. I'm not sure how to feel about that it has turned into a term AI people actually use.
  - gield 3y ago
    The terms concept drift and (to a lesser extent) context drift have been used in AI for decades. The NannyML article uses the term concept drift. Here [1] is a 1996 paper that uses both terms. [1] https://link.springer.com/content/pdf/10.1007/BF00116900.pdf https://link.springer.com/content/pdf/10.1007/BF00116900.pdf
- zwirbl 4y ago
  what a great and at the same time deeply unsettling story, thanks for posting
  - fho 4y ago
    The crazy thing is that the story is quite old now ... at least ten years? Edit: Oh ... It's from 2021? Feels like ten years to me : - (
    - CarVac 4y ago
      The first draft was 2020. https://qntm.org/lena https://qntm.org/lena Still feels like ages ago somehow.
- mattbee 4y ago
  Came here with the same quote in mind :) This whole thing has such "Don't Create The Torment Nexus" vibes.
- zenon 4y ago
  Someone formatted the story like a Wikipedia article: https://i.imgur.com/i0sze4U.jpg https://i.imgur.com/i0sze4U.jpg
visarga 4y ago
Direct link to paper: https://www.nature.com/articles/s41598-022-15245-z https://www.nature.com/articles/s41598-022-15245-z > Using datasets from four different industries (healthcare operations, transportation, finance, and weather) and four standard machine learning models, we identify and describe the main temporal degradation patterns. So they focus on shifting distributions and discover that models degrade in time, it should be an obvious thing, if you don't retrain on new data. It's not about LLMs or diffusion models degrading. Time series prediction is a classical branch of ML with techniques like Moving Average (MA), Autoregressive Integrated Moving Average (ARIMA) or LSTM.
- qsort 4y ago
  Data drift is a problem regardless of the model/prediction task. Which is why models should be monitored in production, something everyone agrees is important, but nobody actually does. It's a new era for "computer says no" types to revel in.
  - noobermin 4y ago
    Sometimes though, "computer says no" is in fact the goal for some people. That some people dont realize so just demonstrates naivety.
    - qsort 4y ago
      It doesn't mean I have to accept that it's the computer's fault, though. Forcing them to take responsibility for their own decision is not naive, it's the counter to that strategy.
      - noobermin 4y ago
        To be clear, I'm not saying that you specifically are naive. I do believe though that this naiveity exists here in tech amongst some proponents.
  - PeterisP 4y ago
deleted 4y ago
[deleted]
he0001 4y ago
If a ML model trains on data that is produced on the internet, and we keep getting more and more internet content produced by ML, wouldn’t that mean that this will introduce a natural degradation of quality of all models? I believe that ML will produce more content faster than the rate of humans correcting it?
- worldofmatthew 4y ago
  AI training data eating AI generated content is very much something that can happen. Combine that with less people willing to write real content, just for it to be stolen for AI training data will see the internet be effectively destroyed as a place to get true information.
  - quickthrower2 4y ago
    We would need regulation to stop model ingesting data they do not have the right to, which would mean something like laws governing ML algorithms, having to declare what data you fed it and so on. Like some kind of SOC2 audit for data provenance. Maybe ML weights are just numbers, but then so is a movie, an mp3, a logo, a brand, and so on.
    - chrischattin 4y ago
      Regulation would introduce a whole new set of problems and unintended consequences.
      - Avamander 4y ago
        Which tradeoffs do we want though? Endlessly degrading AI vomit sounds much worse to me.
- baxuz 4y ago
  https://youtu.be/6i2l-LQ-dXI https://youtu.be/6i2l-LQ-dXI Don Hertzfeldt Couch Gag future
- 4y ago
smackeyacky 4y ago
In some ways, this is a kind of "no shit, sherlock" moment that all machine learning faces. It becomes a dead end, feedback loop like holding a microphone too close to a speaker. Funnily enough, genomic powered genetics in livestock production has the same characteristics. Without the flow of new phenotypic data the genomic models rot rapidly and become useless for prediction. Predictive language models powered by older predictive language models will quickly become nonsense. You have to remember just how deceptive something simple as Eliza was for people who didn't know how it worked.
- visarga 4y ago
  Changes to language are smaller. GPT-4 has a cutoff date in 2021 and is plenty smart and useful. It just doesn't know about the war in Ukraine.
  - smackeyacky 4y ago
    There are at least two things that make information wrong. One is "right, but not right now" i.e time. The right answer delivered late is wrong c.f. real time systems. The other is "garbage in, garbage out" which is related but a bit wider. While these models cannot distinguish input from older crap models, you cannot consider their output reliably right.
    - visarga 4y ago
      I have high hopes in dataset engineering. Scaling the model is already reaching the limits of hardware and funding, the training corpus is almost all the text written on the internet (by some accounts just 10% of it, so maybe we got a couple more years left there). So then what can we do? We turn to the quality of the data. Input examples can be enriched, facts can be checked for support. Based on existing corpus of text we can generate new training examples. Some of them, like code, can also contain execution data and tests.
      - gibolt 4y ago
chewz 4y ago
overfitted backtested models do not predict future very well... anyone surprised?
- thrdbndndn 4y ago
  Yeah, I think you summarized it well. The fact they kinda work for the first few years are mostly just because in real world data don't/can't change that fast. (Disclaimer: I only did ML work in academics, so take it with a grain of salt.)
  - chewz 4y ago
    Well, I have some experience with financial models where they work pretty well in shorter periods (could be months or years). And then there is sudden regime change... Pretty obvious but outside of model's scope...
malf 4y ago
> Another critical decision is that they only investigated pairs of model-dataset with good initial performance. groan
- bootsmann 4y ago
  Overfit in training and then complain that it doesn't generalize well in production, a true classic.
  - epups 4y ago
    If this is what happened, what explains the fact that some of these models performed great for about a year before decaying?
  - noobermin 4y ago
    This seems rather dismissive for something actually published in a real journal and not just on arxiv for once, right?
    - bootsmann 4y ago
      If you check the abstract, then this got published more for describing these drift patterns and showing ways to visualize and detect them rather than dropping the weird statistic the article makes it about.
b800h 4y ago
Shades of that Star Trek episode with Data's daughter in it.
teruakohatu 4y ago
In industry it is commonly accepted that all temporal models decay given enough time. Statistical process control (SPC) was developed to moniter physical processes (such as manufacturing) but can be applied to ML to moniter model output and help determine when retraining is required, or at least alert when a process appears to move outside control. The R qi library generates nice control charts. https://cran.r-project.org/web/packages/qicharts/vignettes/controlcharts.html https://cran.r-project.org/web/packages/qicharts/vignettes/c...
Borrible 4y ago
Life goes on and every time you remember, memory changes.
catoc 4y ago
100% of ML Models Degrade over More Time
- santiviquez 4y ago
  Haha true
cpa 4y ago
“In a real life setting, underlying distribution is stationary about 9% of the time”. Better than I would have expected, actually.
jasfi 4y ago
This is why I suggested that content generated as AI should be indicated as such with HTML tags, META at the high-level. But perhaps an property/value pair is better, such as aiGenerated=true. But as the use of AI generated content proliferates, this is unlikely to happen. We might one day assume that most content generated after a certain year is AI generated. Still, if people don't edit and refine that content enough, it will be considered junk and recommendation algorithms should take care of the rest, so that when GPT-10 crawls the web, it will leave out that junk.
noobermin 4y ago
I'm happy someone did this. People are making rumblings in traditional simulation modeling circles that ML models will replace traditional forward modelling...because I'm assuming a PM who hasn't touched code in a decade was impressed that chatGPT can write their emails for them. I feel like it was never the case that dumb fitting could replace a first principles model you already have in hand, and I always had the concern that extrapolation with these models is just a prone to error as it is when your model is linear least squares for a scatterplot, you cannot fit to data that just isn't in your initial dataset! Sure you can initially but at some point your model breaks down as any model does, a priori, statistical, or whatever. I could have always told you that (and I told my peers that for yeara) because the fact is intuitively obvious but it's great someone is actually publishing about it now. Now we just need similar papers for language models, which we sort of already experience when you ask chatGPT about 2023 events but somehow still cannot pierce the minds of the hypemen.
mjburgess 4y ago
One, I think overlooked issue with AI as ML, is how radically it affords the done-and-finished past over a creative future. When faced with a blank page some percentage of people will not copy a known solution, they'll draw on their experiences and eg., write a book about them. For example, I could write a book about software development without research only inventorying my opinions, arguing for them, illustrating them, and so on. If I were to start with ChatGPT I would, necessarily, only say what has been said. There are so many dangers with the Eliza-effect, and this is one of them. I think the narrative on how these systems work needs more strongly wrestled from their salesmen -- and the real downsides exposed. A client of mine seemed quite insistent that ChatGPT was "creative". I hope in a few years that opinion will be rare, as everyone ends up writing the same article over-and-over. And likewise, in development, using earlier versions of libraries with the most "stackoverflow training data"; using dying languages with the most github repos; and so on. In other words: what we have written the most about is not what we now need to write the most about. And the former is the training data for modern AI. It is not in the world with us, with our concerns: it is a statistical average over "done and dead" cases.
- chii 4y ago
  > inventorying my opinions but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. > If I were to start with ChatGPT I would, necessarily, only say what has been said. And i would also imagine that this isn't too different from someone with a vast experience behind them. And these ML models don't just repeat what has been said - i think there's enough complexity in the model where they could genuinely extract knowledge (or patterns) across different areas, and have it be applied to specific circumstances.
  - iudqnolq 4y ago
    I'm less confident it innovates based on my practical experience with Copilot. It writes boring, verbose code. And it assumes APIs are written in that style as well. That effect is worse the less it knows about an API. So if you want it to be able to complete based on the rest of your code in your project you'd have to tailor your APIs to it, which I won't do. TabNine has a much more primitive model, but I like that they weigh the code on your hard drive much heavier.
noobcoder 4y ago
I have some reservations about the methodology used in this study. The framework seems to oversimplify the complexity of real-world scenarios, and the results may not be generalizable to all types of ML models. The study only focused on temporal data drifts and did not consider other types of data drifts that can also affect model performance
RichieAHB 4y ago
Caveat, I have no experience in ML and a few parts of that article went over my head. However, this seems sensible for application specific ML models, as societal changes move the goalposts. My follow up question is: if a much more general ML model becomes so fundamental to daily life that it steers a human population’s attitudes towards a large amount of the questions they face, would the ML degrade or would it be possible that the population’s behaviour stays within some performant bound for that ML model? For example, could an airport delay remain accurate for a long period of time based on that same huge model controlling many of the outputs that are the inputs for that model? Obviously this doesn’t mean the absolute results remain static, just that the inputs to the model stay within a performant domain of values for the underlying model? Edit: I guess this would describe less of a model describing a prediction of some external dataset and begin approaching more of a system that controlled the inputs. So perhaps this questions isn’t really relevant to the utility of most (all?) modern ML, which is predicting changes in data that is fully “external”.
carom 4y ago
We need living datasets for computer vision. There is no current mechanism for updating something like ImageNet. There is no place you go to point out problems or contribute changes to keep it in line. We really need to make datasets social. This has a whole host of challenges like copyright, ownership, versioning, and even hosting costs. Even with all that it is a tractable problem though.
- RicoElectrico 4y ago
  HuggingFace is almost there. Uploading a re-trained or fine-tuned model is trivial. Same with datasets.
- fullstackchris 4y ago
  > the data drifts It's in fact the data itself is the closest to reality, not the ML model... It's a conundrum in the flow of reality for me: Reality -> data that reflects reality (hopefully as best it can) -> train model based on data (typically takes a non-trivial time scale!) -> make decision / do stuff based on algo with newest data If you ask me, even with all known technology we have, seems like it's impossible to simultaneously aquire realtime data and train a model to operate and or make decisions on exactly that data. It's a catch 22, there will always be a lag Even as humans with super big brains we can't hope to do this outside of extremely simple tasks like "throw and catch the ball"
  - grumple 4y ago
    It's not exactly a hard problem to continuously train a model - although it may be costly. You can even train the model based on every interaction it has; but this quickly leads to degradation because users provide it with data that is of low quality, for example when they intentionally try to make chatbots says racist things, etc.
timlod 4y ago
Some of those graphs are rather hard to pass, as the same domain is presented twice side-by-side, but with different X- and Y-axes. For example, comparing NN and linear regression models we see orders of magnitude difference on the Y-axis. Would really take the point of 'no change in RV model' home if they had the same scale. Apart from that, many years ago, at a previous job, I prototyped automatic recognition of model degradation. I think one of the approaches I tried was comparing KL divergence between predictions on the test set and new data. This way you don't need to have the ground truth available (in which case you can make measures based on the residuals). Worked quite well iirc to signal when one should retrain and/or investigate context drift. Nowadays, with relative maturity in explainability methods, you can go much further and can much better find out what exactly causes the drift. In general, I'm not a big fan of just saying 'error's up, let's retrain'. Why did the error go up? Understanding the data here better is usually quite valuable in the business context.
BiasRegularizer 4y ago
Distribution shift in the real world data will always be inherent to any data driven methods. Unless there are major advances in continual learning for DL models, they will always struggle with distribution shift degradation. Similarly, humans are also prone to the distribution shift unless we get updated information on a specific topic. The key differences are that we are great at continual learning and we are much better at learning abstraction
quickthrower2 4y ago
I am just learning but I am guessing the issue is training is expensive, because you need to iterate many times, and do a massive bunch of matrix operations (and other operations) per iteration? Maybe too expensive to "keep up" with new data coming in? Does a single new piece of data mean you need to start from scratch, or are you "near enough" and so it needs less processing power to incorporate it?
alexmolas 4y ago
Using NN for tabular data has been proved to be a bad idea. I don't think the results regarding NN from this study can be directly applied to other domains such as CV or NLP. However, as others have stated, model degradation is a common problem to all machine learning production models.
lifeinthevoid 4y ago
exactly 91% (sorry for the trolling, couldn't resist)
- catoc 4y ago
  That was my first reaction ("91% ? are you sure it's not 91.2%?") but it's actually a really good blog post.
lahsuk 4y ago
I think AI models are different form ML models, atleast mathematically. ML models and the mathematics behind it are explained thoroughly and people (who learnt about ML models) why certain ML models work or do not work. The AI models which are sprining left and right now I doubt people(again who learnt about AI models) have complete understanding about why they work, similar to how X-rays were used by doctors but they did not know why it worked. I guess for buisness case of ChatGPT if it is brining in the money why should it not be used?
- RamblingCTO 4y ago
  I don't think that's a valid distinction. There's no such thing as "AI" models imho. ChatGPT is also just ML. It's just a specific type of model that's hard to explain because it has so many parameters and embeddings in-between layers and stuff like that. It's not as easily explainable as decision tree-based models maybe. You would be surprised by how many models are actually hard to explain.
jll29 4y ago
> A recent study from MIT, Harvard, The University of Monterrey, and Cambridge showed that.. The post implies ¨Cambridge¨ (University) was involved in this study, which is not correct: it is a purely U.S.-based study by the following four institutions: Monterrey Institute of Technology and Higher Education, Monterrey, Mexico Massachusetts Institute of Technology, Cambridge, USA Whitehead Institute for Biomedical Research, Cambridge, USA Harvard University Medical School, Boston, USA
- santiviquez 4y ago
  Ups that's my bad. Already fixed it. Thanks for pointing it out :)
  - thrdbndndn 4y ago
    You still didn't fix it. There is no "The University of Monterrey". "Monterrey Institute of Technology and Higher Education" is a different university from University of Monterrey.
- thrdbndndn 4y ago
  Small correction, the first one is Mexico-based, not U.S.-based.
Havoc 4y ago
Phrasing like "losing relevance" would be more appropriate than degrade here I think
RamblingCTO 4y ago
Parameter shift is nothing new.
GnarlyWhale 4y ago
I challenged Sutton's reinforcement learning course this term (in spite lacking the appropriate pre-requisites) and, while my grasp of some of the mathematical theory was indeed lacking, some pretty foundational messaging really penetrated. Incorporating curiosity and exploration of new experience into learning algorithms offers so much more utility versus relying solely on offline training and past experiences alone.
borissk 4y ago
100% of human learning models degrade over time :) And it's not only aging. There's a well known saying: "Science Makes Progress Funeral by Funeral". Humans have very hard time changing their believes to adapt to new data, not so surprising that ML models experience a similar phenomenon. As the great Max Planck, himself the originator of the quantum theory in physics, has said, science makes progress funeral by funeral: the old are never converted by the new doctrines, they simply are replaced by a new generation.
xiaodai 4y ago
the other 9% just plain don't work
stuckkeys 4y ago
I look forward to the day where I stop seeing AI shit on my linkedin feed. I took the GPT4 for a test drive. I spent more time telling it, “not that way, but instead this way” then it will respond with “o shit, my bad. You are right. Here is the answer.” Sure it can help with ideas, and potentially improve your critical thinking skills but in terms of efficiency It is certainly not. Can it improve over time? Most likely. But the data is going to be questionable.
- taberiand 4y ago
  I'd stop using a hammer too, if all I did was bash my thumb with it (or to be fair, if the work requires finer precision/more sophisticated tools than what the ChatGPT hammer can currently provide)
  - stuckkeys 4y ago
    That is a interesting take. I like it.
pelorat 4y ago
Obviously. This is an inherent property of neural networks and applies to both biologicals neural networks as well as artificial ones. Whenever you as a person learn something new, you are degrading some other memory that you have.
dschuetz 4y ago
I am not sure if AI aging is the right term to explain what is happening there. If there is a drift in training data that is supposed to feed the model new patterns what will result is not an updated model, but rather a more and more confused one. "Wait, you are saying: yesterday the value of money was 1.0, but today it is 0.99? When you query what is the value of money, please explain first what 'today' and 'yesterday' means please sir" So, if I am getting this right: you cannot update an already trained model with updated data. Doing so will "confuse" the model with contradictions, without the ability to assess what information or concept is valid now. The consequence would be to train new models with the updated training data each time, and discard the old ones. Which is not viable at all, to be honest.
- tap-snap-or-nap 4y ago
  Why is it not viable, do you believe that the costs will exceed the potential benefits even with the improvements that it offers ?
  - dschuetz 4y ago
    Completely retraining AIs with slightly updated datasets will be costly, energy-wise, definitely.
deleted 4y ago
[deleted]
sorenjan 4y ago
Are there ML models that are trained while being used? Humans learn as we go along, but this "train and deploy" that's so common doesn't seem sustainable. If it can't be done online, then it shouldn't be that difficult to save the output for each week and then finetune the production model on that each weekend, right? Especially if there are humans correcting the faulty or uncertain outputs.
- logophobia 4y ago
  That's a pretty standard part of MLOps. I have a fraud model in production, it's being incrementally retrained each week, on a sliding window of data for the last x-months. You can do it "online", which works for some models, but for most need monitoring to make sure they don't go off the rails.
  - sorenjan 4y ago
    That's good to hear, how does it work in practice? Is it basically running the same training as from scratch, but with only the new data, on a separate machine to produce a new version which is then replacing the old production version? Is part of MLOps starting a new training session each week, checking if the loss function looks ok, and then redeploying it? I still think of how humans work. We don't get retrained from time to time to improve, we learn continually as we gain experience. It should be doable in at least some cases, like classification where it's easy to tell if a label is right or wrong.
    - logophobia 4y ago
      The typical setup is very simple: * Take the previous model checkpoint, retrain/finetune it on a window with new data. You typically don't want to retrain everything from scratch, saves time and money. For large models you need specialized GPUs to train them, so typically the training happens separately. * Check the model statistics in depth. We look at way more statistics then just the loss function. * Check actual examples of the model in action * Check the data quality. If the data is bad, then you're just amplifying human mistakes with a model. * Push it to production, monitor the result MLOps practice differs from to team to team, this checklist isn't universal, just one possible approach. Everyone does things a little differently. > I still think of how humans work. We don't get retrained from time to time to improve, we learn continually as we gain experience. It should be doable in at least some cases, like classification where it's easy to tell if a label is right or wrong. For some models, like fraud, correctness is important. Those models need a lot of babysitting. For humans, think about how the average facebooker reacts to misinformation, you don't want that to happen with your model. Other models are ok with more passive monitoring, things like recommendation systems. Continuous online training can be done. Maybe take a look at reinforcement learning? It's not widely applied, has some limitations, but also some interesting applications. These types of things might become more common in the future.
skMed 4y ago
> continuously monitoring the estimated model performance Main point here. Model performance can degrade for any number of reasons and at varying rates. As a starting point, focus on setting up anomaly monitoring for a robust set of model eval metrics tailored to your task: loss, calibration, model staleness, etc. Timely alerts can give you sufficient time to dig in and root cause, roll back a model, etc.
robwwilliams 4y ago
These comments are only very marginally associated with the original post. I was hoping for some insight into model drift.
golemotron 4y ago
I don't think the metaphor of aging makes the problem clearer or approaches the root cause of this phenomenon. A model doesn't 'age' it just stops learning. When we see it that way, the solution is clear and it can be understood in the context of all systems that must change to be effective or survive in their environment. This is one place where anthropomorphism can help.
phemartin 4y ago
Listen to this article (14min) at https://playtext.app/doc/clggm0702001alg0ge6brppom https://playtext.app/doc/clggm0702001alg0ge6brppom
mensetmanusman 4y ago
A feedback loop where AI trains on second order regurgitation of what it has outputted almost necessitated that this happens (and oscillates).
sublinear 4y ago
Why don't we just ask ChatGPT about itself? Duh. /s
aaroninsf 4y ago
Evergreen, given the occurrent ITT of critiques which misunderstand this work as "assertion of an Achilles' Heel" rather than of one among many bugs in the tracker which will duly get prioritized and fixed: Ximm's Law: every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon. Lemma: any statement about AI which uses the word "never" to preclude some feature from future realization is false. Footnote de rigeur warning about reasoning about these systems from a fixed-mindset as "stochastic parrots": these things are absolutely nothing like Markov chain abacuses, ELIZA, or indeed parrots; holding an over-simplified model prevents actual reasoning—especially given the non-linearities of improvement which continue to be demonstrated. C.f. "GPT-5's unicorn." Hypothesis: the optimal internal representation of the world needed to generate language homologous to it, is one which performs abstraction into world-models and state, and learns predictive functions about those models, testing them in the Sagan sense against data. QED such things are not an accident but an inevitable emergent property of these systems, one which indeed we continue to see emerging. TL;DR be very careful of partaking of cynicism and skepticism based on inappropriately simple and rapidly-deprecating models of the status quo.
eurekin 4y ago
For all initially surprised, as to how is it possible for an AI model to age, here are example models measured: - Patient wait time dataset - Weather temperature dataset - Financial stock price dataset - Airport departure delay dataset I'm actually shocked that kind of ML models exist. I'm glad I didn't pursue that career path.
napier 3y ago
Misleading paper title and misleadingly presented research findings. Models don't "degrade" over time; their timeliness and relevance remains frozen pursuant to fine-tuning updates while contemporaneous attentional demands upon them by users drift. They don't wear out like a pair of shoes.

325 points by santiviquez 4y ago

anjc 4y ago

Many comments are focussed on how this will affect models in deployment, but a more troubling implication for me is that evaluations and comparisons across models using past data are affected by this.

drKarl 4y ago

Would LLMs forget things, and would they be affected by the Mandela effect?

Vecr 4y ago
Not sure what they mean, if all the code and model data are the same, it's won't forget anything encoded in the model. Of course the actual post-training context window is really short, it won't remember anything at all that falls out of the context window.

nequo 4y ago

The published study (open access!) that the post writes about: https://www.nature.com/articles/s41598-022-15245-z https://www.nature.com/articles/s41598-022-15245-z

noobermin 4y ago
Generally sci rep is open access.

Tepix 4y ago

These models are so human-like, they even imitate aging! /s

danidiaz 4y ago

https://qntm.org/mmacevedo https://qntm.org/mmacevedo > However, even for these tasks, its performance has dropped measurably since the early 2060s and is now considered subpar compared to more recent uploads. This is primarily attributed to MMAcevedo's lack of understanding of the technological, social and political changes which have occurred in modern society since its creation in 2031. This phenomenon has also been observed in other uploads created after MMAcevedo, and is now referred to as context drift.

noobermin 4y ago
Holy shit, I read the article and assumed I heard of the term before since I kinda of remembered the definition, but I didn't make the connection. I'm not sure how to feel about that it has turned into a term AI people actually use.
- gield 3y ago
  The terms concept drift and (to a lesser extent) context drift have been used in AI for decades. The NannyML article uses the term concept drift. Here [1] is a 1996 paper that uses both terms. [1] https://link.springer.com/content/pdf/10.1007/BF00116900.pdf https://link.springer.com/content/pdf/10.1007/BF00116900.pdf
zwirbl 4y ago
what a great and at the same time deeply unsettling story, thanks for posting
- fho 4y ago
  The crazy thing is that the story is quite old now ... at least ten years? Edit: Oh ... It's from 2021? Feels like ten years to me : - (
  - CarVac 4y ago
    The first draft was 2020. https://qntm.org/lena https://qntm.org/lena Still feels like ages ago somehow.
mattbee 4y ago
Came here with the same quote in mind :) This whole thing has such "Don't Create The Torment Nexus" vibes.
zenon 4y ago
Someone formatted the story like a Wikipedia article: https://i.imgur.com/i0sze4U.jpg https://i.imgur.com/i0sze4U.jpg

visarga 4y ago

Direct link to paper: https://www.nature.com/articles/s41598-022-15245-z https://www.nature.com/articles/s41598-022-15245-z > Using datasets from four different industries (healthcare operations, transportation, finance, and weather) and four standard machine learning models, we identify and describe the main temporal degradation patterns. So they focus on shifting distributions and discover that models degrade in time, it should be an obvious thing, if you don't retrain on new data. It's not about LLMs or diffusion models degrading. Time series prediction is a classical branch of ML with techniques like Moving Average (MA), Autoregressive Integrated Moving Average (ARIMA) or LSTM.

qsort 4y ago
Data drift is a problem regardless of the model/prediction task. Which is why models should be monitored in production, something everyone agrees is important, but nobody actually does. It's a new era for "computer says no" types to revel in.
- noobermin 4y ago
  Sometimes though, "computer says no" is in fact the goal for some people. That some people dont realize so just demonstrates naivety.
  - qsort 4y ago
    It doesn't mean I have to accept that it's the computer's fault, though. Forcing them to take responsibility for their own decision is not naive, it's the counter to that strategy.
    - noobermin 4y ago
      To be clear, I'm not saying that you specifically are naive. I do believe though that this naiveity exists here in tech amongst some proponents.
- PeterisP 4y ago

deleted 4y ago

[deleted]

he0001 4y ago

If a ML model trains on data that is produced on the internet, and we keep getting more and more internet content produced by ML, wouldn’t that mean that this will introduce a natural degradation of quality of all models? I believe that ML will produce more content faster than the rate of humans correcting it?

worldofmatthew 4y ago
AI training data eating AI generated content is very much something that can happen. Combine that with less people willing to write real content, just for it to be stolen for AI training data will see the internet be effectively destroyed as a place to get true information.
- quickthrower2 4y ago
  We would need regulation to stop model ingesting data they do not have the right to, which would mean something like laws governing ML algorithms, having to declare what data you fed it and so on. Like some kind of SOC2 audit for data provenance. Maybe ML weights are just numbers, but then so is a movie, an mp3, a logo, a brand, and so on.
  - chrischattin 4y ago
    Regulation would introduce a whole new set of problems and unintended consequences.
    - Avamander 4y ago
      Which tradeoffs do we want though? Endlessly degrading AI vomit sounds much worse to me.
baxuz 4y ago
https://youtu.be/6i2l-LQ-dXI https://youtu.be/6i2l-LQ-dXI Don Hertzfeldt Couch Gag future
4y ago

smackeyacky 4y ago

In some ways, this is a kind of "no shit, sherlock" moment that all machine learning faces. It becomes a dead end, feedback loop like holding a microphone too close to a speaker. Funnily enough, genomic powered genetics in livestock production has the same characteristics. Without the flow of new phenotypic data the genomic models rot rapidly and become useless for prediction. Predictive language models powered by older predictive language models will quickly become nonsense. You have to remember just how deceptive something simple as Eliza was for people who didn't know how it worked.

visarga 4y ago
Changes to language are smaller. GPT-4 has a cutoff date in 2021 and is plenty smart and useful. It just doesn't know about the war in Ukraine.
- smackeyacky 4y ago
  There are at least two things that make information wrong. One is "right, but not right now" i.e time. The right answer delivered late is wrong c.f. real time systems. The other is "garbage in, garbage out" which is related but a bit wider. While these models cannot distinguish input from older crap models, you cannot consider their output reliably right.
  - visarga 4y ago
    I have high hopes in dataset engineering. Scaling the model is already reaching the limits of hardware and funding, the training corpus is almost all the text written on the internet (by some accounts just 10% of it, so maybe we got a couple more years left there). So then what can we do? We turn to the quality of the data. Input examples can be enriched, facts can be checked for support. Based on existing corpus of text we can generate new training examples. Some of them, like code, can also contain execution data and tests.
    - gibolt 4y ago

chewz 4y ago

overfitted backtested models do not predict future very well... anyone surprised?

thrdbndndn 4y ago
Yeah, I think you summarized it well. The fact they kinda work for the first few years are mostly just because in real world data don't/can't change that fast. (Disclaimer: I only did ML work in academics, so take it with a grain of salt.)
- chewz 4y ago
  Well, I have some experience with financial models where they work pretty well in shorter periods (could be months or years). And then there is sudden regime change... Pretty obvious but outside of model's scope...

malf 4y ago

> Another critical decision is that they only investigated pairs of model-dataset with good initial performance. groan

bootsmann 4y ago
Overfit in training and then complain that it doesn't generalize well in production, a true classic.
- epups 4y ago
  If this is what happened, what explains the fact that some of these models performed great for about a year before decaying?
- noobermin 4y ago
  This seems rather dismissive for something actually published in a real journal and not just on arxiv for once, right?
  - bootsmann 4y ago
    If you check the abstract, then this got published more for describing these drift patterns and showing ways to visualize and detect them rather than dropping the weird statistic the article makes it about.

b800h 4y ago

Shades of that Star Trek episode with Data's daughter in it.

teruakohatu 4y ago

In industry it is commonly accepted that all temporal models decay given enough time. Statistical process control (SPC) was developed to moniter physical processes (such as manufacturing) but can be applied to ML to moniter model output and help determine when retraining is required, or at least alert when a process appears to move outside control. The R qi library generates nice control charts. https://cran.r-project.org/web/packages/qicharts/vignettes/controlcharts.html https://cran.r-project.org/web/packages/qicharts/vignettes/c...

Borrible 4y ago

Life goes on and every time you remember, memory changes.

catoc 4y ago

100% of ML Models Degrade over More Time

santiviquez 4y ago
Haha true

cpa 4y ago

“In a real life setting, underlying distribution is stationary about 9% of the time”. Better than I would have expected, actually.

jasfi 4y ago

This is why I suggested that content generated as AI should be indicated as such with HTML tags, META at the high-level. But perhaps an property/value pair is better, such as aiGenerated=true. But as the use of AI generated content proliferates, this is unlikely to happen. We might one day assume that most content generated after a certain year is AI generated. Still, if people don't edit and refine that content enough, it will be considered junk and recommendation algorithms should take care of the rest, so that when GPT-10 crawls the web, it will leave out that junk.

noobermin 4y ago

I'm happy someone did this. People are making rumblings in traditional simulation modeling circles that ML models will replace traditional forward modelling...because I'm assuming a PM who hasn't touched code in a decade was impressed that chatGPT can write their emails for them. I feel like it was never the case that dumb fitting could replace a first principles model you already have in hand, and I always had the concern that extrapolation with these models is just a prone to error as it is when your model is linear least squares for a scatterplot, you cannot fit to data that just isn't in your initial dataset! Sure you can initially but at some point your model breaks down as any model does, a priori, statistical, or whatever. I could have always told you that (and I told my peers that for yeara) because the fact is intuitively obvious but it's great someone is actually publishing about it now. Now we just need similar papers for language models, which we sort of already experience when you ask chatGPT about 2023 events but somehow still cannot pierce the minds of the hypemen.

mjburgess 4y ago

One, I think overlooked issue with AI as ML, is how radically it affords the done-and-finished past over a creative future. When faced with a blank page some percentage of people will not copy a known solution, they'll draw on their experiences and eg., write a book about them. For example, I could write a book about software development without research only inventorying my opinions, arguing for them, illustrating them, and so on. If I were to start with ChatGPT I would, necessarily, only say what has been said. There are so many dangers with the Eliza-effect, and this is one of them. I think the narrative on how these systems work needs more strongly wrestled from their salesmen -- and the real downsides exposed. A client of mine seemed quite insistent that ChatGPT was "creative". I hope in a few years that opinion will be rare, as everyone ends up writing the same article over-and-over. And likewise, in development, using earlier versions of libraries with the most "stackoverflow training data"; using dying languages with the most github repos; and so on. In other words: what we have written the most about is not what we now need to write the most about. And the former is the training data for modern AI. It is not in the world with us, with our concerns: it is a statistical average over "done and dead" cases.

chii 4y ago
> inventorying my opinions but you assume that your opinions aren't formed from a preexisting body of knowledge from which your past self has learnt and internalized. > If I were to start with ChatGPT I would, necessarily, only say what has been said. And i would also imagine that this isn't too different from someone with a vast experience behind them. And these ML models don't just repeat what has been said - i think there's enough complexity in the model where they could genuinely extract knowledge (or patterns) across different areas, and have it be applied to specific circumstances.
- iudqnolq 4y ago
  I'm less confident it innovates based on my practical experience with Copilot. It writes boring, verbose code. And it assumes APIs are written in that style as well. That effect is worse the less it knows about an API. So if you want it to be able to complete based on the rest of your code in your project you'd have to tailor your APIs to it, which I won't do. TabNine has a much more primitive model, but I like that they weigh the code on your hard drive much heavier.

noobcoder 4y ago

I have some reservations about the methodology used in this study. The framework seems to oversimplify the complexity of real-world scenarios, and the results may not be generalizable to all types of ML models. The study only focused on temporal data drifts and did not consider other types of data drifts that can also affect model performance

RichieAHB 4y ago

Caveat, I have no experience in ML and a few parts of that article went over my head. However, this seems sensible for application specific ML models, as societal changes move the goalposts. My follow up question is: if a much more general ML model becomes so fundamental to daily life that it steers a human population’s attitudes towards a large amount of the questions they face, would the ML degrade or would it be possible that the population’s behaviour stays within some performant bound for that ML model? For example, could an airport delay remain accurate for a long period of time based on that same huge model controlling many of the outputs that are the inputs for that model? Obviously this doesn’t mean the absolute results remain static, just that the inputs to the model stay within a performant domain of values for the underlying model? Edit: I guess this would describe less of a model describing a prediction of some external dataset and begin approaching more of a system that controlled the inputs. So perhaps this questions isn’t really relevant to the utility of most (all?) modern ML, which is predicting changes in data that is fully “external”.

carom 4y ago

We need living datasets for computer vision. There is no current mechanism for updating something like ImageNet. There is no place you go to point out problems or contribute changes to keep it in line. We really need to make datasets social. This has a whole host of challenges like copyright, ownership, versioning, and even hosting costs. Even with all that it is a tractable problem though.

RicoElectrico 4y ago
HuggingFace is almost there. Uploading a re-trained or fine-tuned model is trivial. Same with datasets.
fullstackchris 4y ago
> the data drifts It's in fact the data itself is the closest to reality, not the ML model... It's a conundrum in the flow of reality for me: Reality -> data that reflects reality (hopefully as best it can) -> train model based on data (typically takes a non-trivial time scale!) -> make decision / do stuff based on algo with newest data If you ask me, even with all known technology we have, seems like it's impossible to simultaneously aquire realtime data and train a model to operate and or make decisions on exactly that data. It's a catch 22, there will always be a lag Even as humans with super big brains we can't hope to do this outside of extremely simple tasks like "throw and catch the ball"
- grumple 4y ago
  It's not exactly a hard problem to continuously train a model - although it may be costly. You can even train the model based on every interaction it has; but this quickly leads to degradation because users provide it with data that is of low quality, for example when they intentionally try to make chatbots says racist things, etc.

timlod 4y ago

Some of those graphs are rather hard to pass, as the same domain is presented twice side-by-side, but with different X- and Y-axes. For example, comparing NN and linear regression models we see orders of magnitude difference on the Y-axis. Would really take the point of 'no change in RV model' home if they had the same scale. Apart from that, many years ago, at a previous job, I prototyped automatic recognition of model degradation. I think one of the approaches I tried was comparing KL divergence between predictions on the test set and new data. This way you don't need to have the ground truth available (in which case you can make measures based on the residuals). Worked quite well iirc to signal when one should retrain and/or investigate context drift. Nowadays, with relative maturity in explainability methods, you can go much further and can much better find out what exactly causes the drift. In general, I'm not a big fan of just saying 'error's up, let's retrain'. Why did the error go up? Understanding the data here better is usually quite valuable in the business context.

BiasRegularizer 4y ago

Distribution shift in the real world data will always be inherent to any data driven methods. Unless there are major advances in continual learning for DL models, they will always struggle with distribution shift degradation. Similarly, humans are also prone to the distribution shift unless we get updated information on a specific topic. The key differences are that we are great at continual learning and we are much better at learning abstraction

quickthrower2 4y ago

I am just learning but I am guessing the issue is training is expensive, because you need to iterate many times, and do a massive bunch of matrix operations (and other operations) per iteration? Maybe too expensive to "keep up" with new data coming in? Does a single new piece of data mean you need to start from scratch, or are you "near enough" and so it needs less processing power to incorporate it?

alexmolas 4y ago

Using NN for tabular data has been proved to be a bad idea. I don't think the results regarding NN from this study can be directly applied to other domains such as CV or NLP. However, as others have stated, model degradation is a common problem to all machine learning production models.

lifeinthevoid 4y ago

exactly 91% (sorry for the trolling, couldn't resist)

catoc 4y ago
That was my first reaction ("91% ? are you sure it's not 91.2%?") but it's actually a really good blog post.

lahsuk 4y ago

I think AI models are different form ML models, atleast mathematically. ML models and the mathematics behind it are explained thoroughly and people (who learnt about ML models) why certain ML models work or do not work. The AI models which are sprining left and right now I doubt people(again who learnt about AI models) have complete understanding about why they work, similar to how X-rays were used by doctors but they did not know why it worked. I guess for buisness case of ChatGPT if it is brining in the money why should it not be used?

RamblingCTO 4y ago
I don't think that's a valid distinction. There's no such thing as "AI" models imho. ChatGPT is also just ML. It's just a specific type of model that's hard to explain because it has so many parameters and embeddings in-between layers and stuff like that. It's not as easily explainable as decision tree-based models maybe. You would be surprised by how many models are actually hard to explain.

jll29 4y ago

> A recent study from MIT, Harvard, The University of Monterrey, and Cambridge showed that.. The post implies ¨Cambridge¨ (University) was involved in this study, which is not correct: it is a purely U.S.-based study by the following four institutions: Monterrey Institute of Technology and Higher Education, Monterrey, Mexico Massachusetts Institute of Technology, Cambridge, USA Whitehead Institute for Biomedical Research, Cambridge, USA Harvard University Medical School, Boston, USA

santiviquez 4y ago
Ups that's my bad. Already fixed it. Thanks for pointing it out :)
- thrdbndndn 4y ago
  You still didn't fix it. There is no "The University of Monterrey". "Monterrey Institute of Technology and Higher Education" is a different university from University of Monterrey.
thrdbndndn 4y ago
Small correction, the first one is Mexico-based, not U.S.-based.

Havoc 4y ago

Phrasing like "losing relevance" would be more appropriate than degrade here I think

RamblingCTO 4y ago

Parameter shift is nothing new.

GnarlyWhale 4y ago

I challenged Sutton's reinforcement learning course this term (in spite lacking the appropriate pre-requisites) and, while my grasp of some of the mathematical theory was indeed lacking, some pretty foundational messaging really penetrated. Incorporating curiosity and exploration of new experience into learning algorithms offers so much more utility versus relying solely on offline training and past experiences alone.

borissk 4y ago

100% of human learning models degrade over time :) And it's not only aging. There's a well known saying: "Science Makes Progress Funeral by Funeral". Humans have very hard time changing their believes to adapt to new data, not so surprising that ML models experience a similar phenomenon. As the great Max Planck, himself the originator of the quantum theory in physics, has said, science makes progress funeral by funeral: the old are never converted by the new doctrines, they simply are replaced by a new generation.

xiaodai 4y ago

the other 9% just plain don't work

stuckkeys 4y ago

I look forward to the day where I stop seeing AI shit on my linkedin feed. I took the GPT4 for a test drive. I spent more time telling it, “not that way, but instead this way” then it will respond with “o shit, my bad. You are right. Here is the answer.” Sure it can help with ideas, and potentially improve your critical thinking skills but in terms of efficiency It is certainly not. Can it improve over time? Most likely. But the data is going to be questionable.

taberiand 4y ago
I'd stop using a hammer too, if all I did was bash my thumb with it (or to be fair, if the work requires finer precision/more sophisticated tools than what the ChatGPT hammer can currently provide)
- stuckkeys 4y ago
  That is a interesting take. I like it.

pelorat 4y ago

Obviously. This is an inherent property of neural networks and applies to both biologicals neural networks as well as artificial ones. Whenever you as a person learn something new, you are degrading some other memory that you have.

dschuetz 4y ago

I am not sure if AI aging is the right term to explain what is happening there. If there is a drift in training data that is supposed to feed the model new patterns what will result is not an updated model, but rather a more and more confused one. "Wait, you are saying: yesterday the value of money was 1.0, but today it is 0.99? When you query what is the value of money, please explain first what 'today' and 'yesterday' means please sir" So, if I am getting this right: you cannot update an already trained model with updated data. Doing so will "confuse" the model with contradictions, without the ability to assess what information or concept is valid now. The consequence would be to train new models with the updated training data each time, and discard the old ones. Which is not viable at all, to be honest.

tap-snap-or-nap 4y ago
Why is it not viable, do you believe that the costs will exceed the potential benefits even with the improvements that it offers ?
- dschuetz 4y ago
  Completely retraining AIs with slightly updated datasets will be costly, energy-wise, definitely.

deleted 4y ago

[deleted]

sorenjan 4y ago

Are there ML models that are trained while being used? Humans learn as we go along, but this "train and deploy" that's so common doesn't seem sustainable. If it can't be done online, then it shouldn't be that difficult to save the output for each week and then finetune the production model on that each weekend, right? Especially if there are humans correcting the faulty or uncertain outputs.

logophobia 4y ago
That's a pretty standard part of MLOps. I have a fraud model in production, it's being incrementally retrained each week, on a sliding window of data for the last x-months. You can do it "online", which works for some models, but for most need monitoring to make sure they don't go off the rails.
- sorenjan 4y ago
  That's good to hear, how does it work in practice? Is it basically running the same training as from scratch, but with only the new data, on a separate machine to produce a new version which is then replacing the old production version? Is part of MLOps starting a new training session each week, checking if the loss function looks ok, and then redeploying it? I still think of how humans work. We don't get retrained from time to time to improve, we learn continually as we gain experience. It should be doable in at least some cases, like classification where it's easy to tell if a label is right or wrong.
  - logophobia 4y ago
    The typical setup is very simple: * Take the previous model checkpoint, retrain/finetune it on a window with new data. You typically don't want to retrain everything from scratch, saves time and money. For large models you need specialized GPUs to train them, so typically the training happens separately. * Check the model statistics in depth. We look at way more statistics then just the loss function. * Check actual examples of the model in action * Check the data quality. If the data is bad, then you're just amplifying human mistakes with a model. * Push it to production, monitor the result MLOps practice differs from to team to team, this checklist isn't universal, just one possible approach. Everyone does things a little differently. > I still think of how humans work. We don't get retrained from time to time to improve, we learn continually as we gain experience. It should be doable in at least some cases, like classification where it's easy to tell if a label is right or wrong. For some models, like fraud, correctness is important. Those models need a lot of babysitting. For humans, think about how the average facebooker reacts to misinformation, you don't want that to happen with your model. Other models are ok with more passive monitoring, things like recommendation systems. Continuous online training can be done. Maybe take a look at reinforcement learning? It's not widely applied, has some limitations, but also some interesting applications. These types of things might become more common in the future.

skMed 4y ago

> continuously monitoring the estimated model performance Main point here. Model performance can degrade for any number of reasons and at varying rates. As a starting point, focus on setting up anomaly monitoring for a robust set of model eval metrics tailored to your task: loss, calibration, model staleness, etc. Timely alerts can give you sufficient time to dig in and root cause, roll back a model, etc.

robwwilliams 4y ago

These comments are only very marginally associated with the original post. I was hoping for some insight into model drift.

golemotron 4y ago

I don't think the metaphor of aging makes the problem clearer or approaches the root cause of this phenomenon. A model doesn't 'age' it just stops learning. When we see it that way, the solution is clear and it can be understood in the context of all systems that must change to be effective or survive in their environment. This is one place where anthropomorphism can help.

phemartin 4y ago

Listen to this article (14min) at https://playtext.app/doc/clggm0702001alg0ge6brppom https://playtext.app/doc/clggm0702001alg0ge6brppom

mensetmanusman 4y ago

A feedback loop where AI trains on second order regurgitation of what it has outputted almost necessitated that this happens (and oscillates).

sublinear 4y ago

Why don't we just ask ChatGPT about itself? Duh. /s

aaroninsf 4y ago

Evergreen, given the occurrent ITT of critiques which misunderstand this work as "assertion of an Achilles' Heel" rather than of one among many bugs in the tracker which will duly get prioritized and fixed: Ximm's Law: every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon. Lemma: any statement about AI which uses the word "never" to preclude some feature from future realization is false. Footnote de rigeur warning about reasoning about these systems from a fixed-mindset as "stochastic parrots": these things are absolutely nothing like Markov chain abacuses, ELIZA, or indeed parrots; holding an over-simplified model prevents actual reasoning—especially given the non-linearities of improvement which continue to be demonstrated. C.f. "GPT-5's unicorn." Hypothesis: the optimal internal representation of the world needed to generate language homologous to it, is one which performs abstraction into world-models and state, and learns predictive functions about those models, testing them in the Sagan sense against data. QED such things are not an accident but an inevitable emergent property of these systems, one which indeed we continue to see emerging. TL;DR be very careful of partaking of cynicism and skepticism based on inappropriately simple and rapidly-deprecating models of the status quo.

eurekin 4y ago

For all initially surprised, as to how is it possible for an AI model to age, here are example models measured: - Patient wait time dataset - Weather temperature dataset - Financial stock price dataset - Airport departure delay dataset I'm actually shocked that kind of ML models exist. I'm glad I didn't pursue that career path.

napier 3y ago

Misleading paper title and misleadingly presented research findings. Models don't "degrade" over time; their timeliness and relevance remains frozen pursuant to fine-tuning updates while contemporaneous attentional demands upon them by users drift. They don't wear out like a pair of shoes.