12 ms·

Machine Learning: The High Interest Credit Card of Technical Debt (2014)

576 points by amzans 8y ago

denzil_correa 8y ago
Publication Year : 2014 Previous HN Discussions https://news.ycombinator.com/item?id=10006293 https://news.ycombinator.com/item?id=10006293 https://news.ycombinator.com/item?id=10338575 https://news.ycombinator.com/item?id=10338575 https://news.ycombinator.com/item?id=8775772 https://news.ycombinator.com/item?id=8775772 "The Morning Paper" has a nice summary https://blog.acolyer.org/2016/02/29/machine-learning-the-high-interest-credit-card-of-technical-debt/ https://blog.acolyer.org/2016/02/29/machine-learning-the-hig...
- glup 8y ago
  which means it must be good, right? I feel like there are a few of these frequent flyers... is there anyway to figure out what they are?
  - chucksmash 8y ago
    "Show HN: Hacker News Classics" might fit the bill. Discussion: https://news.ycombinator.com/item?id=16442888 https://news.ycombinator.com/item?id=16442888 App: http://jsomers.net/hn/ http://jsomers.net/hn/ Source: https://github.com/jsomers/hacker-classics https://github.com/jsomers/hacker-classics
israrkhan 8y ago
Software Engineering daily has a podcast, with D. Sculley, the author of this paper. It is quite interesting to listen https://www.softwaredaily.com/post/5913c0e74ee01db33cacd027/Machine-Learning-and-Technical-Debt-with-D-Sculley https://www.softwaredaily.com/post/5913c0e74ee01db33cacd027/...
thelastidiot 8y ago
Is this still relevant?
- joker3 8y ago
  It's more relevant every year.
- glup 8y ago
  nah, thankfully technical debt has been solved.
  - setquk 8y ago
    Gnome solved it. Just burn the universe every five years!
    - paulie_a 8y ago
      That should honestly be a more frequently employed tactic in codebases.
      - favorited 8y ago
        I'm definitely a believer in this. I read Carmack refer to it as "grooming" at one point. Codebases where components and subsystems are periodically rewritten naturally end up being easier to change, which is an undervalued metric.
      - setquk 8y ago
        Definitely agree. However it mostly depends on your product being well isolated little chunks of functionality which is rarely the case. So what happens is you end up with a perpetually half baked Cthulhu.
    - bkor 8y ago
      2.0 came out around Sep 2002 (a bit less than 16 years ago). 3.0 came out in Apr 2011 (7 years ago). Most of the components were the same between the last 2.32 release and 3.0, except for the very visible gnome-panel to gnome-shell switch. Gnome isn't a good example.
      - 8y ago
imh 8y ago
I love this paper. The further I get in my ML/stats career, the more relevant these lessons are. I would recommend anyone interested in building long lived ML products to read this.
joker3 8y ago
This paper will eventually be seen as a landmark in the field of machine learning systems. Read it, learn, and write up what you discover along the way. This literature needs to grow.
arendtio 8y ago
2014 feels like decades ago in AI. Looks like that paper is already a classic.
rasmi 8y ago
If you're interested in this, be sure to read "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" [1] and the Rules of Machine Learning [2]. [1] https://ai.google/research/pubs/pub46555 https://ai.google/research/pubs/pub46555 [2] https://developers.google.com/machine-learning/rules-of-ml/ https://developers.google.com/machine-learning/rules-of-ml/
prabhatjha 8y ago
This is a great paper. It helped us think more rationally and helped us realize that we were not alone having difficulties running ML in production.
deleted 8y ago
[deleted]
tacostakohashi 8y ago
I recently came across this artice: https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting Although it describes the issue pertaining to statistics + machine learning, this is also exactly what end ups up happening with a large codebase without clear requirements or test cases, and people just making incremental, piecemeal changes over time. You end up with an application that has been trained (overfitted) with historical data and usecases, but breaks easily for slightly new variations that are different from anything that has ever been handled by the system before in some trivial way that better designed, cleaner, more abstract system would be able to deal with. Given how much poor coding practices resemble machine learning (albeit in slow motion), it's hard to hold too much hope about what happens when you automate the process.
- mlthoughts2018 8y ago
  The problem with the analogy is that for a learning algorithm, there are clear definitions of the model complexity as it relates directly to the outcome being optimized. YAGNI applied to a model is a penalty term for parameters or various methods of regularization. But when the “goal” of the system is just “arbitrary short term desires of management” you can easily point out the problems, but there is no agreement on what constraints you can use to trade-off against it. Especially for extensibility, where you can get carried away easily with making a system extensible for future changes, many of which turn out to be wasted effort because you did not end up needing that flexibility anyway, and everything changed after Q2 earnings were announced, etc. In those cases, it can actually be more effective engineering to “overfit” to just what the management wants right now, and just accept that you have to pay the pain of hacking extensibility in on a case by case basis. This definitely reduces wasted effort from a YAGNI point of view. The closest thing I could think of to the same idea of “regularizing” software complexity would be Netflix’s ChaosMonkey [0], which is basically like Dropout [1] but for deployed service networks instead of neural networks. Extending this idea to actual software would be quite cool. Something like the QuickCheck library for Haskell, but which somehow randomly samples extensibility needs and penalizes some notion of how hard the code would be to extend to that case. Not even sure how it would work... [0]: < https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey > [1]: < https://en.m.wikipedia.org/wiki/Dropout_(neural_networks) https://en.m.wikipedia.org/wiki/Dropout_(neural_networks) >
dj-wonk 8y ago
I enjoy the paper as a starting point for discussions. However, given the varying definitions of technical debt, I think it is important to see additional elaborations and examples of real-world trade offs in production systems. Perhaps the best overall wisdom this paper tries to impart is this: build awareness, culture, and tooling around your ML systems, both upstream and downstream. Never stop exploring and improving. Relentlessly try to slim down your models, simplify your pipelines, and bring people together to talk about all kinds of dependencies.
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
ericnyamu 8y ago
Check out this nice forum http://mawazo.byethost7.com/ http://mawazo.byethost7.com/
ericnyamu1 8y ago
wow.check out this cool forum http://mawazo.byethost7.com/index.php http://mawazo.byethost7.com/index.php
741258963 8y ago
this is a very nice forum,check it out https://www.bit.ly/2M0hqJa https://www.bit.ly/2M0hqJa
6456456456 8y ago
this is a very nice forum,check it out https://www.bit.ly/2M0hqJa https://www.bit.ly/2M0hqJa
53543543 8y ago
www.bit.ly/2K1FL3T
5435435 8y ago
http://bit.ly/2t9A76a http://bit.ly/2t9A76a
7657657 8y ago
this https://https://bit.ly/2t594ci https://https://bit.ly/2t594ci
88768768 8y ago
nice blog http://mawazo.byethost7.com http://mawazo.byethost7.com

576 points by amzans 8y ago

Publication Year : 2014 Previous HN Discussions https://news.ycombinator.com/item?id=10006293 https://news.ycombinator.com/item?id=10006293 https://news.ycombinator.com/item?id=10338575 https://news.ycombinator.com/item?id=10338575 https://news.ycombinator.com/item?id=8775772 https://news.ycombinator.com/item?id=8775772 "The Morning Paper" has a nice summary https://blog.acolyer.org/2016/02/29/machine-learning-the-high-interest-credit-card-of-technical-debt/ https://blog.acolyer.org/2016/02/29/machine-learning-the-hig...

glup 8y ago
which means it must be good, right? I feel like there are a few of these frequent flyers... is there anyway to figure out what they are?
- chucksmash 8y ago
  "Show HN: Hacker News Classics" might fit the bill. Discussion: https://news.ycombinator.com/item?id=16442888 https://news.ycombinator.com/item?id=16442888 App: http://jsomers.net/hn/ http://jsomers.net/hn/ Source: https://github.com/jsomers/hacker-classics https://github.com/jsomers/hacker-classics

israrkhan 8y ago

Software Engineering daily has a podcast, with D. Sculley, the author of this paper. It is quite interesting to listen https://www.softwaredaily.com/post/5913c0e74ee01db33cacd027/Machine-Learning-and-Technical-Debt-with-D-Sculley https://www.softwaredaily.com/post/5913c0e74ee01db33cacd027/...

thelastidiot 8y ago

Is this still relevant?

joker3 8y ago
It's more relevant every year.
glup 8y ago
nah, thankfully technical debt has been solved.
- setquk 8y ago
  Gnome solved it. Just burn the universe every five years!
  - paulie_a 8y ago
    That should honestly be a more frequently employed tactic in codebases.
    - favorited 8y ago
      I'm definitely a believer in this. I read Carmack refer to it as "grooming" at one point. Codebases where components and subsystems are periodically rewritten naturally end up being easier to change, which is an undervalued metric.
    - setquk 8y ago
      Definitely agree. However it mostly depends on your product being well isolated little chunks of functionality which is rarely the case. So what happens is you end up with a perpetually half baked Cthulhu.
  - bkor 8y ago
    2.0 came out around Sep 2002 (a bit less than 16 years ago). 3.0 came out in Apr 2011 (7 years ago). Most of the components were the same between the last 2.32 release and 3.0, except for the very visible gnome-panel to gnome-shell switch. Gnome isn't a good example.
    - 8y ago

imh 8y ago

I love this paper. The further I get in my ML/stats career, the more relevant these lessons are. I would recommend anyone interested in building long lived ML products to read this.

joker3 8y ago

This paper will eventually be seen as a landmark in the field of machine learning systems. Read it, learn, and write up what you discover along the way. This literature needs to grow.

arendtio 8y ago

2014 feels like decades ago in AI. Looks like that paper is already a classic.

rasmi 8y ago

If you're interested in this, be sure to read "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" [1] and the Rules of Machine Learning [2]. [1] https://ai.google/research/pubs/pub46555 https://ai.google/research/pubs/pub46555 [2] https://developers.google.com/machine-learning/rules-of-ml/ https://developers.google.com/machine-learning/rules-of-ml/

prabhatjha 8y ago

This is a great paper. It helped us think more rationally and helped us realize that we were not alone having difficulties running ML in production.

deleted 8y ago

[deleted]

tacostakohashi 8y ago

I recently came across this artice: https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting Although it describes the issue pertaining to statistics + machine learning, this is also exactly what end ups up happening with a large codebase without clear requirements or test cases, and people just making incremental, piecemeal changes over time. You end up with an application that has been trained (overfitted) with historical data and usecases, but breaks easily for slightly new variations that are different from anything that has ever been handled by the system before in some trivial way that better designed, cleaner, more abstract system would be able to deal with. Given how much poor coding practices resemble machine learning (albeit in slow motion), it's hard to hold too much hope about what happens when you automate the process.

mlthoughts2018 8y ago
The problem with the analogy is that for a learning algorithm, there are clear definitions of the model complexity as it relates directly to the outcome being optimized. YAGNI applied to a model is a penalty term for parameters or various methods of regularization. But when the “goal” of the system is just “arbitrary short term desires of management” you can easily point out the problems, but there is no agreement on what constraints you can use to trade-off against it. Especially for extensibility, where you can get carried away easily with making a system extensible for future changes, many of which turn out to be wasted effort because you did not end up needing that flexibility anyway, and everything changed after Q2 earnings were announced, etc. In those cases, it can actually be more effective engineering to “overfit” to just what the management wants right now, and just accept that you have to pay the pain of hacking extensibility in on a case by case basis. This definitely reduces wasted effort from a YAGNI point of view. The closest thing I could think of to the same idea of “regularizing” software complexity would be Netflix’s ChaosMonkey [0], which is basically like Dropout [1] but for deployed service networks instead of neural networks. Extending this idea to actual software would be quite cool. Something like the QuickCheck library for Haskell, but which somehow randomly samples extensibility needs and penalizes some notion of how hard the code would be to extend to that case. Not even sure how it would work... [0]: < https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey > [1]: < https://en.m.wikipedia.org/wiki/Dropout_(neural_networks) https://en.m.wikipedia.org/wiki/Dropout_(neural_networks) >

dj-wonk 8y ago

I enjoy the paper as a starting point for discussions. However, given the varying definitions of technical debt, I think it is important to see additional elaborations and examples of real-world trade offs in production systems. Perhaps the best overall wisdom this paper tries to impart is this: build awareness, culture, and tooling around your ML systems, both upstream and downstream. Never stop exploring and improving. Relentlessly try to slim down your models, simplify your pipelines, and bring people together to talk about all kinds of dependencies.

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

ericnyamu 8y ago

Check out this nice forum http://mawazo.byethost7.com/ http://mawazo.byethost7.com/

ericnyamu1 8y ago

wow.check out this cool forum http://mawazo.byethost7.com/index.php http://mawazo.byethost7.com/index.php

741258963 8y ago

this is a very nice forum,check it out https://www.bit.ly/2M0hqJa https://www.bit.ly/2M0hqJa

6456456456 8y ago

this is a very nice forum,check it out https://www.bit.ly/2M0hqJa https://www.bit.ly/2M0hqJa

53543543 8y ago

www.bit.ly/2K1FL3T

5435435 8y ago

http://bit.ly/2t9A76a http://bit.ly/2t9A76a

7657657 8y ago

this https://https://bit.ly/2t594ci https://https://bit.ly/2t594ci

88768768 8y ago

nice blog http://mawazo.byethost7.com http://mawazo.byethost7.com