9 ms·

I recently came across this artice: https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting Although it describes the issue pertai

I recently came across this artice: https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting Although it describes the issue pertaining to statistics + machine learning, this is also exactly what end ups up happening with a large codebase without clear requirements or test cases, and people just making incremental, piecemeal changes over time. You end up with an application that has been trained (overfitted) with historical data and usecases, but breaks easily for slightly new variations that are different from anything that has ever been handled by the system before in some trivial way that better designed, cleaner, more abstract system would be able to deal with. Given how much poor coding practices resemble machine learning (albeit in slow motion), it's hard to hold too much hope about what happens when you automate the process.

mlthoughts2018 8y ago
The problem with the analogy is that for a learning algorithm, there are clear definitions of the model complexity as it relates directly to the outcome being optimized. YAGNI applied to a model is a penalty term for parameters or various methods of regularization. But when the “goal” of the system is just “arbitrary short term desires of management” you can easily point out the problems, but there is no agreement on what constraints you can use to trade-off against it. Especially for extensibility, where you can get carried away easily with making a system extensible for future changes, many of which turn out to be wasted effort because you did not end up needing that flexibility anyway, and everything changed after Q2 earnings were announced, etc. In those cases, it can actually be more effective engineering to “overfit” to just what the management wants right now, and just accept that you have to pay the pain of hacking extensibility in on a case by case basis. This definitely reduces wasted effort from a YAGNI point of view. The closest thing I could think of to the same idea of “regularizing” software complexity would be Netflix’s ChaosMonkey [0], which is basically like Dropout [1] but for deployed service networks instead of neural networks. Extending this idea to actual software would be quite cool. Something like the QuickCheck library for Haskell, but which somehow randomly samples extensibility needs and penalizes some notion of how hard the code would be to extend to that case. Not even sure how it would work... [0]: < https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey > [1]: < https://en.m.wikipedia.org/wiki/Dropout_(neural_networks) https://en.m.wikipedia.org/wiki/Dropout_(neural_networks) >
gautamnarula 8y ago
I really like this extension of the concept of overfitting to codebases in general. I especially noticed this in libraries/packages that were "community owned" in a company--instead of one team owning the package and being the authority on deciding the long term roadmap and communicating with other teams about feature requests, deprecations, documentation, bug fixes, etc, the community at large, where "community" was very broadly defined as a team that for whatever reason had an interest in using/maintaining/adding onto the package, would collectively own the package. Naturally, the result was exactly the scenario you described. Each team hacked on their own bit of functionality for their specific purpose, while doing their best to not affect or break the increasingly precarious tightrope of backwards compatibility. There was no long term architectural vision, so there was a definite need for refactoring--and yet no team had the incentive to invest the amount of time needed to do that.The documentation was woefully incomplete as well, and few people understood how the entire thing worked since each team would only interact with their small fraction of the code.
- tomelders 8y ago
  Two principles I live by (much to the annoyance of my bosses) 1. Don't fear the refactor. 2. If you don't want to rebuild your entire application from scratch, don't worry, a competitor will do it for you. There's nothing wrong with creating something in increments. It's the fear of revisiting something that destroy's a code base.
  - nicodjimenez 8y ago
    "It's the fear of revisiting something that destroy's a code base." So true.
    - thomasmeeks 8y ago
      Yep, when fear creeps in around modifying a part of an application it is time to have a very serious conversation about fixing that. It is one of the few cases where I find the refactor vs. creating customer value argument is more clear cut -- letting that fear linger is likely to spread to other parts of the code & turns into a human problem pretty fast. Fixing might be a presentation, tests, documentation, refactoring, rewrite, deprecation, whatever. Just don't let it languish and the fear grow.
nkozyra 8y ago
Overfitting is a quantifiable problem. If you're not doing robust data segregation and CV you're not even engaging in elementary ML practices.
- juliend2 8y ago
  What do you mean by CV? I'm not familiar with those terms. Thank you.
  - lostcolony 8y ago
    Never seen the acronym (not really in the space) but I assume cross validation.
  - nkozyra 8y ago
    As sibling points out, cross validation, which is the front-line approach to avoiding overfitting for supervised classification problems.
  - DEADBEEFC0FFEE 8y ago
    I assumed Code Versioning so that if you have robust data segmentation you have less uncertainty about the impact of change. However, I'm a tourist here and hope OP comes back to share.
    - BenFrantzDale 8y ago
      I assumed Computer Vision.
    - tomrod 8y ago
      Cross-validation: testing model fit on non-training data
  - sbhn 8y ago
    Camouflaged Vacuity
  - thousandautumns 8y ago
logicallee 8y ago
Fantastic insight, really top-notch. Just some random thoughts in no particular order - curious what you make of them: - On the subject of incremental piecemeal changes over time with no requirements: don't you all find that in your workflows (when you're doing something for yourself), it is hard to step back and "architect" something? It is easier to just let it evolve. - Likewise it takes real work and thought to organize something as simple as a spice rack. (I just keep opened packages of spices in the cupboard.) The knowledge that company is coming is one of the few pushes. But it kind of feels like it's being done for show. - It's hard to add architecture when you know there's no team that is coding against it as an API. It's just you. It feels like that extra power is, kind of wasteful. - The other thing is that it may be the case that you know there is some deeper level of architecture. In the case of my spices, for example, most of the opened spice packets I mentioned are actually mixes. (Such as grilled chicken spice mix.) - If I had to architect my own spice rack, I should start by learning which spices I'm actually using more of. And since what I'm doing works, I don't actually care. Plus, it would be a step down: the first time I mixed my own spices, I would probably end up with a worse dish than pouring some out of a premixed packet. - The first time you architect a "proper" framework rather than let your machine learning algorithm "overfit", the result is probably demonstrably worse. - That is a lot of pressure on not architecturing, and just continuing to (over)-fit.
- jacob019 8y ago
  well-constructed != over-architected
- kthejoker2 8y ago
  This is where good logging helps. The lifehack is to throw all your spices in a box and only pull hthem out when you need them and then leave them on the rack. Then throw away any spice you haven't used in n months and add it to a blacklist. The ones you use frequently should be prominently displayed and texted with extra care and possibly set up for autorenewal from the grocery. Only introduce new spices when there's a recipe, and buy just the amount you need. So too with code. Log your code paths, prune little used features, optimize the hell out of the most frequently used ones, introduce features sparingly and with purpose... I like this spice metaphor, thabks for it.
jdwithit 8y ago
Agree with the other commenters, very interesting insight. It maybe doesn't fit the metaphor quite as well, but as an operations person, I've frequently run into the "underfitting" problem. For example, we run Chef to manage our physical and virtual infrastructure. There are a ton of community-authored Chef cookbooks available. Which at first blush, sounds great. But often, they have grown over time to become these awful hydras that try to be all things to all people. PR after PR has added support for the specific use case of every organization that wants to run the cookbook in their own special way. The "Getting Started" section of the README eventually becomes a dumping ground of 900 attributes you need to set correctly, and yet somehow it still doesn't quite perform how you'd like. In many cases, we've tried to use community cookbooks and even merge our own customizations back upstream. Only to eventually give up and write our own version that's 50 lines of Chef DSL/Ruby instead of 5,000 but does exactly what we need, the way we need, and no more. It's very possible to make a system too generic and configurable, to the point where it loses all meaning.
- mwaitjmp 8y ago
  Found the exact same thing regarding the community cookbooks. We do use some though, it depends on the complexity and how well they work. I've either written some from scratch, taking pointers from the community ones or forked them to make them simpler and better suit our needs. Pull requests have been made where it makes sense. Glad to hear we're not the only ones who found the community ones not perfect for every need.
- rusk 8y ago
  > There are a ton of community-authored Chef cookbooks available. Which at first blush, sounds great. Welcome to software development! Not as easy as it looks is it :) EDIT you may find these articles helpful (or at the very least food for thought): - https://blog.codinghorror.com/dependency-avoidance/ https://blog.codinghorror.com/dependency-avoidance/ - https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-invented-here-syndrome/ https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
matchagaucho 8y ago
Hopefully code reviewers with institutional knowledge can advise on where to apply pruning and prevent code overfitting. Pruning is also the common ML practice to prevent statistical overfitting.
hyperpallium 8y ago
Epicycles within epicycles eventually get replaced with a clean redesign (https://wikipedia.org/wiki/Paradigm_shift https://wikipedia.org/wiki/Paradigm_shift) The tricky bit is mostly that you need a new theory of the data to have a better abstraction. That's the tricky bit. Models generated by DL lack even a paradigm or theory or abstraction.
phonebucket 8y ago
"Given how much poor coding practices resemble machine learning (albeit in slow motion), it's hard to hold too much hope about what happens when you automate the process." Your whole argument seems to be based on your personal experiences. Perhaps it is also thus vulnerable to some sort of overfitting :)
tylll 8y ago
nice https://bit.ly/2JTdt8x https://bit.ly/2JTdt8x
randomsearch 8y ago
This is a brilliant insight. One of the problems I’ve seen in research into technical debt is the lack of a good definition. This insight could form the basis of one.

mlthoughts2018 8y ago

The problem with the analogy is that for a learning algorithm, there are clear definitions of the model complexity as it relates directly to the outcome being optimized. YAGNI applied to a model is a penalty term for parameters or various methods of regularization. But when the “goal” of the system is just “arbitrary short term desires of management” you can easily point out the problems, but there is no agreement on what constraints you can use to trade-off against it. Especially for extensibility, where you can get carried away easily with making a system extensible for future changes, many of which turn out to be wasted effort because you did not end up needing that flexibility anyway, and everything changed after Q2 earnings were announced, etc. In those cases, it can actually be more effective engineering to “overfit” to just what the management wants right now, and just accept that you have to pay the pain of hacking extensibility in on a case by case basis. This definitely reduces wasted effort from a YAGNI point of view. The closest thing I could think of to the same idea of “regularizing” software complexity would be Netflix’s ChaosMonkey [0], which is basically like Dropout [1] but for deployed service networks instead of neural networks. Extending this idea to actual software would be quite cool. Something like the QuickCheck library for Haskell, but which somehow randomly samples extensibility needs and penalizes some notion of how hard the code would be to extend to that case. Not even sure how it would work... [0]: < https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey > [1]: < https://en.m.wikipedia.org/wiki/Dropout_(neural_networks) https://en.m.wikipedia.org/wiki/Dropout_(neural_networks) >

gautamnarula 8y ago

I really like this extension of the concept of overfitting to codebases in general. I especially noticed this in libraries/packages that were "community owned" in a company--instead of one team owning the package and being the authority on deciding the long term roadmap and communicating with other teams about feature requests, deprecations, documentation, bug fixes, etc, the community at large, where "community" was very broadly defined as a team that for whatever reason had an interest in using/maintaining/adding onto the package, would collectively own the package. Naturally, the result was exactly the scenario you described. Each team hacked on their own bit of functionality for their specific purpose, while doing their best to not affect or break the increasingly precarious tightrope of backwards compatibility. There was no long term architectural vision, so there was a definite need for refactoring--and yet no team had the incentive to invest the amount of time needed to do that.The documentation was woefully incomplete as well, and few people understood how the entire thing worked since each team would only interact with their small fraction of the code.

tomelders 8y ago
Two principles I live by (much to the annoyance of my bosses) 1. Don't fear the refactor. 2. If you don't want to rebuild your entire application from scratch, don't worry, a competitor will do it for you. There's nothing wrong with creating something in increments. It's the fear of revisiting something that destroy's a code base.
- nicodjimenez 8y ago
  "It's the fear of revisiting something that destroy's a code base." So true.
  - thomasmeeks 8y ago
    Yep, when fear creeps in around modifying a part of an application it is time to have a very serious conversation about fixing that. It is one of the few cases where I find the refactor vs. creating customer value argument is more clear cut -- letting that fear linger is likely to spread to other parts of the code & turns into a human problem pretty fast. Fixing might be a presentation, tests, documentation, refactoring, rewrite, deprecation, whatever. Just don't let it languish and the fear grow.

nkozyra 8y ago

Overfitting is a quantifiable problem. If you're not doing robust data segregation and CV you're not even engaging in elementary ML practices.

juliend2 8y ago
What do you mean by CV? I'm not familiar with those terms. Thank you.
- lostcolony 8y ago
  Never seen the acronym (not really in the space) but I assume cross validation.
- nkozyra 8y ago
  As sibling points out, cross validation, which is the front-line approach to avoiding overfitting for supervised classification problems.
- DEADBEEFC0FFEE 8y ago
  I assumed Code Versioning so that if you have robust data segmentation you have less uncertainty about the impact of change. However, I'm a tourist here and hope OP comes back to share.
  - BenFrantzDale 8y ago
    I assumed Computer Vision.
  - tomrod 8y ago
    Cross-validation: testing model fit on non-training data
- sbhn 8y ago
  Camouflaged Vacuity
- thousandautumns 8y ago

logicallee 8y ago

Fantastic insight, really top-notch. Just some random thoughts in no particular order - curious what you make of them: - On the subject of incremental piecemeal changes over time with no requirements: don't you all find that in your workflows (when you're doing something for yourself), it is hard to step back and "architect" something? It is easier to just let it evolve. - Likewise it takes real work and thought to organize something as simple as a spice rack. (I just keep opened packages of spices in the cupboard.) The knowledge that company is coming is one of the few pushes. But it kind of feels like it's being done for show. - It's hard to add architecture when you know there's no team that is coding against it as an API. It's just you. It feels like that extra power is, kind of wasteful. - The other thing is that it may be the case that you know there is some deeper level of architecture. In the case of my spices, for example, most of the opened spice packets I mentioned are actually mixes. (Such as grilled chicken spice mix.) - If I had to architect my own spice rack, I should start by learning which spices I'm actually using more of. And since what I'm doing works, I don't actually care. Plus, it would be a step down: the first time I mixed my own spices, I would probably end up with a worse dish than pouring some out of a premixed packet. - The first time you architect a "proper" framework rather than let your machine learning algorithm "overfit", the result is probably demonstrably worse. - That is a lot of pressure on not architecturing, and just continuing to (over)-fit.

jacob019 8y ago
well-constructed != over-architected
kthejoker2 8y ago
This is where good logging helps. The lifehack is to throw all your spices in a box and only pull hthem out when you need them and then leave them on the rack. Then throw away any spice you haven't used in n months and add it to a blacklist. The ones you use frequently should be prominently displayed and texted with extra care and possibly set up for autorenewal from the grocery. Only introduce new spices when there's a recipe, and buy just the amount you need. So too with code. Log your code paths, prune little used features, optimize the hell out of the most frequently used ones, introduce features sparingly and with purpose... I like this spice metaphor, thabks for it.

jdwithit 8y ago

Agree with the other commenters, very interesting insight. It maybe doesn't fit the metaphor quite as well, but as an operations person, I've frequently run into the "underfitting" problem. For example, we run Chef to manage our physical and virtual infrastructure. There are a ton of community-authored Chef cookbooks available. Which at first blush, sounds great. But often, they have grown over time to become these awful hydras that try to be all things to all people. PR after PR has added support for the specific use case of every organization that wants to run the cookbook in their own special way. The "Getting Started" section of the README eventually becomes a dumping ground of 900 attributes you need to set correctly, and yet somehow it still doesn't quite perform how you'd like. In many cases, we've tried to use community cookbooks and even merge our own customizations back upstream. Only to eventually give up and write our own version that's 50 lines of Chef DSL/Ruby instead of 5,000 but does exactly what we need, the way we need, and no more. It's very possible to make a system too generic and configurable, to the point where it loses all meaning.

mwaitjmp 8y ago
Found the exact same thing regarding the community cookbooks. We do use some though, it depends on the complexity and how well they work. I've either written some from scratch, taking pointers from the community ones or forked them to make them simpler and better suit our needs. Pull requests have been made where it makes sense. Glad to hear we're not the only ones who found the community ones not perfect for every need.
rusk 8y ago
> There are a ton of community-authored Chef cookbooks available. Which at first blush, sounds great. Welcome to software development! Not as easy as it looks is it :) EDIT you may find these articles helpful (or at the very least food for thought): - https://blog.codinghorror.com/dependency-avoidance/ https://blog.codinghorror.com/dependency-avoidance/ - https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-invented-here-syndrome/ https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...

matchagaucho 8y ago

Hopefully code reviewers with institutional knowledge can advise on where to apply pruning and prevent code overfitting. Pruning is also the common ML practice to prevent statistical overfitting.

hyperpallium 8y ago

Epicycles within epicycles eventually get replaced with a clean redesign (https://wikipedia.org/wiki/Paradigm_shift https://wikipedia.org/wiki/Paradigm_shift) The tricky bit is mostly that you need a new theory of the data to have a better abstraction. That's the tricky bit. Models generated by DL lack even a paradigm or theory or abstraction.

phonebucket 8y ago

"Given how much poor coding practices resemble machine learning (albeit in slow motion), it's hard to hold too much hope about what happens when you automate the process." Your whole argument seems to be based on your personal experiences. Perhaps it is also thus vulnerable to some sort of overfitting :)

tylll 8y ago

nice https://bit.ly/2JTdt8x https://bit.ly/2JTdt8x

randomsearch 8y ago

This is a brilliant insight. One of the problems I’ve seen in research into technical debt is the lack of a good definition. This insight could form the basis of one.