3 ms·

"There are two algorithms." In the case of machine learning techniques that employ a learning algorithm (e.g. SGD) and a separate classification algorithm (e.g

"There are two algorithms." In the case of machine learning techniques that employ a learning algorithm (e.g. SGD) and a separate classification algorithm (e.g. forward prop), the situation is no different. Gradient descent in itself doesn't take "unbiased" data and pollute it with "bias" -- neither does, forward propagation. These algorithms are merely functions that are _parametrized_ by a potentially biased learned model (which really entirely amounts to having biased data). I recommend taking a look at Stanford's Intro to Machine Learning CS 229 course notes for a brief overview of these concepts. I can't speak anymore highly of the material in this course; it's by far the best out there, and it's how I learned ML. "The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased." Right, the "final model" (which comes directly from the data) will be biased, but not the ALGORITHM (the "math," as you call it). For example, I can't make an unbiased data set "biased" by using linear regression instead of logistic regression. It would be quite a tremendous discovery if you somehow figured out a way SGD in itself was biased, but the burden of proof on such an extraordinary claim would be on you. So like I said in my parent post, I can see how the data can be biased, but I'm struggling to find any biases in the algorithms themselves (assuming they are properly implemented and not an Underhanded C Contest [1] entry). Granted, if we're not actually using ML, and we're using something like a hardcoded decision tree or FSM instead (it was pretty strongly implied that we weren't though because we were talking about data-driven algorithms), then sure, that could very easily be a biased algorithm, but I've maintained that all along. [1] http://www.underhanded-c.org/ http://www.underhanded-c.org/

pdkl95 9y ago
> Gradient descent in itself doesn't take "unbiased" data and pollute it with "bias" That's correct, but the decision to use gradient descent at all is a huge bias. The mean, median, and mode all describe a dataset's central tendency, but their values can be very different.
- Hydraulix989 9y ago
  I fully understand what you are trying to say, but I don't think the analogy given in the second sentence is enough. Let's assume that the hypothetical well-intentioned but biased programmer developing our sentencing algorithm is competent enough to only use a basic Gaussian model like you are suggesting if it REALLY fits the data well (yes, a model of central tendency is about as rudimentary as it gets, whereas SGD is used in most state-of-the-art ML research -- the stuff that is popularly known as "deep learning"). Here's a thought experiment for you then: Can you give an example of how one's decision to use gradient descent might impact a data-driven sentencing algorithm? Or just in general? In contemporary machine learning, it is true that choosing "hyperparameters" is still pretty intuition-guided and based on trial-and-error (even "guessing" if you will). I wouldn't think those choices could be made maliciously, or even in a "biased" way (okay, _maybe_ I could choose comically out-of-range, nonsensical values -- but that's really a stretch). If anything, in a hypothetical world, I would feel comfortable letting the accused choose the provably-convergent machine learning algorithm and hyperparameter values of their wishes, and then that would be the sentencing model that is used to determine their fate. As you can see, I claim that their choices wouldn't be able to "bias" anything at all. As has been true all along, I can certainly see how the _dataset_ could be biased, but I claim that the choice of hyperparameters (and even the algorithm itself) is quite a bit more "immune" to human biases. For one thing, we aren't even capable of fully understanding the learned models produced by state-of-the-art ML algorithms; they are effectively "black boxes." Despite all of the downvotes, I am genuinely interested in continuing this discussion, and I am curious what you have to say so I will continue to spend more of my karma expressing and arguing for my (unpopular) views.

pdkl95 9y ago

> Gradient descent in itself doesn't take "unbiased" data and pollute it with "bias" That's correct, but the decision to use gradient descent at all is a huge bias. The mean, median, and mode all describe a dataset's central tendency, but their values can be very different.

Hydraulix989 9y ago
I fully understand what you are trying to say, but I don't think the analogy given in the second sentence is enough. Let's assume that the hypothetical well-intentioned but biased programmer developing our sentencing algorithm is competent enough to only use a basic Gaussian model like you are suggesting if it REALLY fits the data well (yes, a model of central tendency is about as rudimentary as it gets, whereas SGD is used in most state-of-the-art ML research -- the stuff that is popularly known as "deep learning"). Here's a thought experiment for you then: Can you give an example of how one's decision to use gradient descent might impact a data-driven sentencing algorithm? Or just in general? In contemporary machine learning, it is true that choosing "hyperparameters" is still pretty intuition-guided and based on trial-and-error (even "guessing" if you will). I wouldn't think those choices could be made maliciously, or even in a "biased" way (okay, _maybe_ I could choose comically out-of-range, nonsensical values -- but that's really a stretch). If anything, in a hypothetical world, I would feel comfortable letting the accused choose the provably-convergent machine learning algorithm and hyperparameter values of their wishes, and then that would be the sentencing model that is used to determine their fate. As you can see, I claim that their choices wouldn't be able to "bias" anything at all. As has been true all along, I can certainly see how the _dataset_ could be biased, but I claim that the choice of hyperparameters (and even the algorithm itself) is quite a bit more "immune" to human biases. For one thing, we aren't even capable of fully understanding the learned models produced by state-of-the-art ML algorithms; they are effectively "black boxes." Despite all of the downvotes, I am genuinely interested in continuing this discussion, and I am curious what you have to say so I will continue to spend more of my karma expressing and arguing for my (unpopular) views.