Agreed. Someone might actually implement linear regression in the way described! That would be embarrassing. Another thing worth mentioning is the behavior o

Agreed. Someone might actually implement linear regression in the way described! That would be embarrassing. Another thing worth mentioning is the behavior of gradient descent for problems where the Hessian is not well-conditioned (i.e., highly eccentric elliptical contours in the cost function). This would be well-illustrated by a simple example - I think a case where the y = a x + b line had a very large a would do the trick, because then a and b would be tightly coupled (choose the wrong "a" and the intercept would change a lot, due to the large slope). The number of iterations will skyrocket, which is why gradient descent is avoided if possible in favor of conjugate gradient or related approaches.