5 ms·
Derivatives, Gradients, Jacobians and Hessians
- leopoldj 1y agoThank you so much for posting. I finally understand Jacobian matrix. The key is to know this applies to a function that returns multiple values. The wikipedia article was difficult to understand, until now! Note: Technically, a function can map an input to a single output only. Here, when we say a function returns multiple values, we mean a single set of multiple values. For example, a function that outputs the heating and cooling cost of a building. Where as, a circle is not a function because it outputs two y values for a single x.
- whatever1 1y agoI can look around me and find the minimum of anything without tracing its surface and following the gradient. I can also identify immediately global minima instead of local ones. We all can do it in 2-3D. But our algorithms don’t do it. Even in 2D. Sure if I was blindfolded, feeling the surface and looking for minimization direction would be the way to go. But when I see, I don’t have to. What are we missing?
- Chinjut 1y agoYou're thinking of situations where you are able to see a whole object at once. If you were dealing with an object too large to see all of, you'd have to start making decisions about how to explore it.
- 3eb7988a1663 1y agoThe mental image I like: imagine you are lost in a hilly region with incredibly dense fog such that you can only see one foot directly in front of you. How do you find the base of the valley? Gradient descent: take a step in the steepest downward direction. Look around and repeat. When you reach a level area, how do you know you are at the lowest point?
- adrianN 1y agoThe inputs you can process visually are of trivial size even for naive algorithms, and probably also simple instances. I certainly can’t find global minima in 2d for any even slightly adversarial function.
- ks2048 1y agoWhen you look at a 2D surface, you directly observe all the values on that surface. For a loss-function, the value at each point must be computed. You can compute them all and "look at" the surface and just directly choose the lowest - that is called a grid search. For high dimensions, there's just way too many "points" to compute.
- cubefox 1y agoWhat about sampling at low resolution? If the hills and valleys aren't too close together, this should give a good indication of where the global minimum is.
- xigoi 1y ago> If the hills and valleys aren't too close together That’s a big “if”.
- samsartor 1y agoAnd remember, optimization problems can be _incredibly_ high-dimensional. A 7B parameter LLM is a 7-billion-dimensional optimization landscape. A grid-search with a resolution of 10 (ie 10 samples for each dimension) would requre evaluating the loss function 10^(7*10^9) times. That is, the number of evaluations is a number with 7B digits.
- pestatije 1y agotouch and sight sense essentially the same...the difference is in the magnitudes involved
- fancyfredbot 1y agoYour visual cortex is a massively parallel processor.
- hackinthebochs 1y agoYou're ignoring all the calculations that go on unconsciously that realize your conscious experience of "immediately" apprehending the global minima.
- i_am_proteus 1y agoWithout looking up the answer (because someone has already computed this for you), how would you find the highest geographic point (highest elevation) in your country?
- cinntaile 1y agoWhat if you're trying to find the minimum of something that you can't see? Or what if the differences are so small that you can't perceive them with your eyes even though you can see?
- jpeloquin 1y agoEvaluating a function using a densely spaced grid and plotting it does work. This is brute-force search. You will see the global minima immediately in the way you describe, provided your grid is dense enough to capture all local variation. It's just that when the function is implemented on the computer, evaluating so many points takes a long time, and using a more sophisticated optimization algorithm that exploits information like the gradient is almost always faster. In physical reality all the points already exist, so if they can be observed cheaply the brute force approach works well. Edit: Your question was good. Asking superficially-naive questions like that is often a fruitful starting point for coming up with new tricks to solve seemingly-intractable problems.
- whatever1 1y agoThanks! It does feels to me that we do some sort of sampling, definitely is not a naive grid search. Also I find it easier to find the minima in specific directions (up, down, left, right) rather than let’s say a 42 degree one. So some sort of priors are probably used to improve sample efficiency.
- GuB-42 1y agoYour eyes compute gradients, as part of the shitton of visual processing your brain does to get an estimate of where the local and global minima are. It is not perfect though, see the many optical illusions. But we follow gradients all the time, consciously or not. You know you are at the bottom of the hole when all the paths go up for instance.
- dcanelhas 1y agoIt has been suggested [citation needed] that the optical illusions of movement caused by gradients are there to to compensate for the time it takes to process the visual input. This should let you have an understanding of what is going on in the world around you right now, based on what happened on your retinas a few milliseconds ago. it's not a bug - it's a feature :D
- nwallin 1y agoWhen you look at, for instance, a bowl, or even one of those egg carton mattress things, and you want to find the global minimum, you are looking at a surface which is 2 dimensions in and 1 dimension out. It's easy enough for your brain to process several thousand points and say ok the bottom of the bowl is right here. When a computer has a surface which is 2 dimensions in and 1 dimension out, you can actually just do the same thing. Check like 100 values in the x/y directions and you only have to check like 10000 values. A computer can do that easy peasy. When a computer does ML with a deep neural network, you don't have 2 dimensions in and 1 dimension out. You have thousands to millions of dimensions in and thousands to millions of dimensions out. If you have 100000 inputs, and you check 1000 values for each input, the total number of combinations is 1000^100000. Then remember that you also have 100000 outputs. You ain't doin' that much math. You ain't. So we need fancy stuff like Jacobians and backtracking.
- whatever1 1y agoI don’t think it’s that simple. For the egg carton your eye will not spend almost any time looking at its top. You will spend most of the time sampling the bottom. I don’t know what we do, but it does not feel like a naive grid search.
- cvoss 1y agoI really don't think you have the ability to use self-reflection to discern an algorithm that occurs in your unconscious visual cortex in a split second. You wouldn't feel like you were doing a naive grid search even if a naive grid search is exactly what you were doing. You have suggested that the process in your mind to find a global minimum is immediate, apparently to contrast this with a standard computational algorithm. But such comparison fails. I don't know whether you mean "with few computational steps" or "in very little time"; the former is not knowable to you; the latter is not relevant since the hardware is not the same.
- zoogeny 1y agoPeople here are giving you mathematical answers which is what you are asking for, but I want to challenge your intuition here. In construction, grading a site for building is a whole process involving surveying. If you dropped a person on a random patch of earth that hasn't previously been levelled and gave them no tools, it would be a significant challenge for that person to level the ground correctly. What I'm saying is, your intuition that "I can look around me and find the minimum of anything" is almost certainly wrong, unless you have a superpower that no other person has.
- whatever1 1y agoThat is true we are only good at doing it for specific directions of the objective function. The one that we perceive as the minimizing direction. If you tell me find the minimum with a direction of 53 degrees likely I will fail, because I can’t easily visualize where this direction points towards
- raffael_de 1y agowell, first of all ... you can't. and it is very easy to come up with all sorts of (not even special) cases where you simply couldn't for literally obvious reasons. what you are imagining is some sort of stereoscopic ray tracing. that is anyway much more compute intensive then calculating a derivative.
- shoo 1y agoMany practical optimisation problems are less like "let's go hiking and climb a literal hill which we can see in front of us" and more like "find the best design in this space of possible designs that maximises some objective" Here are some alternative example problems, that are a lot more high dimensional, and also where the dimensions are not spatial dimensions so your eyes give you absolutely no benefit. (a) Your objective is to find a recipe that produces a maximally tasty meal, using the ingredients you have in your kitchen cupboard. To sample one point in recipe-space, you need to (1) devise a recipe, (2) prep and cook a candidate meal following the recipe, and (3) evaluate the candidate recipe, say by serving it to a bunch of your friends and family. That gets you one sample point. Maybe there are 1 trillion possible "recipes" you could make. Are you going to brute-force cook and serve them all to find a meal that maximises tastiness, or is there a more efficient way that requires fewer plan recipe->prep&cook->serve->evaluate cycles? (b) Your objective is to find the most efficient design of a bridge, that can support the required load and stresses, while minimising the construction cost.
- sestep 1y agoA bit more advanced than this post, but for calculating Jacobians and Hessians, the Julia folks have done some cool work recently building on classical automatic differentiation research: https://iclr-blogposts.github.io/2025/blog/sparse-autodiff/ https://iclr-blogposts.github.io/2025/blog/sparse-autodiff/
- flerovium114 1y agoHave you tried using Enzyme (https://enzyme.mit.edu/ https://enzyme.mit.edu/)? It operates on the LLVM IR, so it's available in any language that breaks down into LLVM (e.g., Julia, where I've used it for surface gradients) and it produces highly optimized AD code. Pretty cool stuff.
- sestep 1y agoYeah I've used it (cool project indeed!), albeit mostly just in a project I and others in the autodiff community maintain which benchmarks many different autodiff tools against each other: https://github.com/gradbench/gradbench https://github.com/gradbench/gradbench
- ziofill 1y agoMmh, this is a bit sloppy. The derivative of a function f::a -> b is a function Df::a -> a -o b where the second funny arrow indicates a linear function. I.e. the derivative Df takes a point in the domain and returns a linear approximation of f (the jacobian) at that point. And it’s always the jacobian, it’s just that when f is R -> R we conflate the jacobian (a 1x1 matrix in this case) with the number inside of it.
- ndriscoll 1y agoA perhaps nicer way to look at things[0] is to hold onto your base points explicitly and say Df:: a -> (b, a -o b) = (f(p),A(p)) where f(p+v)≈f(p)+A(p)v. Then you retain the information you need to define composition Dg∘Df=D(g∘f)=(Dg._1∘Df._1, Dg(Df._1)_.2∘Df._2). i.e the chain rule. [0] which I learned from this talk https://youtube.com/watch?v=17gfCTnw6uE https://youtube.com/watch?v=17gfCTnw6uE
- ziofill 1y agoYes! I love Conal Eliot’s work. The one you wrote is the compositional derivative which augments the regular derivative by also returning the function itself (otherwise composition won’t work well). For anyone interested look up “the simple essence of automatic differentiation”.
- dbacar 1y agoI respect the time you spent to write such a post with all those limited input alternatives (bowes).
- ndriscoll 1y agoYou can do ≈ by long holding = on Android/Gboard. The only way I know to get ∘ is to copy/paste it from a Unicode reference. Likewise with ⊸, which I was too lazy to look up and didn't know the name of, but now I know is MULTIMAP (U+22B8).
- tomsmeding 1y ago
- flufluflufluffy 1y agoFantastic post! As short as it needs to be while still communicating its points effectively. I love walking up the generalization levels in math.
- throwpr 1y ago[dead]
- amelius 1y ago> (...) The derivative of w with respect to x. Another way of saying that is “If you added 1 to x before plugging it into the function, this is how much w would change Incorrect!
- dang 1y agoOk, but a good HN post should explain what is correct, so those who don't know can learn.
- wiosnaintel 1y ago> (...) if the function was a straight line There are a bit more to go from local linearization to a complete view of a derivative but it's not exactly incorrect.
- amelius 1y agoLooks like the author fixed it. This was the original page: https://web.archive.org/web/20250817152111/https://blog.demofox.org/2025/08/16/derivatives-gradients-jacobians-and-hessians-oh-my/ https://web.archive.org/web/20250817152111/https://blog.demo...
- divbzero 1y agoWould love to see div and curl added to this post.
- nickpsecurity 1y ago"What I just described is an iterative optimization method that is similar to gradient descent. Gradient descent simulates a ball rolling down hill to find the lowest point that we can, adjusting step size, and even adding momentum to try and not get stuck in places that are not the true minimum." That is so much easier to understand than most descriptions. The whole opening was.
- GistNoesis 1y agoThe way that really made me understand gradients and derivative was when visualizing them as Arrow Maps. I even made a small tool https://github.com/GistNoesis/VisualizeGradient https://github.com/GistNoesis/VisualizeGradient . This visualization helps understand optimization algorithm. Jacobians can be understood as a collection of gradients when considering each coordinates of the output independently. My mental picture for Hessian is to associate each point with the shape of a parabola (or saddle), which best match the function locally. It's easy to visualize once you realize it's the shape of what you see when you zoom-in on the point. (Technically this mental picture is more of a hessian + gradient tangent plane simultaneously multivariate Taylor expansion but I find them hard to mentally separate the slope from the curvature).
- MathMonkeyMan 1y agoThe "eigenchris" Youtube channel teaches tensor algebra, differential calculus, general relativity, and some other topics. When I started thinking of vector calculus in terms of multiplying both vector components and the corresponding basis vectors, there was a nice unification of ordinary vector operations, jacobians, and the metric tensor.
- GistNoesis 1y agoSome times it's useful to see the elemental transformations, but often striving for a higher level view makes understanding easier. It's particularly true when you try to apply it to physics. Often where you are introduced to vector calculus for the first time, things like Maxwell's equation, or fluid mechanics. In physics there are often additional constraints, like conserved quantities. Like you calculate a scalar the total energy of a system, call it the Hamiltonian, auto-differentiate it with respect of the degrees of freedom, and you got very complex vector equations. But taking a step back you realise it's just a field of "objects" of a certain type and you are just locally comparing these "objects" to their neighbors. And the whole mess of vector calculus is reduced to in which direction, you rotate, stretch and project these objects. (As an example you can imagine a field of balls in various orientation, whose energy is defined by the difference of orientation between neighboring balls) When you wrap around that the whole point of vector calculus (and why it was invented) is to describe a state as an integral (a sum of successive) of linear infinitesimal transformations, this make more sense. These "objects" being constrained and continuous, by moving in infinitesimal steps along the tangent space (and then reproject to satisfy the constraints (or exponentiating in a Clifford algebra to stay in the space) ). All the infinitesimal transformations are "linear", and linear transformations are rotating, stretching, mirroring, projecting to various degrees.
- fouronnes3 1y agoThere's something that's always been deeply confusing to me about comparing the Jacobian and the Hessian because their nature is very different. The Hessian shouldn't have been called a matrix. The Jacobian describes all the first order derivatives of a vector valued function (of multiple inputs), while the Hessian is all the second order derivatives of a scalar valued output function (of multiple inputs). Why doesn't the number of dimensions of the array increase by one as the derivation order increases? It does! The object that fully describes second order derivation of a vector valued function of multiple inputs is actually a 3 dimensionnal tensor. One dimension for the original vector valued output, and one for each derivation order. Mathematicians are afraid of tensors of more than 2 dimensions for some reason and want everything to be a matrix. In other words, given a function R^n -> R^m: Order 0: Output value: 1d array of shape (m) (a vector) Order 1: First order derivative: 2d array of shape (m, n) (Jacobian matrix) Order 2: Second order derivative: 3d array of shape (m, n, n) (array of Hessian matrices) It all makes sense! Talking about "Jacobian and Hessian" matrices as if they are both naturally matrices is highly misleading.
- xigoi 1y agoI’ve been introduced to the Hessian in the context of finding the extrema of functions with multiple variables, where it does not make sense to consider arbitrary output dimensions (what is the minimum of a vector function?). In this context, it is also important to find the definiteness of the underlying quadratic form, which is easier if you treat it as a matrix se you can apply Sylvester’s rule.
- brantmv 1y agoMathematicians are afraid of higher order tensors because they are unruly monsters. There's a whole workshop of useful matrix tools. Decompositions, spectral theory, etc. These tools really break down when you generalize them to k-tensors. Even basic concepts like rank become sticky. (Iirc, the set of 3-tensors of tensor rank ≤k is not even topologically closed in general. Terrifying.) If you hand me some random 5-tensor, it's quite difficult to begin to understand it without somehow turning it into a matrix first by flattening or slicing or whatever. Don't get me wrong. People work with these things. They do their best. But in general, mathematicians are afraid of higher order tensors. You should be too.
- tired_and_awake 1y agoAbout a decade ago I was interviewed for Apple's self driving car project and an exec on the project asked me to define these exact 4 things in great detail and provide examples. Shrugs.
- vismit2000 1y agoThis is a fantastic video on Jacobian [Mathemaniac]: https://www.youtube.com/watch?v=wCZ1VEmVjVo https://www.youtube.com/watch?v=wCZ1VEmVjVo