4 ms·

I think you are referring to what's known as the "curse of dimensionality," where as dimensionality increases, the distance between points tends to become more

by ttul 2y ago

I think you are referring to what's known as the "curse of dimensionality," where as dimensionality increases, the distance between points tends to become more uniform and large. However, nearest neighbor search can still work effectively because of several key factors: 1. Real data rarely occupies the full high-dimensional space uniformly. Instead, it typically lies on or near a lower-dimensional manifold embedded within the high-dimensional space. This is often called the "manifold hypothesis." 2. While distances may be large in absolute terms, _relative_ distances still maintain meaningful relationships. If point A is closer to point B than to point C in this high-dimensional space, that proximity often still indicates semantic similarity. 3. The data points that matter for a given problem often cluster in meaningful ways. Even in high dimensions, these clusters can maintain separation that makes nearest neighbor search useful. Let me give a concrete example: Consider a dataset of images. While an image might be represented in a very high-dimensional space (e.g., thousands of pixels), images of dogs will tend to be closer to other dog images than to images of cars, even in this high-dimensional space. The meaningful features create a structure that nearest neighbor search can exploit. Spam filtering is another area where nearest neighbor is used to good effect. When you know that a certain embedding representing a spam message (in any medium - email, comments, whatever), then if other messages come along and are _relatively_ close to that one, you may conclude that they are on the right side of the manifold to be considered spam. You could train a special model to define this manifold, but spam changes all the time and constant training doesn’t work well.

osigurdson 2y ago
>> as the "curse of dimensionality," where as dimensionality increases, the distance between points tends to become more uniform and large Since embeddings are the middle layer of an ANN, doesn't this suggest that there are too many dimensions used during training. I would think a training goal would be to have relatively uniform coverage of the space
- ttul 2y ago
  The curse of dimensionality in neural network embeddings actually serves a valuable purpose, contrary to what might seem intuitive. While the tendency for distances to become more uniform and large in high-dimensional spaces might appear problematic, this characteristic helps maintain clear separation between different semantic concepts. Rather than aiming for uniform coverage of the embedding space, neural networks benefit from having distinct clusters with meaningful gaps between them – much like how a library benefits from having clear separation between different subject areas, rather than books being randomly distributed throughout the space. The high dimensionality of embeddings provides several key advantages during training and inference. The additional dimensions allow the network to better preserve both local and global relationships between data points, while providing the capacity to capture subtle semantic nuances. This extra capacity helps prevent information bottlenecks and provides more pathways for gradient descent during training, leading to more stable optimization. In essence, the high-dimensional nature of embeddings, despite its counterintuitive properties, is a feature rather than a bug in neural network architecture. TL;DR: High-dimensional embeddings in neural networks are actually beneficial - the extra dimensions help keep different concepts clearly separated and make training more stable, even though the distances between points become more uniform.

by ttul 2y ago

osigurdson 2y ago

>> as the "curse of dimensionality," where as dimensionality increases, the distance between points tends to become more uniform and large Since embeddings are the middle layer of an ANN, doesn't this suggest that there are too many dimensions used during training. I would think a training goal would be to have relatively uniform coverage of the space

ttul 2y ago
The curse of dimensionality in neural network embeddings actually serves a valuable purpose, contrary to what might seem intuitive. While the tendency for distances to become more uniform and large in high-dimensional spaces might appear problematic, this characteristic helps maintain clear separation between different semantic concepts. Rather than aiming for uniform coverage of the embedding space, neural networks benefit from having distinct clusters with meaningful gaps between them – much like how a library benefits from having clear separation between different subject areas, rather than books being randomly distributed throughout the space. The high dimensionality of embeddings provides several key advantages during training and inference. The additional dimensions allow the network to better preserve both local and global relationships between data points, while providing the capacity to capture subtle semantic nuances. This extra capacity helps prevent information bottlenecks and provides more pathways for gradient descent during training, leading to more stable optimization. In essence, the high-dimensional nature of embeddings, despite its counterintuitive properties, is a feature rather than a bug in neural network architecture. TL;DR: High-dimensional embeddings in neural networks are actually beneficial - the extra dimensions help keep different concepts clearly separated and make training more stable, even though the distances between points become more uniform.