38 ms·
Look long enough at literature on any machine learning task, and someone invariably gets the idea to turn the data into an image and do machine learning on that
by wongarsu 1y ago
Look long enough at literature on any machine learning task, and someone invariably gets the idea to turn the data into an image and do machine learning on that. Sometimes it works out (turning binaries into images and doing malware detection with a CNN surprisingly works), usually it doesn't. Just like in this example the images usually end up as a kludge to fix some deficiency in the prevalent input encoding.
I can certainly believe that images bring certain advantages over text for LLMs: the image representation does contain useful information that we as humans use (like better information hierarchies encoded in text size, boldness, color, saturation and position, not just n levels of markdown headings), letter shapes are already optimized for this kind of encoding, and continuous tokens seem to bring some advantages over discrete ones. But none of these advantages need the roundtrip via images, they merely point to how crude the state of the art of text tokenization is
- deleted 1y ago[deleted]
- pegasus 1y agoExactly. The example the article give of reducing resolution as a form of compression highlights the limitations of the visual-only proposal. Blurring text is a poor form of compression, preserving at most information about paragraph sizes. Summarizing early paragraphs (as context compression does in coding agents) would be much more efficient.
- vanderZwan 1y agoDoesn't this more or less boil down to OCR scans of books having more privileged information than a plaintext file? In which case a roundtrip won't add anything? [0] https://web.archive.org/web/20140402025221/http://m.nautil.us/issue/6/secret-codes/teaching-me-softly https://web.archive.org/web/20140402025221/http://m.nautil.u...
- adzm 1y agoA great example of this is changing music into an image and using that to train and generate new images that get converted back into music. It was surprisingly successful. I think this approach is still used by the current music generators.
- bjourne 1y agoYou are talking about piano roll notation, I think. While it's 2d data, it's not quite the same as actual image data. E.g., 2d conv and pooling operations are useless for music. The patterns and dependencies are too subtle to be captured by spatial filters.
- yberreby 1y agoI've seen this approach applied to spectrograms. Convolutions do make enough sense there.
- adzm 1y agoI am talking about using spectrograms (Fourier transform into frequency domain then plotted over time) that results in a 2d image of the song, which is then used to train something like stable diffusion (and actually using stable diffusion by some) to be able to generate these, which is then converted back into audio. Riffusion used this approach.
- Mistletoe 1y agoHow does Spotify “think” about songs when it is using its algos to find stuff I like?
- lbourdages 1y agoDoes it really need to think about the song contents? It can just cluster you with other people that listen to similar music and then propose music they listen to that you haven't heard.
- vanderZwan 1y agoThat's one method they use, but "just cluster" is doing a lot of heavy lifting in that sentence. It's why Erik Bernhardsson came up with the Approximate Nearest Neighbors Oh Yeah algorithm (or ANNOY for short) > We use it at Spotify for music recommendations. After running matrix factorization algorithms, every user/item can be represented as a vector in f-dimensional space. This library helps us search for similar users/items. We have many millions of tracks in a high-dimensional space, so memory usage is a prime concern. [0] https://erikbern.com/2013/04/12/annoy.html https://erikbern.com/2013/04/12/annoy.html [1] https://github.com/spotify/annoy?tab=readme-ov-file https://github.com/spotify/annoy?tab=readme-ov-file
- arbot360 1y agoAnother great example of this working is the genomic variant calling models from Deepmind "DeepVariant". They use the "alignment pile-up" images which are also used by humans to debug genomic alignments, with some additional channels to further feature engineer the CNN.
- rangestransform 1y agoThis reminds me of how trajectory prediction networks for autonomous driving used to use a CNN to encode scene context (from map and object detection rasters) until vectornet showed up