9 ms·
Bytes are all you need: Transformers operating directly on file bytes
- RecycledEle 3y agoI wonder what would happen if someone connected a Transformer between the inputs and output of an embedded system. Could we get a robotic arm to catch a falling ball? I suspect the lack of awareness of time would mess it up. What if we had the Transformer take inputs from the state of the world (as determined by other software) and output commands? I wonder what it could do.
- ZeroCool2u 3y agoI think PALM-E[1] is pretty close to what you're describing. [1]: https://arstechnica.com/information-technology/2023/03/embodied-ai-googles-palm-e-allows-robot-control-with-natural-commands/ https://arstechnica.com/information-technology/2023/03/embod...
- xp84 3y ago> In a video example, a researcher grabs the chips from the robot and moves them, but the robot locates the chips and grabs them again. Well, that’s it boys, I think we’ve successfully created a Terminator. To paraphrase Kyle Reese, “It is a chip-grabbing machine, and it absolutely will not stop!! Until it has acquired all the chips.”
- deleted 3y ago[deleted]
- moffkalast 3y agoReplace chips with money and bam, it's man.
- bluefishinit 3y agoInteresting thought to have a model training co-processor that reads all of the data inputs and outputs from the actual processor. There's a ton of sequence information flowing through there even on a single machine. Then you'd basically have a model that was a "virtual machine" mirror of your actual cpu and the data it's interacted with. I'm not sure what would emerge from that, but it's super interesting.
- vczf 3y agoCould be used to "compress"/predict common computations, a la Nvidia's AI video compression. Distributed computing that's distributed across time as well as space.
- bigyikes 3y agoWouldn’t be surprised if this was already done to some degrees for branch prediction
- yyyk 3y agoAlready the case for more then a decade. https://chasethedevil.github.io/post/the_neural_network_in_your_cpu/ https://chasethedevil.github.io/post/the_neural_network_in_y...
- tehsauce 3y agoSeems in similar spirit to the “perceiver” architecture from deepmind a couple years ago: https://arxiv.org/abs/2107.14795 https://arxiv.org/abs/2107.14795
- doctoboggan 3y agoBoth tif and wav are lossless so this doesn’t really surprise me. It is however nice to see Apple researchers publishing, you don’t see that often at the forefront of transformer or generative model research. I hope it means that everyone here on HN is right and they are seriously looking into running the generative models locally on their own silicon
- brucethemoose2 3y agoOh, and also, lossless does not necessarily mean readable. Hence they note zlib compressed PNGs are untenable.
- brucethemoose2 3y agoI mean, they published their own metal stable diffusion implementation on GitHub very quickly last year. I dont think this was in doubt
- sroussey 3y agoAnd then… nothing after?
- haswell 3y agoI suspect they prefer to use their standard marketing avenues to better control the narrative. I'm sure we'll be hearing a lot more next week, and in Apple fashion it'll all be grounded in customer facing use cases and associated developer frameworks.
- refulgentis 3y agoThat's not AI research
- brucethemoose2 3y agoIts desperately needed though. If AMD or Intel started going around github and writing in acceleration for random popular ML repos (and not just one off abandoned demos), that would change so much.
- brucethemoose2 3y agoAs they note, most media compression is going to throw a monkeywrench into the whole thing. But I was kinda hoping they would test gpu texture compression. AFAIK its a much simpler compression scheme.
- Taek 3y agoWhy not enforce that decompression is a necessary part of the data cleaning? There's no reason to operate on complex formats like mp4 directly.
- valine 3y agoIs data compression really a problem? Compression algorithms are meticulously tuned to preserve important details. In theory, a model that operates on compressed data has a real advantage over one that operates on raw data. You’ve reduced the memory requirements without sacrificing any important features of the input data.
- wizzwizz4 3y ago> Additionally, we demonstrate that ByteFormer has applications in privacy-preserving inference. ByteFormer is capable of performing inference on particular obfuscated input representations with no loss of accuracy. We also demonstrate ByteFormer's ability to perform inference with a hypothetical privacy-preserving camera which avoids forming full images by consistently masking $90\%$ of pixel channels, while still achieving $71.35\%$ accuracy on ImageNet. I'm not certain they know what "privacy-preserving" means. All the claims they've made around privacy look, to the lay-person (me), to be meaningless: • Permuting the input values doesn't change anything, because. If anything, this suggests that transformers might be able to approximate the original image, given an unknownly-permuted image – but so can humans, so that's nothing new. https://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#ECB-weakness https://en.wikipedia.org/wiki/Block_cipher_modes_of_operatio... • Their "partially-masked image" just looks like a noisy image, not a redacted one. Basic information theory suggests it's not really privacy-preserving at all. Is it normal for AI papers to be so hypey? Like, this part is literally security-by-obscurity. > As our method can handle highly nonlinear JPEG encodings, we expect it to perform well on a variety of alternative encodings that an outside observer might not be able to easily guess. I don't see how any of section 4.2 contributes to the paper, other than letting them make a bold claim about a buzzword in a disproportionate amount of the abstract and conclusion.
- l33t233372 3y agoThey say their model works for a hypothetical privacy preserving camera that masks 90% of the pixels. I’m not sure there’s much more to it; I think you’re reading too far into it. It shows the power of their model and possible applications towards privacy. It’s not a stance that that hypothetical camera is actually great for privacy.
- wizzwizz4 3y agoBut a camera that masks 90% of the pixels isn't privacy-preserving. It's just a 90s consumer-grade webcam. They haven't shown that the approach works with actual privacy measures, which makes their claims in this area dubious.
- bentt 3y agoI tried to get GPT-4 to act as a compiler and it didnt go so well but it felt like it was mainly because it didnt believe it could. After much consternation it was willing to put together a hello world in x86 assembly.
- optimalsolver 3y agoThe little language model that could.
- vidarh 3y agoA lot of problems are avoided with strategic "as a [suitable role], [task]" or "you are a ...". Not sure what you tried, but variations of that have been enough to bypass all kinds of weird objections for me.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- Aerbil313 3y agoThis seems like a downgrade. Intuition suggests it should lead to better performance if you first parse/process the input to represent the actual input space better.
- refulgentis 3y agoYou'd think this, but, RGB color is ~meaningless and all the image stuff works great. I long for a model that is trained in a perceptual space, and yet, I doubt it'll matter
- wizzwizz4 3y agoIt's not faithfully representative of the post-processing performed by the average person's visual system, but that doesn't make it meaningless by any stretch of the imagination.
- refulgentis 3y agoIntuitively, it’s meaningless. What’s brighter, 255 G or 255 R?
- wizzwizz4 3y agoDepends on the picture. https://en.wikipedia.org/wiki/Checker_shadow_illusion https://en.wikipedia.org/wiki/Checker_shadow_illusion
- refulgentis 3y agoThat's called viewing conditions in color science: it's sort of like saying "in a pitch black room, the same!" Let's assume the same viewing conditions, i.e it's not a trick question: answer is green, by about 30% IIRC
- majormajor 3y agoThat seems much more like "incomplete" than "meaningless." There's enough meaning to answer this question, no: "what's greener, 255 G or 255 R"? Or "what's greener, #FF6600 or #FF0066?"
- AndrewKemendo 3y agoI definitely appreciate the technical privacy focus here and I’m curious if that is the whole purpose here. I don’t see that many benefits to this method other than privacy - the ability to process multiple data types with a singular architecture is really cool but not SOTA beating or comparable really to large multimodal systems like imagebind [1] [1] https://arxiv.org/pdf/2305.05665.pdf https://arxiv.org/pdf/2305.05665.pdf
- quickthrower2 3y agoHad a quick skim. I was really struggling to see the practical advantage of this. Because you can convert different formats to what the model needs easily. It feels like some strawman are set up. In the world of the paper, people have to painstakingly convert the image to the correct format for the model, and "hand-crafting a model stem for each modality" But once you have set up a model, and got it working for say .tiff then to work on any other format you can just use ImageMagick or something? Unless you want the meta data too? I think the use case for a model that works on bytes is as a kind of "easy to install" package on local devices, like a security camera, regular camera etc that will work with whatever the local file format is. Also, a dollar into the "is/are all you need" jar please.
- thelastparadise 3y ago> Also, a dollar into the "is/are all you need" jar please. "Is/Are All You Need Considered Harmful"
- IIAOPSW 3y agoHarm is all you need.
- kmstout 3y agoAll you need is harm (ba da-da da-da) All you need is harm (ba da-da da-da) All you need is harm Harm, harm is all you need
- pwdisswordfishc 3y agoThe Unreasonable Effectiveness Considered Harmful for Fun and Profit is All You Need
- bloaf 3y agoOne interesting application is to imagine you have a bunch of data in some proprietary binary format, with metadata in plain text. This is saying you could potentially make a network that generates new instances of the proprietary format without having to know the details of that format.
- fernly 3y agoFascinating philosophical question: if an AI observes a stream of bytes but doesn't decode a camera's output into an image -- and yet can raise an alarm if the camera is aimed at a certain class of subject -- does the AI "see" the camera image? They should have named the system "Blindsight"[1]. Apple has a serious corporate concern about proposed legal requirements to detect and report child porno[2], while other legal requirements try to protect user privacy. This appears to be an attempt to say, "We can detect your CP image storage without ever actually looking at your stored images." [1] https://en.wikipedia.org/wiki/Blindsight https://en.wikipedia.org/wiki/Blindsight [2] https://en.wikipedia.org/wiki/Regulation_to_Prevent_and_Combat_Child_Sexual_Abuse https://en.wikipedia.org/wiki/Regulation_to_Prevent_and_Comb...
- deleted 3y ago[deleted]
- troops_h8r 3y agoYou might like the scifi novel of the same name by Peter Watts
- escapecharacter 3y agoThe real conclusion from your question is that the AI never “sees” the image even if it did decode it.
- messe 3y agoAnd how do you draw that conclusion?
- nextworddev 3y agoWhat does it mean to “see”? Are we assuming that eyes are involved?
- holoduke 3y agoThe AI doesnt decode the image with a human written decoder. But it decodes with learned model parameters. So it still decodes.
- arakageeta 3y agoIn terms of processing raw bytes, inference on raw camera data, bypassing the ISP, is a more relevant application. This can reduce latency in camera inference.
- airgapstopgap 3y agoInteresting release from Apple but there's been a similar (and I think more promising architecturally) paper from Meta lately: > MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers https://arxiv.org/abs/2305.07185 https://arxiv.org/abs/2305.07185