7 ms·
Cool hack overall, but some of the statements about protobuf strike me as odd. > Exploit a Protobuf Flaw to Easily Remove All Ads by Changing One Byte He's in
by kentonv 3y ago
Cool hack overall, but some of the statements about protobuf strike me as odd.
> Exploit a Protobuf Flaw to Easily Remove All Ads by Changing One Byte
He's intentionally corrupting the tag on one field of the protobuf. The "flaw" is apparently that Protobuf ignores tag numbers it doesn't recognize. But that's not a flaw, it's a core design feature of Protobuf, designed to allow extensibility.
> Notice how the Protobuf response payload is 1.87 MiB? As I said, Google makes it computationally expensive to decode, alter, and re-encode without the C++ source proto files, but a quick linear scan takes no effort at all.
1.87MB is not that big, and presumably these messages only come through occasionally, not in a constant stream, so I'm a bit perplexed about the purported performance barrier here.
The text seems to be claiming that Protobuf encoding is designed to be expensive to decode, but it's actually the opposite, Protobuf is intended to be efficient to decode.
He says you need the source proto (schema) files to make it efficient, but this isn't really true, you can decode directly into UnknownFieldSet easily enough.
Or better, he could have written his own fake .proto schema that covers just the one field he is aiming to remove. This would have been much less error-prone than the string-scanning approach which could accidentally match other data where the same byte sequence happened to appear by coincidence.
> While computationally expensive, decoding, editing, and re-encoding without the original schema leads to a modified encoding. This is likely because we cannot detect if ZigZag encoding is being used, or if a number is an int32, int64, sint32/64, varint, etc., plus the order of object fields is normally non-deterministic. Here is some Protobuf trivia on the matter:
I think he's misunderstanding here. The point is that the Protobuf encoder is allowed to encode fields in any order, hence decoding and immediately re-encoding a message can lead to different bytes. But, the receiver is supposed to treat the message the same regardless. It's unlikely that the YouTube app is going to notice if the field order changes.
(I used to work on Protobufs, a long time ago.)
- notjoemama 3y agoBrilliant response and observations. I’m not sure I’d agree with this one tiny assertion though: “1.87MB is not that big“ I have lived in rural communities most of my life and unless I’m on my own Wi-Fi, this actually is a big file to download. Maybe they have a workaround for mobile? Mostly I just want to add that rural Wi-Fi still struggles with Web 2.0 architecture. More often we’re operating on 2-4G speeds. Although if you’re in any kind of metro area with a population that supports better infrastructure, 1.87MB has mostly become a small file size. Except maybe at 6pm when everyone on your cable trunk is streaming content. :)
- sedatk 3y agoIf you struggle with 2MB payload, how do you expect to stream video, which is usually hundreds of megabytes? You’re right about 2MB being a large payload in some network configurations, but I think it isn’t that big in Youtube’s context.
- SoftTalker 3y ago720p or less.
- Zuiii 3y agoI used to stream all my videos 144p less than 5 years ago because streaming any higher resulted in a stuttering buffering mess. Youtube was the only streaming service I could watch because most other sites didn't bother with resolutions/bitrates so low. Maybe that's why youtube won out. Much like how whatsapp made it their mission to run on every phone they could get their hands on. Now we have proper internet I don't but don't for a second doubt that a sizable portion of the internet still suffer from slow speeds, even those living in developed countries.
- LordShredda 3y agoI also have a theory that Minecraft got so popular on YouTube because the videos look legible with large textures even on 144p
- miah_ 3y agoI'm also in the rural 'my ISP is a WIFI ISP' boat.. I pay for 10/2! While they do burst its not uncommon for my speeds to dip down into single digits. Sometimes I have to tune my video streams down as low as they'll go. Packet latency can frequently be so bad that playing online games is VERY DIFFICULT so I mostly play single player, downloads of new games often take 24 hours+. My ISP when I talk to them about their speed issues "Well, 7mbps is all you need to stream Hulu", which is.. a terrible response. Want to do something else while streaming? What if _two_ people want to stream? What if I want to stream a video while my partner is playing a game online? Ads are simply a _HUGE WASTE_ of my bandwidth, so I filter as much as I can.
- arkadiyt 3y ago> without the C++ source proto files Shameless plug: I wrote a project to generate source proto files from binaries called protodump [1] - it regenerates all the message/field definitions (including the original names). It would just require pulling the binary off the AppleTV box [1]: https://github.com/arkadiyt/protodump https://github.com/arkadiyt/protodump
- ComputerGuru 3y agoHey, that’s really cool! Upvoted! ¿What binary formats are supported? - I couldn’t see that in the README and I’m guessing it isn’t universal (though I could be wrong!). As for this specific context, presumably at least the bulk of the fields are shared between YouTube clients across architectures and platforms - it might be easier to just run the dump against binaries extracted from the YouTube apk than getting the specific Apple TV YouTube app binary. (For iPhone, you can just make a full iTunes backup but for Apple TV I imagine you’re going to need to do some forensic binary spelunking.)
- kentonv 3y agoIt looks like it is in fact universal. Just glancing at the code here, it looks like the tool searches any arbitrary file for bytes that look like encoded protobuf descriptors, specifically looking for bytes that are plausibly the beginning of a FileDescriptorProto message defined here: https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/descriptor.proto https://github.com/protocolbuffers/protobuf/blob/main/src/go... This takes advantage of the fact that such descriptors are commonly compiled into programs that use protobuf. The descriptors are usually embedded as constant byte arrays. That said, not all protobuf implementations embed the descriptors and those that do often have an option to inhibit such embedding (at the expense of losing some dynamic introspection features).
- Scaevolus 3y agoThe trickiest part of decoding protobufs blind is how embedded messages and strings have the same tag type, but that's pretty easy to handle. You can write a simple protobuf decoder in a couple hundred lines of code if you don't want to pull in the full protoc dependency: https://github.com/kubernetes/test-infra/blob/master/gubernator/pb_glance.py https://github.com/kubernetes/test-infra/blob/master/guberna... https://github.com/kubernetes/test-infra/blob/master/gubernator/testgrid.py https://github.com/kubernetes/test-infra/blob/master/guberna...
- krackers 3y ago>but that's pretty easy to handle. How do you solve it? Is it just heuristically checking where if the varlength byte sequence itself contains tags, it's likely to be a message?
- cbsmith 3y agoI had much the same points of confusion when reading this. It's not like the design principles behind Protubuf are a secret. This is all documented plainly.
- motiejus 3y ago> (I used to work on Protobufs, a long time ago.) What and understatement. For those unaware, Kenton made protobufs to what they are today. Protobuf was my first dip into what are IDLs; it seemed like a magical idea at the time (having discovered protobuf ater writing my own rudimentary IDL first, protobuf was amazing).
- omneity 3y agoAs the old saying goes, the best way to get help on the internet is to post a wrong answer. In this case, getting a reply straight off the source.