7 ms·
Spotting base64 encoded JSON, certificates, and private keys
- Faaak 1y agoNitpick, but enclosing the first string in single quotes would make the reading better: $ echo '{"' | base64 Vs $ echo "{\"" | base64
- mdaniel 1y agoThat's a huge pet peeve of mine, but similar to the control-r comment elsewhere, I have just come to terms with the fact that most developers are allergic to shell
- delecti 1y agoOh that's nifty. Spotting base64 encoded strings is easy enough (and easy enough to test that I give it a shot if I'm even vaguely curious), but I'd never looked at them closely enough to spot patterns.
- morkalork 1y agoAfter copy and pasting enough access tokens into various tools you pick up on it pretty fast.
- rgovostes 1y agoThere is a Base64 quasi-fixed point: $ echo -n Vm0 | base64 Vm0w It can be extended indefinitely one character at a time, but there will always be some suffix.
- thaumasiotes 1y agoNote that the suffix will grow in length with the input, making it less and less interesting. (Because the output is necessarily 8/6 the size of the input, the suffix always adds 33% to the length.)
- o11c 1y agoFor reference, a program to generate the quasi-fixed point from scratch: #!/usr/bin/env python3 import base64 def len_common_prefix(a, b): assert len(a) < len(b) for i in range(len(a)): if a[i] != b[i]: return i return len(a) def calculate_quasi_fixed_point(start, length): while True: tmp = base64.b64encode(start) l = len_common_prefix(start, tmp) if l >= length: return tmp[:length] print(tmp[:l].decode('ascii'), tmp[l:].decode('ascii'), sep='\v') # Slicing beyond end of buffer will safely truncate in Python. start = tmp[:l*4//3+4] # TODO is this ideal? if __name__ == '__main__': final = calculate_quasi_fixed_point(b'\0', 80) print(final.decode('ascii')) This ultimately produces: Vm0wd2QyUXlVWGxWV0d4V1YwZDRWMVl3WkRSV01WbDNXa1JTVjAxV2JETlhhMUpUVmpBeFYySkVUbGho
- syncsynchalt 1y agoFrom the other direction, you'd call it a tail-eating unquine?
- metalliqaz 1y agoSomething similar pops up if you have to spend a lot of time looking at binary blobs with a hex editor. Certain common character sequences become familiar. This also leads to choosing magic numbers in data formats that decode to easily recognized ASCII strings. I'm sure if I worked with base64 I'd be choosing something that encoded nicely into particular strings for the same purpose.
- skissane 1y agoRelated trick I've learnt: binary data containing lots of 0x40 may be EBCDIC text, or binary data containing embedded EBCDIC strings – 0x40 is EBCDIC space character Probably not a very useful trick outside of certain specific environments
- metalliqaz 1y agoso, uhh... insurance or banking?
- shortrounddev2 1y agoI discovered this when I created a JWT system for my internship. I got really good at spotting JWTs, or any base64 encoded json payloads in our Kafka streams
- cogman10 1y agoI don't really love this. It just feels so wasteful. JWT does it as well. Even in this example, they are double base64 encoding strings (the salt). It's really too bad that there's really nothing quite like json. Everything speaks it and can write it. It'd be nice if something like protobuf was easier to write and read in a schemeless fashion.
- reactordev 1y agoWe just need to sacrifice n*field_count to a header describing the structure. We also need to define allowed types.
- Muromec 1y ago>Everything speaks it and can write it. asn.1 is super nice -- everything speaks it and tooling is just great (runs away and hides)
- dlt713705 1y agoWhat’s wrong with this? The purpose of Base64 is to encode data—especially binary data—into a limited set of ASCII characters to allow transmission over text-based protocols. It is not a cryptographic library nor an obfuscation tool. Avoid encoding sensitive data using Base64 or include sensitive data in your JWT payload unless it is encrypted first.
- zokier 1y agoJSON is already text based and not binary so encoding it with base64 is bit wasteful. Especially if you are going to just embed the text in another json document. And of course text-based things themselves are quite wasteful.
- pak9rabid 1y agoExactly. Using base64 as an obfuscation tool, or (shudder) encryption is seriously misusing it for what it was originally intended for. If that's what you need to do then avoid using base64 in favor for something that was designed to do that.
- 1y ago
- gabesullice 1y agoI love this post style. Never stop learning friend!
- zavec 1y agoYeah, people are being snarky and saying it's obvious, but it was new to me! I guess I'm not staring at base64 all that often. It's a neat trick though, now I'm going to pay attention next time I have an opportunity to use it.
- mmastrac 1y agoI built a JWT support library at work (https://github.com/geldata/gel-rust/tree/master/gel-jwt https://github.com/geldata/gel-rust/tree/master/gel-jwt) and I can confirm that JWTs all sound like "eyyyyyy" in my head.
- isoprophlex 1y agoeyy lmao
- Muromec 1y ago>JWTs all sound like "eyyyyyy" in my head. "eeey bruh, open the the API it's me"
- proactivesvcs 1y ago"Is that you again, punk zip?" when seeing the first few bytes of a zip file.
- ragmodel226 1y agoProbably shouldn’t call Phil Katz a punk
- ragmodel226 1y agoAlso, MZ in exe is Mark Zbikowski.
- layer8 1y agoIt's like how all certificates sound like "miiiiii".
- snickerdoodle12 1y agoIsn't this obvious to anyone who has seen a few base64 encoded json strings or certificates? ey and LS are a staple.
- mmastrac 1y ago`MII` for RSA private keys.
- Muromec 1y agoMII is not RSA, it's an opening header of asn1 structure encoded to DER -- 30 82 0x which is basically "{" when which can be pretty much anything from x509 certificate to private keys fro ECDSA. Actual RSA oid is somewhere in the middle.
- mmastrac 1y agoTrue, but for the most part, RSA keys are the only keys that anyone encounters that start with long SEQUENCEs requiring two-byte lengths. `eY` could be any JSON, but it's most likely going to be a JWT. Neither is a perfect signal, but contextually is more likely correct than not.
- Muromec 1y agoThat depends on the kind of abyss you are staring into. Mine had plenty of non-RSA keys, certificates (which are of course two-byte length all the time) and CMS containers.
- FelipeCortez 1y agoI thought so too, but xkcd 1053 / lucky 10000, I guess! I knew about ey but not LS
- InfoSecErik 1y agoIMO depends on your career. I did a lot of pentesting with Burp Suite so I was able to (forced to) pick it up.
- athorax 1y agoBase64 encoded yaml files will also be LS-prefixed if they have the document marker (---)
- thibaultamartin 1y agoThat's right, I've added an errata to clarify. Thanks for the heads up!
- DPDmancul 1y agoYou can increase the guess accuracy a little by looking for the "tLS" characters , skipping the first 3 chars. Also this is a mnemonic about TLS and identifies all strings starting with 5 dashes, excluding so most of yaml documents
- gnabgib 1y agoYou can spot Base64 encoded JSON. The PEM format (that begins with `-----BEGIN [CERTIFICATE|CERTIFICATE REQUEST|PRIVATE KEY|X509 CRL|PUBLIC KEY]-----`) is already Base64 within the body.. the header and footer are ASCII, and shouldn't be encoded[0] (there's no link to the claim so perhaps there's another format similar to PEM?) You can't spot private keys, unless they start with a repeating text sequence (or use the PEM format with header also encoded). [0]: https://datatracker.ietf.org/doc/html/rfc7468 https://datatracker.ietf.org/doc/html/rfc7468
- ctz 1y agoThe other base64 prefix to look out for is `MI`. `MI` is common to every ASN.1 DER encoded object (all public and private keys in standard encodings, all certificates, all CRLs) because overwhelmingly every object is a `SEQUENCE` (0x30 tag byte) followed by a length introducer (top nibble 0x8). `MII` is very very common, because it introduces a `SEQUENCE` with a two byte length.
- Muromec 1y agoI for one wait for the day when quantum computers will break all the encryption forever so nobody will have to suffer broken asn1 decoders, plaintext specifications of machine-readable formats and unearned aura of arcane art that surrounds the whole thing.
- ctz 1y agoasn1 enjoyers can also look forward to the sweet release of death. though if you end up in hell you might end up staring at XER for the rest of eternity
- schoen 1y agoYou'll also see "AQAB" a lot. This is the base64 version of the integer representation of 65537, the usual public exponent parameter e in modern RSA implementations.
- thibaultamartin 1y ago
- yahoozoo 1y agobabby's first base64
- cluckindan 1y agoOn mobile, the long rows in the code blocks blow up the layout.
- Muromec 1y agoAfter staring one time too much at base64-encoded or hex-encoded asn1 I started to believe that scene in the Matrix where operator was looking at raw stream from Matrix at his terminal and was seeing things in it.
- cestith 1y agoYears ago I was part of a group of people I knew who could read and edit large parts of sendmail.cf by hand without using m4. Other people who had to deal with mail servers at the time certainly treated it like a superpower.
- Muromec 1y agoWhere I work right now superpower of the day is pressing ctrl-r in the terminal.
- nickdothutton 1y agoA significant part of my 1st ever job consisted of editing sendmail.cf’s by hand. Occasionally had to defer to my boss at the time for the real mind bending stuff. I now believe that he was in fact a non-human alien.
- quesera 1y agoIn some ways, I miss those days. Spending hours wrangling sendmail.cf, and finally succeeding, felt like a genuine accomplishment. Nowadays, things just work, mostly. How boring.
- anyfoo 1y agoI feel that nowadays, it's a combination of "things just work" and "if they don't, good luck figuring out why". I recently installed Tru64 UNIX on a DEC Alpha I got off eBay. I felt like it was more sluggish than it should be, so I looked around at man-Pages about the VM (virtual memory, not virtual machine) subsystem, and was amazed how cleanly and detailed it was described, and what insights I could get about its state. The sys_attrs_vm man-page alone, which just describes every VM-layer tunable, gave a pretty good description of what the VM subsystem does, how each of those tunables affects it, and why you might want to change it. Nowadays, things are massively complex, underdocumented (or just undocumented), constantly changing, and often inconsistent between sub-parts. Despite thinking that I have both wide and deep knowledge (I'm a low-level code kernel dev), it often takes me ages to figure out the root cause of sometimes even simple problems.
- curiousObject 1y agoYikes! It would be smart to bury these strings in an ad hoc obfuscation so they aren’t so obvious. It doesn’t even need to be much better than ROT13. Security by obscurity is good for this situation.
- dhosek 1y agoKind of reminds me of a junior being amazed when I was able to read ascii strings out of a hex stream. Us old folks have seen a lot.
- schoen 1y agoFor anyone here who's never pondered it ("today's lucky 10,000"?), there's a lot of intentional structure in the organization of ASCII that comes through readily in binary or hex. https://altcodeunicode.com/ascii-american-standard-code-for-information-interchange/ https://altcodeunicode.com/ascii-american-standard-code-for-... The first nibble (hex digit) shows your position within the chart, approximately like 2 = punctuation, 3 = digits, 4 = uppercase letters, 6 = lowercase letters. (Yes, there's more structure than that considering it in binary.) For digits (first nibble 3), the value of the digit is equal to the value of the second nibble. For punctuation (first nibble 2), the punctuation is the character you'd get on a traditional U.S. keyboard layout pressing shift and the digit of the second nibble. For uppercase letters (first nibble 4, then overflowing into first nibble 5), the second nibble is the ordinal position of the letter within the alphabet. So 41 = A (letter #1), 42 = B (letter #2), 43 = C (letter #3). Lowercase letters do the same thing starting at 6, so 61 = a (letter #1), 62 = b (letter #2), 63 = c (letter #3), etc. The tricky ones are the overflow/wraparound into first nibble 5 (the letters from letter #16, P) and into first nibble 7 (from letter #16, p). There you have to actually add 16 to the letter position before combining it with the second nibble, or think of it as like "letter #0x10, letter #0x11, letter #0x12..." which may be less intuitive for some people). Again, there's even more structure and pattern than that in ASCII, and it's all fully intentional, largely to facilitate meaningful bit manipulations. E.g. converting uppercase to lowercase is just a matter of adding 32, or logical OR with 0x00100000. Converting lowercase to uppercase is just a matter of subtracting 32, or logical AND with 0x11011111. For reading hex dumps of ASCII, it's also helpful to know that the very first printable character (0x20) is, ironically, blank -- it's the space character.
- schoen 1y agoI should just have put the printable character chart right here in the post for people to compare: 0 1 2 3 4 5 6 7 8 9 A B C D E F .. 2 ! " # $ % & ' ( ) * + , - . / 3 0 1 2 3 4 5 6 7 8 9 : ; < = > ? 4 @ A B C D E F G H I J K L M N O 5 P Q R S T U V W X Y Z [ \ ] ^ _ 6 ` a b c d e f g h i j k l m n o 7 p q r s t u v w x y z { | } ~ I don't have a mnemonic for punctuation characters with second nibble >9, or for the backtick. The @ can be remembered via Ctrl+@ which is a way of typing the NUL character, ASCII 00 (also not coincidental; compare to Ctrl+A, Ctrl+B, Ctrl+C... for inputting ASCII 01, 02, 03...).
- iou 1y ago“Welcome to the party, pal!”
- deleted 1y ago[deleted]
- netsharc 1y agoGood knowledge, now explain why it's like that. {" is ASCII 01111011, 00100010 Base64 takes 3 bytes x 8 bits = 24 bits, groups that 24 bit-sequence into four parts of 6 bits each, and then converts each to a number between 0-63. If there aren't enough bits (we only have 2 bytes = 16 bits, we need 18 bits), pad them with 0. Of course in reality the last 2 bits would be taken from the 3rd character of the JSON string, which is variable. The first 6 bits are 011110, which in decimal is 30. The second 6 bits are 110010, which in decimal is 50. The last 4 bits are 0010. Pad it with 00 and you get 001000, which is 8. Using an encoding table (https://base64.guru/learn/base64-characters https://base64.guru/learn/base64-characters), 30 is e, 50 is y and 8 is I. There's your "ey". Funny how CS people are so incurious now, this blog post touches the surface but didn't get into the explanation.
- perching_aix 1y agoI'd be very hesitant to consider this as some runaway symbol of "CS people being incurious now" over the author simply not being this deeply invested in this at the time of writing in the context of their discovery, especially since it almost certainly doesn't actually matter for them beyond the pattern existing, if even that does.
- netsharc 1y ago> it almost certainly doesn't actually matter for them beyond the pattern existing, if even that does. https://web.cs.ucdavis.edu/~rogaway/classes/188/materials/the%20machine%20stops.pdf https://web.cs.ucdavis.edu/~rogaway/classes/188/materials/th...
- perching_aix 1y agoAnd there goes my limit of curiosity now regarding this. I'm interested in what you have to say, but not 25 page mini-novel PDF from someone else interested. I'm glad you enjoyed that piece, but I have no interest in reading it, nor do I think it's reasonable for you to expect me to be interested. Much like with the author and the specifics of this encoding.
- karel-3d 1y agoI debugged way too many JWT tokens I know eyJhbG by heart
- karel-3d 1y agothey technically don't need to begin like that! JWT is JSON and is therefore infamously vague... but in practice they for some reason always begin with "alg" so always like eyJhbG
- xg15 1y agoHas anyone tried to send a JWT token with the fields in a different order (e.g. a long key first and key ID and algorithm behind) and see how many implementations will break?
- karel-3d 1y agothere are better things to do, like send json that has "alg" twice, each different (one of them "none" ideally) and different implementations handle it differently
- syncsynchalt 1y agoI didn't even realize I knew that string, but I recognized it immediately from your post.
- Sophira 1y agoMathematically, base64 is such that every block of three characters of raw input will result in four characters of base64'd output. These blocks can be considered independent of each other. So for example, with the string "Hello world", you can do the following base64 transformations: * "Hel" -> "SGVs" * "lo " -> "bG8g" * "wor" -> "d29y" * "ld" -> "bGQ=" These encoded blocks can then be concatenated together and you have your final encoded string: "SGVsbG8gd29ybGQ=" (Notice that the last one ends in an equals sign. This is because the input is less than 3 characters, and so in order to produce 4 characters of output, it has to apply padding - part of which is encoded in the third digit as well.) It's important to note that this is simply a byproduct of the way that base64 works, not actually an intended thing. My understanding is that it's basically like how if you take an ASCII character - which could be considered a base 256 digit - and convert it to hexadecimal (base 16), the resulting hex number will always be two digits long - the same two digits, at that - even if the original was part of a larger string. In this case, every three base 256 digits will convert to four base 64 digits, in the same way that it would convert to six base 16 digits.
- zokier 1y agonitpick but ascii would be base128, largest ascii value is 0x7f which in itself is a telltale if you are looking at hex dumps.
- Sophira 1y agoYeah, I was aware of that, but I figured it was the easiest way to explain it. It's true that "character representation of a byte" is more accurate, but it doesn't roll off the tongue as easily.
- Sophira 1y agoBy the way, I would guess that this is almost certainly why LLMs can actually decode/encode base64 somewhat well, even without the help of any MCP-provided tools - it's possible to 'read' it In a similar way to how an LLM might read any other language, and most encoded base64 on the web will come with its decoded version alongside it.
- tetha 1y agoReminds me of 1213486160[1] Besides that, I just spent way too much time figuring out this is an encrypted OpenTofu state. It just looked way too much like a terraform state but not entirely. Tells ya what I spend a lot of time with at work. This is probably another interesting situation in which you cannot read the state, but you can observe changes and growth by observing the ciphertext. It's probably fine, but remains interesting. 1: https://rachelbythebay.com/w/2016/02/21/malloc/ https://rachelbythebay.com/w/2016/02/21/malloc/
- koolba 1y agoWell duh. It’s a deterministic encoding. Does not matter if it’s base64, hex, or even rot13. Is this the state of modern understanding of basic primitives?
- calibas 1y agoThe encoded JSON string is going to start with "ey", unless there's whitespace in the first couple characters. Also, it seem like the really important point is kind of glossed over. Base64 is not a kind of encryption, it's an encoding that anybody can easily decode. Using it to hide secrets in a GitHub repo is a really really dumb thing to do.
- palunon 1y agoThere is actual encryption here. The base64 JSON only encodes the salt and parameters of the key derivation function used to encrypt the data.
- cfontes 1y agoNot directly correlated but I know a old guy that can decrypt EBCDIC and credit card positional data format on the fly. And sometimes it was a "feeling" he couldn't explain it properlly but knew exactly the value, name and other data. It was amazing to see him decode VISA and MASTER transactions on the fly in logs and other places.
- andrepd 1y agoThat's got to be the most niche party trick I've ever heard of.
- VoidWhisperer 1y agoI would hope that these logs don't include the full details of the credit card (such as number/cvv).. if it does, the company that is logging this info could end up having some issues with Visa/MC Edit: Now that I looked at it a little deeper, i'm assuming they are talking about these[0] sort of files? [0]: https://docs.helix.q2.com/docs/card-transaction-file https://docs.helix.q2.com/docs/card-transaction-file
- Aspos 1y agoPCI DSS is a relatively new thing. Before it card data flew in the open
- Rygian 1y agoI've seen that done live, during audits, on live logs on the screen. Needles to say, audit didn't fly first time round (those logs should have been redacted).
- userbinator 1y agoI can do the same with several proprietary network protocols and data formats I've worked on, as well as some x86 Asm - once you start seeing enough of it, you begin to absorb it almost like learning a language.
- benatkin 1y agoI'm more partial to PCFkb2N0eXBlIGh0bWw+
- Naru41 1y agoFunny trivia. But of course -- there is absolute zero reason to base64 encode ascii text. Evenmore laughable to put Json encoded in base64 text inside regular Json.
- nneonneo 1y agoUseful ones to know: - `R0lGOD` - GIF files - `iVBOR` - PNG files - `/9j/` - JPG files - `eyJ` - JSON - `PD94` - XML - `MII` - ASN.1 file, such as a certificate or private key These are nice to know since they show up pretty frequently (images in data: URLs, JSON/XML/ASN.1 in various protocols).
- feliixh 1y agoVery useful!
- 0points 1y ago> GIF, PNG, JPG It makes more sense to transmit binary formats in binary. You would save bandwidth, memory and a decoding step. Then you could also inspect the header bytes, instead of memorizing how they present in some intermediate encoding.
- demurgos 1y agoSometimes you need to embed binary data in a text format (e.g. JSON).
- naikrovek 1y agothe amount of data i have stuffed into json as base64 encoded text makes me sick. i wrote a glTF model converter once. 99% of those millions of JSON files I wrote were base64 encoded binary data. a single glTF model sometimes wants to be two files on disk. one for the JSON and one for the binary data, and you use the JSON to describe where in the binary data the vertices are defined, and other windows for the various other bits like the triangles, triangle fans, textures, and other stuff are stored. But you can also base64 encode that data and put it in the JSON file and not have a messy double-file model. so that's what I did and I hated it. but it still felt better than having .gltf files and .bin files which together made up a single model file.
- mr_mitm 1y agoKnowing these magic bytes in base64 is mostly relevant in situations in which you see data encoded by other people, which means you probably had no control over the encoding. Other people (or rather every body) sometimes do things which don't make sense.
- pabs3 1y agoAre there any secret scanning tools that know about base64 encoded stuff?
- bashwizard 1y agoA dev who have never seen a base64 string before? Fascinating.
- doppelgunner 1y agoYou should not store sensitive data into jwt.
- envolt 1y agoI've seen too many JWT tokens that it is very natural at this point