10 ms·

Base64 Encoding, Explained

• bediger4000 3y ago
  > Base64 encoding takes binary data and converts it into text, specifically ASCII text. Perpetuates the idea that there's "binary" and "text", which is incorrect, but also implies you can't encode ordinary ASCII text into base64.
  • farhanhubble 3y ago
    The article actually shows an example of text to base64 encoding. But base64 is generally used for encoding data in places where only ascii is admissible like URLs and inlined binary blobs
    • jillesvangurp 3y ago
      It's part of a lot of web standards and also commonly used for crypto stuff. E.g. the plain text files in your .ssh directory are typically in base64 encoding; if you use basic authentication that's $user:$passwd base64 encoded in a header; you can indeed use it to have images and other inline content in the url in web pages; email attachments are usually base64 encoded. And so on. One of those things any decent standard library for just about any language would need.
  • Aachen 3y ago
    There is binary and text, though. Many bit sequences aren't valid in a given text encoding (such as UTF-whatever) and so trying to use them as text is an error. I understand what you mean, of course text can be represented and treated as binary, and the inverse often as well although it isn't necessarily true. Even in Windows-1252, where the upper 127 characters are in use, there are control characters such as null, delete, and EOT which I'd be impressed if a random chat program preserves them across the wire. I also don't read an implication that ASCII couldn't be converted to b64
  • numlock86 3y ago
    > Perpetuates the idea that there's "binary" and "text" [...] Well, there is binary, and there is text. Sure, all text - like "strawman" ;) - is binary somehow, but not all binary data is text, nor can be even interpreted as such, even if you tried really hard ... like all those poor hex editors.
    • imetatroll 3y ago
      All text is binary. Everything is binary. "Somehow" is an odd choice of words here.
      • roydivision 3y ago
        Came to say this.
      • wongarsu 3y ago
        Text is text. Text is encodable as binary. If text was binary, that encoding would be unique, but it isn't. Latin-1 encodes "Ü" differently than UTF-8 does, and even the humble "A" could be a 0x41 in ASCII and UTF-8 or a 0xC1 in EBCDIC
        • imetatroll 3y ago
          This is just ... not how I look at it at all. Everything is represented in powers of two... binary.
          • taway1237 3y ago
            Everything is representable as binary, but not everything is binary. The abstract concept of 'A' has no inherent binary representation, and 0x41 is just one of the options. Representing Pi or e calls for even more abstract encoding, even though they are a very specific concept. Text is not binary, but text has to be encoded to binary (one way or another) in order to do any kind of computer assisted processing of it. But we tend to think of text in abstract terms, instead of "utf-8 encoded bytes", hence this abstraction is useful.
  • Sohcahtoa82 3y ago
    > but also implies you can't encode ordinary ASCII text into base64. I don't think it implies that at all. Text isn't binary. Text can be encoded in binary, and there are different ways to do it. ASCII, UTF-8/16/32, latin-1, Shift-JIS, Windows-1252, etc. Many can't encode all text characters, especially languages that don't use the Latin alphabet. The fact that you have to ensure you're using the correct encoding when processing text from a binary stream is proof enough that text isn't binary. Python before 3.x allowed you to treat binary and text as equal, and it often caused problems.
  • xp84 3y ago
    Most developers (etc) use "binary data" or "a binary format" as a shorthand for "not even remotely ASCII or Unicode" - as opposed to the opposite, like a .txt file or HTML or Markdown, where it's readable and printable to the screen. Of course if it's in a file or in RAM or whatever, it's always ultimately stored as 0s and 1s but that's not the sense we mean here.
• adamzochowski 3y ago
  There is also base64URL , where the encoding uses different ascii characters that are url safe. I have seen some devs use BASE64URL but call it just base64, and that can lead to some problems for unaware. https://datatracker.ietf.org/doc/html/rfc4648#section-5 https://datatracker.ietf.org/doc/html/rfc4648#section-5
  • JimDabell 3y ago
    The problem with base64url is that ~ and . are not letters. This means that double-clicking on something encoded with base64url isn’t going to select the whole thing if you want to copy-paste it. This is annoying needless friction in a huge number of use cases. Base62 encoding (0-9A-Za-z) is almost as efficient as base64url and retains URL safety but is more easily copy-pasted. If you want to eliminate ambiguity for human readers, you can drop to Base58 but in almost all cases, if you are BaseXX-encoding something, it’s long enough that copy-pasting is the norm, so it doesn’t usually matter. https://en.wikipedia.org/wiki/Base62 https://en.wikipedia.org/wiki/Base62
    • iainmerrick 3y ago
      The problem with base64url is that ~ and . are not letters. No, typically the extra characters used are “-“ and “_”. That’s what the table in the IETF link shows.
      • Acinyx 3y ago
        The issue remains: "-" breaks double clicking to select the full string, which means you'll have to manually select all the characters before copying. Same thing happens with UUIDs: using double clicking, you can only select one block at a time. This isn't a major issue, which means there's no easy answer and it generally comes down to preference if this is a requirement or not.
        • yrro 3y ago
          Double-clicking foo-bar_baz in GNOME Terminal selects the entire string. Anyway, this is something that is user-configurable surely?
• erhaetherth 3y ago
  Funny. I'm trying to write a base50 encoder now. No good reason, just 'cuz. Can't quite figure out what to do with the half a bit. Gotta carry it forward somehow until I have a full char again but haven't come up with a good scheme.
  • rustybolt 3y ago
    What do you mean, half a bit? A typical base50 encoder would use 50 characters, so say 309 would get encoded as 69 (309 = 6 * 50 + 9 * 1).
    • cjm42 3y ago
      He meant approximately half a bit. Base64 uses 64 characters, which means 6 bits per character. 50 characters gives you about 5.64 bits per character. Unlike Base64's 3-bytes-equals-4-characters, that never comes out evenly.
  • VMG 3y ago
    write a base-n encoder and use 50 as a parameter
  • meindnoch 3y ago
    Interesting choice. You won't be able to do it in batches because 50^n = 2^m doesn't have a solution with integer n and m. So you basically have to implement a bignum with division and modulo operations.
    • charlieyu1 3y ago
      On the other hand, base91 is a real thing
      • meindnoch 3y ago
        Base91 encodes 13 bits (8192 possibilities) in two base91 symbols (8281 possibilities), so it's not 100% efficient.
• benjaminwai 3y ago
  Just a note with the Bash encoding method. It should be with the -n option: $ echo -n "abcde" |base64 Otherwise, without the -n, echo injects an extra newline character to the end of the string that would become encoded.
  • manojlds 3y ago
    Yeah been bit by few times. Somehow keep forgetting.
  • meindnoch 3y ago
    Simply don't use echo if you want predictable output. Use printf. https://linux.die.net/man/1/printf https://linux.die.net/man/1/printf
    • ndsipa_pomu 3y ago
      This is the way (however, the parent's use of "echo" would be fine as it's not using a variable and so won't be interpreting a dash as an extra option etc)
      • gdavisson 3y ago
        echo -n is not safe, because some versions of echo will just print "-n" as part of their output (and add a newline at the end, as usual). In fact, XSI-compliant implementations are required to do this (and the same for anything else you try to pass as an option to echo). According to the POSIX standard[1], "If the first operand is -n, or if any of the operands contain a <backslash> character, the results are implementation-defined." [1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html#tag_20_37_05 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...
        • ndsipa_pomu 3y ago
          Thanks - I wasn't aware that echo was that problematic as I target bash (usually v4 and above) from my scripts. I just tested it out with: sh /bin/echo -n "test" /bin/echo: 3: Syntax error: "(" unexpected I didn't realise until recently that printf can also replace a lot of uses of the "date" command which is helpful with logging as it avoids calling an external command for every line logged.
• franky47 3y ago
  > It's important to remember that we are not encrypting the text here. Thank you for emphasizing this. Many junior devs have been bitten by not being told early enough the difference between encryption (requires a secret to be reversed), hashing (cannot be reversed) and encoding (can always be trivially reversed). Also good to know that while the output looks random, it follows the same entropy as the input. Ie: don't base64 encode your password to make it stronger.
  • sirk390 3y ago
    I really though you were going to say "Chat-GPT detected" when quoting that message.
  • globular-toast 3y ago
    This is why computer science as a discipline matters.
  • deleted 3y ago
    [deleted]
  • tinco 3y ago
    This is a nitpick and not pertinent to anything, but base64 encoding your password could make it stronger. Password strength is not just about entropy, high entropy is simply the most effective way of making your password stronger. If your password is 100% randomly generated (as it should be) then base64 encoding it won't do anything. If however your password is engineered to be easier to remember, for example by using dictionaries or some kind of scheme that has a lower entropy, then the base64 encoding step adds a single bit of strength to your password. Meaning anyone who is brute forcing your password using a smart password cracker, has to configure that cracker to also consider base64 encoding as a scheme, basically forcing it to perform a single extra operation for every try. Anyway, useless information, you shouldn't be using password schemes like that. The horse shoe battery staple type password style should be quite sufficient I think.
    • franky47 3y ago
      One could argue that base64 having a shorter output length than its input would weaken any given password, assuming a brute force attack (not a dictionary one).
• Zamicol 3y ago
  Any time base conversion comes up, I shamelessly plug my arbitrary base converter: https://convert.zamicol.com https://convert.zamicol.com The base64 under "useful alphabets" is the "natural", iterative divide by radix, base. There's the RFC's "bucket" conversion base under extras.
• ulrischa 3y ago
  Embedding images with base64 encoding is pretty cool. I wonder why it is not more used
  • slivanes 3y ago
    I think mostly because it’s less efficient byte transfer size, roughly +25%, and also it won’t be downloaded as an image in parallel.
    • wingi 3y ago
      Gzip will fix this problem. But these ebessed imgaes are not cache-able.
      • neallindsay 3y ago
        Gzip or Brotli will help, but the result will still be bigger when base64 is in the middle.
      • Sohcahtoa82 3y ago
        The only problem with using any sort of compression in HTTP is that unless you add extra countermeasures, you open your page up to the BREACH vulnerability: https://en.wikipedia.org/wiki/BREACH https://en.wikipedia.org/wiki/BREACH
    • ulrischa 3y ago
      That is a point
  • rasz 3y ago
    Its not cool when all you know is Base64. Vivaldi uses it to store inline thumbnails in \Vivaldi\User Data\Default\Sessions Tabs_ Session_ files. A simple file containing list of 100 Tabs takes multi megabytes. They also used to store inline thumbnails in Bookmarks file, ~200 Bookmarks file was taking 10-20MB!
    • tinus_hn 3y ago
• simbyotic 3y ago
  Perhaps a bit pedantic, but would be more accurate to say that Base64 encodes binary data into a subset of ASCII characters, since ASCII has 128 code points - 95 printable characters and 33 control characters - whereas Base64 uses 64 - 65 if we include the padding - of those.
• bathwaterpizza 3y ago
  Thanks for the read, it's a very simple encoding but I never decided to find out how it works either, good to know.
• BlackFly 3y ago
  If you ever need to encode something and expect people to type it out... I recommend using https://en.wikipedia.org/wiki/Base32 https://en.wikipedia.org/wiki/Base32 instead. Nothing more frustrating than realizing that (because of bad fonts often) that I was an l or a 1 or that o was an O or was it a 0?
• deleted 3y ago
  [deleted]
• sharkski 3y ago
  A use case I used it for recently was to encase multiple scripts and even files into one script.
• 2-718-281-828 3y ago
  historically base64 was developed to make binary data printable.
  • cpach 3y ago
    I thought it was developed for sending binary files over SMTP/Usenet.
    • 2-718-281-828 3y ago
      possible. "printable" seems to be just the category but not that it was actually printed. of course some people print base64 of encrypted passwords (for offline storage) that would otherwise contain unprintable characters. usenet exists since 1979/80 [1] and base64 was first described in 1987 [2]. 1: https://en.wikipedia.org/wiki/Usenet https://en.wikipedia.org/wiki/Usenet 2: https://base64.guru/learn/what-is-base64 https://base64.guru/learn/what-is-base64
    • beej71 3y ago
      Maybe, but uuencode was there first! :)
      • kps 3y ago
        MIME picked Base64 because uuencode uses a larger character set which, while fine for its original uu (Unix-to-Unix) purposes, made it less robust to cross-platform weirdness.
    • wichert 3y ago
      uuencode was used for SMTP/Usenet. Base64 became popular as part of MIME if I remember correctly.
• beej71 3y ago
  I throw base64 in at the end when I'm teaching students about hex and binary. It's always fun to ask them to come up with the digits.
• soliton4 3y ago
  it didnt go into detail about the purpose of the = / == padding. and it also didnt show in the example how to handle data that can not be devided into groups of 6 bits without bits left over. i think i have an understanding of how to do it but it would be nice to be certain. could someone address the following 2 questions in a short and exhaustive way: - when do you use =, when do you use == and do you always add = / == or are there cases where you dont add = / == ? - how to precisely handle leftover bits. for example the string "5byte". and is there anything to consider when decoding?
  • tangent128 3y ago
    Your questions are related. For context: since a base64 character represents 6 bits, every block of three data bytes corresponds to a block of four base64 encoded characters. (83 == 24 == 64) That means it's often convenient to process base64 data 4 characters at a time. (in the same way that it's often convenient to process hexadecimal data 2 characters at a time) 1) You use = to pad the encoded string to a multiple of 4 characters, adding zero, one, or two as needed to hit the next multiple-of-4. So, "543210" becomes "543210==", "6543210" becomes "6543210=", and "76543210" doesn't need padding. (You'll never need three = for padding, since one byte of data already needs at least two base64 characters) 2) Leftover bits should just be set to zero; the decoder can see that there's not enough bits for a full byte and discard them. 3) In almost all modern cases, the padding isn't necessary, it's just convention. The Wikipedia article is pretty exhaustive: https://en.wikipedia.org/wiki/Base64 https://en.wikipedia.org/wiki/Base64
  • eatporktoo 3y ago
    from the article: "Every Base64 digit represents 6 bits of data. There are 8 bits in a byte, and the closest common multiple of 8 and 6 is 24. So 24 bits, or 3 bytes, can be represented using four 6-bit Base64 digits." So you're essentially encoding in groups of 24 bits at a time. Once the data ends, you pad out the remainder of the 24 bits with = instead of A because A represents 000000 as data. For the record, I had to read the whole thing twice to understand that too.
    • jameshart 3y ago
• p4bl0 3y ago
  A funny thing about Base64: if you iterate encoding starting from any string, a growing prefix of the result tends towards a fixed point. In Bash: $ function iter { N="$1" CMD="$2" STATE=$(cat) for i in $(seq 1 $N); do STATE=$(echo -n $STATE | $CMD) done cat <<EOF $STATE EOF } $ echo "HN" | iter 20 base64 | head -1 Vm0wd2QyUXlVWGxWV0d4V1YwZDRWMVl3WkRSV01WbDNXa1JTVjAxV2JETlhhMUpUVmpBeFYySkVU $ echo "Hello Hacker News" | iter 20 base64 | head -1 Vm0wd2QyUXlVWGxWV0d4V1YwZDRWMVl3WkRSV01WbDNXa1JTVjAxV2JETlhhMUpUVmpBeFYySkVU $ echo "Bonjour Hacker News" | iter 20 base64 | head -1 Vm0wd2QyUXlVWGxWV0d4V1YwZDRWMVl3WkRSV01WbDNXa1JTVjAxV2JETlhhMUpUVmpBeFYySkVU EDIT: I just remembered that when I found that out by pure serendipity more than 10 years ago I tweeted cryptically about it [1] and someone made a blog post on the subject which I submitted here but it didn't generate discussion [2]. Someone else posted it on Reddit /r/compsci and it generated fruitful discussion there, correcting the blog post [3]. The blog is down now but the internet archive has a copy of it [4]. [1] https://twitter.com/p4bl0/status/298900842076045312 https://twitter.com/p4bl0/status/298900842076045312 [2] https://news.ycombinator.com/item?id=5181256 https://news.ycombinator.com/item?id=5181256 [3] https://www.reddit.com/r/compsci/comments/18234a/the_base64_encoder_has_a_fixed_point/ https://www.reddit.com/r/compsci/comments/18234a/the_base64_... [4] https://web.archive.org/web/20130315082932/http://fmota.eu/blog/base64-fixed-point.html https://web.archive.org/web/20130315082932/http://fmota.eu/b...
  • deleted 3y ago
    [deleted]
  • jstanley 3y ago
    Whoa, that's really neat!
  • preciousoo 3y ago
    Aww the article linked in Reddit is dead
    • p4bl0 3y ago
      As explained in my comment above, the blog is dead, but link 4 in my comment is a backup of this article by archive.org.
• devadvance 3y ago
  Note: depending on the language, you might have to do more work to ensure that the original data is in a format that base64 encoding will support. For example, in JavaScript, that involves making sure it's a well-formed string. I did a write-up of that here: https://web.dev/articles/base64-encoding https://web.dev/articles/base64-encoding.
• bakkoting 3y ago
  There's some additional interesting details, and a surprising amount of variation in those details, once you start really digging into things. If the length of your input data isn't exactly a multiple of 3 bytes, then encoding it will use either 2 or 3 base64 characters to encode the final 1 or 2 bytes. Since each base64 character is 6 bits, this means you'll be using either 12 or 18 bits to represent 8 or 16 bits. Which means you have an extra 4 or 2 bits which don't encode anything. In the RFC, encoders are required to set those bits to 0, but decoders only "MAY" choose to reject input which does not have those set to 0. In practice, nothing rejects those by default, and as far as I know only Ruby, Rust, and Go allow you to fail on such inputs - Python has a "validate" option, but it doesn't validate those bits. The other major difference is in handling of whitespace and other non-base64 characters. A surprising number of implementations, including Python, allow arbitrary characters in the input, and silently ignore them. That's a problem if you get the alphabet wrong - for example, in Python `base64.standard_b64decode(base64.urlsafe_b64encode(b'\xFF\xFE\xFD\xFC'))` will silently give you the wrong output, rather than an error. Ouch! Another fun fact is that Ruby's base64 encoder will put linebreaks every 60 characters, which is a wild choice because no standard encoding requires lines that short except PEM, but PEM requires _exactly_ 64 characters per line. I have a writeup of some of the differences among programming languages and some JavaScript libraries here [1], because I'm working on getting a better base64 added to JS [2]. [1] https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612af59#user-content-fn-ruby2-90cdb254bd1cd57763f6d5685c8ae89e https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612a... [2] https://github.com/tc39/proposal-arraybuffer-base64 https://github.com/tc39/proposal-arraybuffer-base64
• rezmason 3y ago
  Here is my Base64 encoder shader: https://github.com/Rezmason/excel_97_egg/blob/main/glsl/base64.glsl https://github.com/Rezmason/excel_97_egg/blob/main/glsl/base... I got it down to about thirteen lines of GLSL: https://github.com/Rezmason/excel_97_egg/blob/main/glsl/base64.min.glsl https://github.com/Rezmason/excel_97_egg/blob/main/glsl/base... I use it for Cursed Mode of my side project, which renders the WebGL framebuffer to a Base64-encoded, 640x480 pixel, indexed color BMP, about 15 times per second: https://rezmason.github.io/excel_97_egg/?cursed=1 https://rezmason.github.io/excel_97_egg/?cursed=1
  • drawkbox 3y ago
    Blast from the past, the Excel easter egg. Solid.
• aydyn 3y ago
  Can someone give a plausible explanation why base64 is used instead of base85? Both are transcoded in linear time (Even base91 is possible in linear time but potentially still slower due to more difficult SIMD). People have been making the argument for over a decade that base64 is incumbent and so people stick with it due to interoperability. But base85 represents a 20% compression gain for basically free from a computational perspective. Isn't that worth switching over as a widely used standard?
  • chubot 3y ago
    Was it free when base64 was invented? CPUs were slower relative to memory in those days, so it may have made a difference. Many more programs were CPU bound, and people wrote assembly more often Also base64 is just easier to code
  • kps 3y ago
    One of the reasons for the shift from uuencoding to base64 is that the latter uses almost exclusively alphanumeric characters and consequently is more robust against cross-platform or ‘smart’ transformations or quoting or escaping issues.
  • barryrandall 3y ago
    20% better isn't enough to motivate every single producer and consumer to incur the switching costs or taking on an IPv6-scale migration process.
  • layer8 3y ago
    Base85 uses ASCII characters 33 (“!”) through 117 (“u”), which includes a number of inconvenient characters, like backslash and single and double quotes. The reason to use an encoding at all instead of just raw binary is not just to avoid nonprintable characters, but also to avoid certain printable characters with meta semantics, like string delimiters and escape characters. An alternative would be the encoding specified by RFC 1924, which uses a different, noncontiguous set of characters. It still has the drawback that dividing by 85 is a bit slower than dividing by 64 (which is just a bit shift). Last but not least, Base64 has the benefit of being easily recognizable by a human. Due to its relatively restricted character set, it doesn’t look like just line noise, but also doesn’t look like more intelligible syntax, or like hex, etc. It sits in a middle sweet spot.
• kelsey9876543 3y ago
  fun fact, gpt speaks b64 as a native language, and you can instruct it - in base64 - to only respond to you in base64, and it will comply and switch to only speaking base64
• vikrant17 3y ago
  I am still not convinced with the reason for using base64. 1. "Another common use case is when we have to store or transmit some binary data over the network that's supposed to handle text, or US-ASCII data. This ensures data remains unchanged during transport." What does it mean by network that handles text? Why should the network bother about the kind of data in the packet. If the receivers end is expecting a binary data, then why is there a need to encode it using base64. Also if data is changed during transport like "bit-flipping" or some corruption, then should't it affect the credibility of the base64 endcoded data as well. 2. "they cannot be misinterpreted by legacy computers and programs unlike characters such as <, >, \n and many others." My question here is what happens if the legacy computers interpret characters like <,, > incorrectly? If you sent a binary data, isn't that better since its just 0's and 1's and only the program that understands that binary data, will interpret?