2 ms·

> Here’s the alphabet, in encoding order: > ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'()+,-;=@[]^_`{}~ Yeah I definitely don't want

by abound 13d ago

> Here’s the alphabet, in encoding order: > ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'()+,-;=@[]^_`{}~ Yeah I definitely don't want any of those last ~22 characters in file names if I need to do anything with them from a shell

brookst 13d ago
What you don’t want a file named `reboot`?
dgacmu 13d ago
Yeah, I'm sympathetic to their desire for compression and efficiency but base64URL is way more shell-friendly. I quite like base58 for human-presented identifiers also. I tend to use it instead of hex for having ~8-16 character IDs of things.
- jerf 13d ago
  Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. It may intuitively feel like having 26 more characters to encode with would be hugely more efficient but for the extra risk of what all those characters get us we're only getting just over .5 more bits per character. I need to be either pretty desperate for efficiency, rare in 2026, or very, very sure that those things are only going places where those characters aren't risky or inconvenient, which is pretty difficult to guarantee since even if it is in fact true today who's to say it will be tomorrow, for this to be worth the risk of the extra characters. Even just the extra characters in base64 are often annoying and we already have multiple common base64 standards because even trying to find just two more characters to fill out a-zA-Z0-9 is hard.
  - dspillett 13d ago
    > Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. Also on efficiency, you are trading off code complexity for a little extra storage: base64 is a nice round 6 bits per character meaning every three bytes encoded is four output. Neither base84 nor base58 align on convenient bit boundaries like that so choosing the output character is more faf. Padding could be more complicated too. Now if you are looking at a per-character limit where the characters are multi-byte (say SSMS shortcuts which are limited to 32767 UCS2 characters) then some form of base4096 (12 bits per character so three 8-bit bytes to two output characters) might be useful. Yes, I have done this: putting a long analysis proc (a replacement for sp_help & friends) into a “shortcut” I was getting close to the 32K-char limit so compressed and base64ed the code and included an unpacker. This was more than enough to deal with the problem (TBH, just stripping comments would have done!) and simplified things in some ways as I no longer needed to escape quotes and such, but I went one step further played with writing a B4096 encoder because I like playing with that sort of silliness. I went with 4096 due to aligning nicely with 4-bit boundaries, and finding 4096 useable characters (avoiding control characters, undefined codes, and other unprintables) is easy. If the limit is actually 32767 or any Unicode characters (including those not in the UCS2 or UTF16 base plane set) then you could perhaps get even more daft though I'm pretty sure it is just 16-bit characters and not full Unicode.
- Asooka 13d ago
  For efficiency, I would prefer to store files with a random short filename that's just letters and digits and store the original name in a separate database.
Dwedit 13d ago
Even just "-" in a filename is problematic, since you can name actual files with "--" at the beginning, and the name could match a switch.
- badc0ffee 13d ago
  Typically you can use the -- arg to indicate that everything to the right should not be interpreted as a switch.
  - PhilipRoman 13d ago
    and for programs that don't respect this convention ./--file
  - IsTom 13d ago
    Yes, but typically people are not doing this on every command and if you're globbing files it'll get used as flag.
orra 13d ago
Moreover, I'd be nervous about the fact that filenames on Windows tend to be case insensitive.
- LoganDark 13d ago
  macOS has a case-insensitive filesystem by default too.
  - tobyhinloopen 13d ago
    And don’t you dare changing it unless you like random bugs
    - applfanboysbgon 13d ago
      Sell me on this. Why would you want to change it? What good can actually come of allowing foo and Foo to coexist in the same directory?
      - LoganDark 13d ago
        It can be nice to checkout some Git repositories that have this. It can also be nice to catch miscapitalizations of filename imports when those would fail on Linux, etc. One of my past workplaces actually had that problem, they mis-cased imports and the product failed to build on Linux. Nobody caught it because they all used macOS or Windows, I was the first to try WSL (this was before I switched back to Mac) You would be asking for a world of hurt to put your root filesystem in a case-sensitive volume though. All sorts of software and applications have silently relied on case-insensitivity for decades. You can sometimes fix it on a case-by-case basis, but sometimes you can't, and it's also annoying for it to happen in the first place. Better to have a separate case-sensitive volume just for the stuff you want to be case-sensitive.
      - retrac 13d ago
        Practically? Quite a few codebases from *nix systems rely on the case sensitivity. To do a successful git clone you need a file system that handles both Makefile and makefile, or whatever. I ran into it more than once back in the day when I used Mac OS X as my primary OS. More generally -- case sensitivity is a conceptual nightmare in the Unicode era. Should Cyrillic or Greek be case-insensitive as well? Etc. Do you really want the full complexity of Unicode string handling in your file system? I would suggest treating file names like raw bytes. On modern Linux, anything but NUL is valid.
mmooss 13d ago
I learned, and thought everyone else learned, that the safe characters for POSIX filenames are: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_- That is, the only safe non-alphanumeric characters are dash "-" and underscore "_". Period "." is also considered 'safe' but is tricky for obvious reasons and [IMHO, for the most reliability] best avoided except where necessary. Authoritative source: POSIX.1-2024 standard, sec. 3.265 Portable Filename Character Set https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_265 https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...
- edoceo 13d ago
  Sometimes, if your file starts with a dash things can be tricky. Don't make any mistakes removing a file named '-fr'
raverbashing 13d ago
100% this (Also for anything that needs to work on a shell script or whatever) I roll my eyes every time some "super secure secret generator" gives me a password with &^%#$ that I need to use on a shell script. Thanks for making my life harder I guess "oh but you just need to escape them" Yes please be my guest trying to deal with character escape whack-a-mole
- noAnswer 13d ago
  It makes me sad that shells, operating systems and languages have basically made zero progress / had zero innovation in the last 60 years. "Escaping" should be something some graybeard is mumbling about. Yet here we are in 2026, still having to be on the lookout for a wild /../ Even PowerShell is boilerplate, boilerplate galore.
  - 1718627440 13d ago
    It's just a fundamental problem of encoding the boundary between different (meta-)languages in-band. It will always be a problem, until it's possible to implement the separation out-band, which makes the respective language no longer self-contained, as it needs to be implemented in the layer above. This occurs now with LLMs too. It's a fundamental problem, of the problem description, not the implementation.
    - noAnswer 13d ago
      Yeah, the innovation would be to not make everything in-band. Which implies proper file formats for source code for example. I know, that is blasphemy against 7bit ASCII. It's like the telephone networks of old. They eventually realized that that is not so good. The computer world is still in its analog telephone phase. ;-)
      - raverbashing 12d ago
        Yes I want to scream everytime a tool gives me crap for "no newline at end of file" or other similar BS I don't care. I never cared. And every sane tool knows how to deal with it Shut up and stop thinking we're still in the 70s
hnlmorg 13d ago
I don’t see how they’re any worse than spaces in file names, which is an edge case you need to cater for already anyway.
- eklitzke 13d ago
  You only "need" to do this if you are handling arbitrary untrusted filenames. It's pretty common for people to write shell scripts that are only used for handling files they create that don't have unusual characters, and these shell scripts work just fine.
  - hnlmorg 13d ago
    Until they don’t. Maybe the old team leave or a new requirement comes in.
- hn993302 13d ago
  I don't want spaces either. Of course your scripts should never assume any of this, and this only matters when doing stuff manually (or ai-yolo) in the shell.
- crabbone 13d ago
  They also have single quote, dollar and backtick... this is a lot worse than just whitespace. The problem with whitespace surface when the programmer made no effort to escape the string used as a file name. This adds a whole new vector of problems that happen when the programmer tried to escape the string, but it didn't work. It was too long ago, so, I've forgotten the names, but I remember the saga with PHP escaping (and failing) SQL queries (some sort of sql_real_now_honest_real_escape_string() function or something like that).
  - hnlmorg 13d ago
    It depends on the shell you use and how your file names are handled. Bash is a clusterfuck waiting to happen when it comes to file names, but not all shells would be at risk here. For example the shell I maintain would be fine because file names are tokenised like how variables are tokenised in SQL strings when using parametrised queries (to use your SQL/PHP example).
louky 13d ago
Same! Although I still have muscle-memory replacing spaces with _ or - when creating/saving files on any OS. Hasn't done me wrong lo these past 40 years. As for the others tailing the ASCII, no thanks.
- euroderf 13d ago
  Why do file dialogs have a button for this ?
hn993302 13d ago
If I'm choosing a filename, it's lowercase alphanumeric plus underscore. Usually not even the underscore.
dwedge 12d ago
One of my apps uses base64 filenames just to avoid any issue with shell (the filename comes from POST and could be anything). Not the usecase of base84 but one that I couldn't use base84 for.

by abound 13d ago

brookst 13d ago

What you don’t want a file named `reboot`?

dgacmu 13d ago

Yeah, I'm sympathetic to their desire for compression and efficiency but base64URL is way more shell-friendly. I quite like base58 for human-presented identifiers also. I tend to use it instead of hex for having ~8-16 character IDs of things.

jerf 13d ago
Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. It may intuitively feel like having 26 more characters to encode with would be hugely more efficient but for the extra risk of what all those characters get us we're only getting just over .5 more bits per character. I need to be either pretty desperate for efficiency, rare in 2026, or very, very sure that those things are only going places where those characters aren't risky or inconvenient, which is pretty difficult to guarantee since even if it is in fact true today who's to say it will be tomorrow, for this to be worth the risk of the extra characters. Even just the extra characters in base64 are often annoying and we already have multiple common base64 standards because even trying to find just two more characters to fill out a-zA-Z0-9 is hard.
- dspillett 13d ago
  > Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. Also on efficiency, you are trading off code complexity for a little extra storage: base64 is a nice round 6 bits per character meaning every three bytes encoded is four output. Neither base84 nor base58 align on convenient bit boundaries like that so choosing the output character is more faf. Padding could be more complicated too. Now if you are looking at a per-character limit where the characters are multi-byte (say SSMS shortcuts which are limited to 32767 UCS2 characters) then some form of base4096 (12 bits per character so three 8-bit bytes to two output characters) might be useful. Yes, I have done this: putting a long analysis proc (a replacement for sp_help & friends) into a “shortcut” I was getting close to the 32K-char limit so compressed and base64ed the code and included an unpacker. This was more than enough to deal with the problem (TBH, just stripping comments would have done!) and simplified things in some ways as I no longer needed to escape quotes and such, but I went one step further played with writing a B4096 encoder because I like playing with that sort of silliness. I went with 4096 due to aligning nicely with 4-bit boundaries, and finding 4096 useable characters (avoiding control characters, undefined codes, and other unprintables) is easy. If the limit is actually 32767 or any Unicode characters (including those not in the UCS2 or UTF16 base plane set) then you could perhaps get even more daft though I'm pretty sure it is just 16-bit characters and not full Unicode.
Asooka 13d ago
For efficiency, I would prefer to store files with a random short filename that's just letters and digits and store the original name in a separate database.

Dwedit 13d ago

Even just "-" in a filename is problematic, since you can name actual files with "--" at the beginning, and the name could match a switch.

badc0ffee 13d ago
Typically you can use the -- arg to indicate that everything to the right should not be interpreted as a switch.
- PhilipRoman 13d ago
  and for programs that don't respect this convention ./--file
- IsTom 13d ago
  Yes, but typically people are not doing this on every command and if you're globbing files it'll get used as flag.

orra 13d ago

Moreover, I'd be nervous about the fact that filenames on Windows tend to be case insensitive.

LoganDark 13d ago
macOS has a case-insensitive filesystem by default too.
- tobyhinloopen 13d ago
  And don’t you dare changing it unless you like random bugs
  - applfanboysbgon 13d ago
    Sell me on this. Why would you want to change it? What good can actually come of allowing foo and Foo to coexist in the same directory?
    - LoganDark 13d ago
      It can be nice to checkout some Git repositories that have this. It can also be nice to catch miscapitalizations of filename imports when those would fail on Linux, etc. One of my past workplaces actually had that problem, they mis-cased imports and the product failed to build on Linux. Nobody caught it because they all used macOS or Windows, I was the first to try WSL (this was before I switched back to Mac) You would be asking for a world of hurt to put your root filesystem in a case-sensitive volume though. All sorts of software and applications have silently relied on case-insensitivity for decades. You can sometimes fix it on a case-by-case basis, but sometimes you can't, and it's also annoying for it to happen in the first place. Better to have a separate case-sensitive volume just for the stuff you want to be case-sensitive.
    - retrac 13d ago
      Practically? Quite a few codebases from *nix systems rely on the case sensitivity. To do a successful git clone you need a file system that handles both Makefile and makefile, or whatever. I ran into it more than once back in the day when I used Mac OS X as my primary OS. More generally -- case sensitivity is a conceptual nightmare in the Unicode era. Should Cyrillic or Greek be case-insensitive as well? Etc. Do you really want the full complexity of Unicode string handling in your file system? I would suggest treating file names like raw bytes. On modern Linux, anything but NUL is valid.

mmooss 13d ago

I learned, and thought everyone else learned, that the safe characters for POSIX filenames are: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_- That is, the only safe non-alphanumeric characters are dash "-" and underscore "_". Period "." is also considered 'safe' but is tricky for obvious reasons and [IMHO, for the most reliability] best avoided except where necessary. Authoritative source: POSIX.1-2024 standard, sec. 3.265 Portable Filename Character Set https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_265 https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...

edoceo 13d ago
Sometimes, if your file starts with a dash things can be tricky. Don't make any mistakes removing a file named '-fr'

raverbashing 13d ago

100% this (Also for anything that needs to work on a shell script or whatever) I roll my eyes every time some "super secure secret generator" gives me a password with &^%#$ that I need to use on a shell script. Thanks for making my life harder I guess "oh but you just need to escape them" Yes please be my guest trying to deal with character escape whack-a-mole

noAnswer 13d ago
It makes me sad that shells, operating systems and languages have basically made zero progress / had zero innovation in the last 60 years. "Escaping" should be something some graybeard is mumbling about. Yet here we are in 2026, still having to be on the lookout for a wild /../ Even PowerShell is boilerplate, boilerplate galore.
- 1718627440 13d ago
  It's just a fundamental problem of encoding the boundary between different (meta-)languages in-band. It will always be a problem, until it's possible to implement the separation out-band, which makes the respective language no longer self-contained, as it needs to be implemented in the layer above. This occurs now with LLMs too. It's a fundamental problem, of the problem description, not the implementation.
  - noAnswer 13d ago
    Yeah, the innovation would be to not make everything in-band. Which implies proper file formats for source code for example. I know, that is blasphemy against 7bit ASCII. It's like the telephone networks of old. They eventually realized that that is not so good. The computer world is still in its analog telephone phase. ;-)
    - raverbashing 12d ago
      Yes I want to scream everytime a tool gives me crap for "no newline at end of file" or other similar BS I don't care. I never cared. And every sane tool knows how to deal with it Shut up and stop thinking we're still in the 70s

hnlmorg 13d ago

I don’t see how they’re any worse than spaces in file names, which is an edge case you need to cater for already anyway.

eklitzke 13d ago
You only "need" to do this if you are handling arbitrary untrusted filenames. It's pretty common for people to write shell scripts that are only used for handling files they create that don't have unusual characters, and these shell scripts work just fine.
- hnlmorg 13d ago
  Until they don’t. Maybe the old team leave or a new requirement comes in.
hn993302 13d ago
I don't want spaces either. Of course your scripts should never assume any of this, and this only matters when doing stuff manually (or ai-yolo) in the shell.
crabbone 13d ago
They also have single quote, dollar and backtick... this is a lot worse than just whitespace. The problem with whitespace surface when the programmer made no effort to escape the string used as a file name. This adds a whole new vector of problems that happen when the programmer tried to escape the string, but it didn't work. It was too long ago, so, I've forgotten the names, but I remember the saga with PHP escaping (and failing) SQL queries (some sort of sql_real_now_honest_real_escape_string() function or something like that).
- hnlmorg 13d ago
  It depends on the shell you use and how your file names are handled. Bash is a clusterfuck waiting to happen when it comes to file names, but not all shells would be at risk here. For example the shell I maintain would be fine because file names are tokenised like how variables are tokenised in SQL strings when using parametrised queries (to use your SQL/PHP example).

louky 13d ago

Same! Although I still have muscle-memory replacing spaces with _ or - when creating/saving files on any OS. Hasn't done me wrong lo these past 40 years. As for the others tailing the ASCII, no thanks.

euroderf 13d ago
Why do file dialogs have a button for this ?

hn993302 13d ago

If I'm choosing a filename, it's lowercase alphanumeric plus underscore. Usually not even the underscore.

dwedge 12d ago

One of my apps uses base64 filenames just to avoid any issue with shell (the filename comes from POST and could be anything). Not the usecase of base84 but one that I couldn't use base84 for.