6 ms·
I invented a “dash encoding”, a new encoding scheme for URL paths
- dataflow 5y agoMaybe a dumb question, but would a backslash have worked?
- kragen 5y agoIn summary, Simon presents this bijective map, though this omits the motivation: s.replace("-", "--").replace(".", "-.").replace("/", "-/") I think the encoding would be more broadly applicable if you replaced the special characters with sequences of nonspecial characters. For example: s.replace("-", "--").replace(".", "-d").replace("/", "-s") s.replace("-", "--").replace(".", "-*").replace("/", "-,") Then, for example, relative URLs don't break, and you can use it to generate filenames that are subject to Apache content-negotiation. That's outside the scope of Simon's intended use, of course, which is Datasette, which generates all its pages dynamically rather than storing them in the firesystem. Interestingly, though Simon doesn't call this out, I think both his original scheme and my -d-s version above preserve lexicographical ordering on strings. My -*-, scheme doesn't (-* sorts before -- instead of after it) but I think this version fixes that: s.replace("-", "--").replace(".", "-@").replace("/", "-|") These non-alphabet-using schemes preserve homomorphism for alphabetic translations like .lower(), .upper(), and .title(), which is to say that, for any f from the set lambda s: s.lower(), lambda s: s.upper(), lambda s: s.title(), there exists a g such that dash_encode(f(s)) == g(dash_encode(s)) for any string. (And, as it happens, g == f, but the definition of homomorphism doesn't require that.) Simon wanted to "avoid changing the string at all if possible," but I think that's sort of a bad idea; it means that bugs that apply either the forward or inverse transformation the wrong number of times will often slip through initial testing. If we weaken that to "keep the name in the URL as close to readable as possible," while adding the opposite requirement, "ensure that no string maps to itself," we could use something like this: '-' + s.replace("-", "--").replace(".", "-@").replace("/", "-|") Any character would work for the prefix, and if you're using the generated names as Unix filenames, maybe ',', '=', or '_' would also be minimally obtrusive visually and less likely to result in problems in the shell. I hadn't seen Debuggex before. It looks pretty nice.
- lilyball 5y agoDownloading the resulting files is a really strong argument against the original "dash encoding" as proposed, because if I try to download the example /legislators/legislator_terms-/1.json path I'll end up with a file named `1.json` on my filesystem even though I'm download the file corresponding to the whole `legislator_terms/1` table. Or perhaps if I fetch a URL for the table `foo.json` and I don't add my own extension, I'll be viewing a URL ending in the component `foo-.json` even though it's not JSON data. Percent-encoding really would be the right solution here if only middleware would leave it alone instead of decoding and re-encoding it. In the absence of that, whatever encoding scheme is used must ensure the results do not leave any slashes or periods in the resulting string. As you've pointed out, other encodings (using the base dash character) can still preserve lexicographic ordering so there seems to be no reason to retain the actual period and slash characters.
- simonw 5y agoThis is an excellent point that I had not considered, thanks! I'm actually leaning towards a slightly different scheme now that would address this, where I use regular percentage encoding but with the hyphen character instead of the percentage character. I'll write an update if I decide to go with that approach.
- kragen 5y agoI look forward to reading it! Your blog is always a pleasure.
- lilyball 5y agoAre you intentionally trying to preserve lexicographic ordering? If so, doing percent-encoding-but-with-dash is reasonable and pretty easy to explain. If you don't care about lexicographic ordering then I would recommend picking a different character than dash. Even if Datasette's particular set of tables don't use dash in their name, if you're hoping other people will use this same scheme then using a more esoteric character might be nice. Using &/$/! would still preserve ordering relative to alphabetic characters and are not commonly found in URL paths.
- olliej 5y agoThe tldr is there are times you want the actual url that the client requested, but many tools helpfully unescape the urls, and it seems like by spec (api? Actual http?) Apache cannot provide the encoded url, so presumably it isn’t unique there due to people trying to be compatible. The solution was just a different escaping scheme that other apps in the stack don’t recognize and so can’t be “helpful” with, which honestly seems like a reasonable solution given the constraints
- junon 5y agoSeems like a deficiency in web servers, not a deficiency in HTTP or URL encoding. But yeah, neat solution given the constraints.
- dwaite 5y agoThe author says they could not use percentage encoding for the period. There is no such restriction in URLs; you can use percentage encoding freely. The interpretation is not always consistent, however - there are patterns such as using %80-%FF for UTF-8 encoded scalars these days, but assuming that in a broadly used URL handler library will eventually bite you.
- simonw 5y agoI tried using percentage encoding for a period and found that when I copied and pasted the URL from my browser it came back as a . character - and since easy copy and pasting is a requirement for me that ruled it out. I should add that to the post!
- TazeTSchnitzel 5y agoWouldn't it also need to encode % specially, in case it is part of the original key?
- simonw 5y agoYes, I now think so - and "-%" won't work for that because the aforementioned badly behaving proxies could mangle that in certain circumstances. I'm looking at an alternative solution to address that now. Will update once I settle on that.
- time4tea 5y agoYes, unfortunately WSGI is "broken by specification" in that all path parts are decoded before passing to the underlying application. This doesn't crop up that often, but some schemes are completely broken by it, for example DOI, which can have parts like "10.1002/0470841559.ch1", and normalising the whole uri in one go will break. Path parts need to be normalised individually. Edit: typo, replace 'design' with specification. Its not intended, probably, but a conforming implementation must implement something that cannot work with certain paths.
- ianbicking 5y agoI was involved with WSGI back when this was being decided... it was definitely a known problem, but a particularly good solution never revealed itself. The CGI variables it was based on had decoded semantics so it couldn't be redefined in place, we'd need some other names. Some gateways might have already decoded so recreating an accurate original URL might be hard in some deployments. There's another common key that includes the unencoded complete URL but WSGI had a split URL representing previous URL-prefix based dispatching (which came from CGI) and so there was some ambiguity there if you tried to make use of it. Two variables representing the same thing is a recipe for confusion and possible security problems, even worse when the two things have such subtly different quoting semantics. So it never really got fixed. Something like this is there to solve not just errors because of gateways doing decoding, but also browsers and email clients and everything else that might break a URL.
- alex_duf 5y agoI think the `.json` can be solved by looking at the `Accept` header rather than switching the URL
- ryanbrunner 5y agoFormat via extension is definitely more convenient for debugging though, and there's cases where you want to allow direct browser navigation to a particular format in which case you won't be capable of modifying the Accept header (one I use pretty often is sending users to a .csv version of a page to download the contents.
- simonw 5y agoI was planning to implement that a while ago, but I ran into a MASSIVE blocker on that. It turns out Cloudflare - the CDN I use for a bunch of my projects (and often recommend to others) - ignores the "Vary: Accept" HTTP header in most cases! This means you can't safely put a URL that varies on Accept behind a Cloudflare cache - because there's a risk that the cached copy will be stored in an unexpected format, in a way that affects future traffic. I'm really annoyed about this. Notes on that here: https://github.com/simonw/datasette/issues/1534 https://github.com/simonw/datasette/issues/1534
- zzo38computer 5y agoIt could use the Accept header if there is no extension. For example, if it ends with ".html" then it will display the HTML, if it ends with ".json" then the JSON, and if neither then use the Accept header. Having an extension to override it is useful; when not explicitly specified then it could use the Accept header to do it instead. (Another comment mentions that Cloudflare doesn't work with it, but Cloudflare has a lot of undesirable features anyways, so I do not use it.) Another possibility might be <link rel="data">, in case you want to access data (which could have multiple formats (e.g. CSV, JSON, etc), so could occur more than once) without having the URL already or having to run the web application to do everything.
- IshKebab 5y agoSeems like old school query parameters would have avoided this problem. Interesting!