14 ms·
I Broke Rust's Package Manager for Windows Users
- ziikutv 9y agoWhat did you end up calling the new crate? Edit: I suggest "terminated"
- filleokus 9y agoThe .toml file in the master branch on Github seem to still call it "nul": https://github.com/SSheldon/nul/blob/master/Cargo.toml https://github.com/SSheldon/nul/blob/master/Cargo.toml I can't find it on crates.io though.
- ziikutv 9y agoI guess we can say, its To be Determined or entirely scrapped.
- sasheldon 9y agoI haven't published it again because I hadn't thought of a name (and nothing published is using it, so no urgent need). I like terminated! Good suggestion :D
- hmottestad 9y agoMakes me wonder if I can make a crate called "../.." and have it overwrite some user files.
- callumjones 9y agoIt looks like the files were managed by Git (see the output where the checkout errors out), so no that won't work.
- glandium 9y agoI thought git handled the windows special file names. Am I misremembering or is it a difference (and thus a bug) in libgit2? (used by cargo, afaik)
- kibwen 9y agoI believe that git chokes by default on special file names on Windows, but I think there's a config variable that you can set to fix it. I don't know if libgit2 differs here.
- steveklabnik 9y agoRegular old git will fail; I tried to check out MINIX's source code a month ago and the clone failed since a file was named COM.
- kibwen 9y agoCrate identifiers are required to be [a-zA-Z] for the first character and [a-zA-Z0-9_-] for the rest. So, no. :P
- aisofteng 9y agoMore succinctly: [A-z]+[A-z0-9_-]
- kibwen 9y agoIf you look closely at your comment, you'll realize the hilarious HN bug that prevents me from writing it as you suggest. :P
- majewsky 9y ago[a-zA-Z][a-zA-Z0-9_-]* FTFY
- deleted 9y ago[deleted]
- wruza 9y ago[\]^_` included, or is that a special case?
- jwilk 9y ago[A-z] is not the same as [A-Za-z] . The former includes a few punctuation characters that are between "Z" and "a".
- protomyth 9y agoFor those who don't use Windows and might need this info: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
- encryptThrow32 9y agoI recommend ':?', as it will work in POSIX, but not Windows.
- akerro 9y agoThis sounds like end of Rust for next month for me...
- joshu 9y agoRemember not to name anything pr#6 for either...
- HedleyLamar 9y agoDon't they have a continuous integration system where they run the unit tests on all platforms for all checkins to master?
- steveklabnik 9y agoYes. Why do you ask?
- akerro 9y agoObviously, he wanted to know.
- pvg 9y agoIn the Mac System 7-ish days, people used to earnestly warn each other not to name a file '.Sony' (a special name reserved for the floppy driver) as it supposedly trashed your HD. Although I've never heard of anyone reproducing it.
- Xylakant 9y agoTrying to name a folder "trash" led to the error message "The name 'trash' is reserved for the operating system."
- db48x 9y agoNow's your chance: https://archive.org/details/mac_MacOS_7.0.1_compilation https://archive.org/details/mac_MacOS_7.0.1_compilation I'd try it myself, but I've only got my phone with me.
- pvg 9y ago7.0.1 might be a little late, at least, for the supposed catastrophic results. It doesn't like the file at all but nothing dreadful seems to happen.
- db48x 9y agoI tried it in that and System 6 (https://archive.org/details/mac_MacOS_6.0.8); https://archive.org/details/mac_MacOS_6.0.8); System 6 actually didn't care at all. An interesting bug. I haven't actually done so, but earlier versions are available if you know where to look (https://archive.org/details/mac_Paint_2 https://archive.org/details/mac_Paint_2).
- tatterdemalion 9y agoAs the blog post mentioned, we solve the issue by deleting the crate from the package repository and reserving these problematic names. The incident lasted about 2 and a half hours. Crate names have to be one or more valid idents connected by hyphens, so no other clever names like `/home` would be possible to upload. We already had some crate names reserved and we just needed to add these to the list.
- kibwen 9y ago> The incident lasted about 2 and a half hours. And because it was a weekend, much of that time involved me trying to figure out who had the proper credentials for crates.io, and then texting those people until one of them responded. :)
- derefr 9y agoReserving just the crate names won't cover your bases, though, no? I'm not clear on what exists as part of a crate—but if there's any user control over the filenames of the contents of the crate (e.g. if the crate's source code is in there) then any crate might contain a file named e.g. "nul.rs", triggering the same problem.
- kibwen 9y agoI think you're misunderstanding the problem described in the OP. When you build a project via cargo using the default settings, it fetches the git repository at https://github.com/rust-lang/crates.io-index https://github.com/rust-lang/crates.io-index to enable it to resolve dependencies locally. This git repository contains metadata for each library on crates.io, where the metadata for a given library is located in a file with the same name as the name of that library. When the OP uploaded a library whose name was an illegal filename on Windows, git unexpectedly choked when updating the local crate index repo, impacting all Windows users. It sounds like the concern you're describing is a different matter. It's likely true that if the source of a crate contains a file named "nul.rs", cargo on Windows will fail if it attempts to git-fetch the source (unless you're using Linux Subsystem for Windows, anyway). While this would indeed be a problem, it would only affect users who elect to use specific libraries, rather than serving as a denial-of-service for every Rust user on Windows.
- Strilanc 9y agoUrgh, this "nul" filename / reserved filename bug is probably in a lot of software.
- tannhaeuser 9y agoEvery MS-DOS programmer of old knows about nul, con, and the other reserved names. Those might come from CP/M actually (so are even older), and Atari TOS had them as well I believe.
- Sir_Cmpwn 9y agoSounds more like a problem with stupid Windows design choices than with anything you did.
- alkonaut 9y agoBecause there aren't any of those in posix...
- Sir_Cmpwn 9y agoAre you kidding? POSIX is perfect /s
- dagw 9y agoWindows is, for better or worse, fiercely proud of its backwards compatibility. So it's not so much a stupid Windows design choice as a 'stupid' DOS 1.0 design choice (and not even so much a choice as simply a quirk of how the DOS 1.0 file system worked) that Windows doesn't want to break backwards comparability with.
- wand3r 9y agoI agree with parent that it's a bit crazy; but I wouldn't be as critical. to your point; presumably even if they dropped DOS support something between DOS and now likely relies on that. It's a fine line.
- captn3m0 9y agoWhat I don't understand is why cargo fetches the entire crate list and create a directory for every crate (even if you never install it). Why not just have a single file with the entire list? The issue mentions they use a trie, but why use the filesystem as the trie store? Why not have a single file?
- nerdponx 9y agoAFAIK this is how the BSD Ports system works too.
- kibwen 9y agoThe original authors of cargo, wycats and carllerche, aren't around today to ask (it's a weekend!) though IRC attempted to answer regardless: <foo> to keep the number of files in a single directory down <foo> tools become unhappy with hundreds of thousands+ of things in a single dir <foo> as do filesytems <bar> why not just a flat file <bar> or sqlite or whatever <qux> right now it uses git's deduplication feature <qux> aka, when downloading updates you only download the objects that changed <qux> but it mostly works on a per file basis <qux> so git hashes each file and if the hash didnt change, it doesnt download an update <qux> but if it did, it treats it as completely new file, even if its just a little change
- kibwen 9y agoUpdate: <wycats> Because of this: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 <wycats> I ran some scenarios against huge repos when I first worked on cargo <wycats> Trying to minimize the cost of operations <wycats> I landed on the current strategy, and GitHub in the above thread more or less endorsed what we were already doing at that time <wycats> Also see https://github.com/rust-lang/cargo/issues/2452
- comex 9y agoIt's still fundamentally a waste of disk space. On my system, as of a minute ago, ~/.cargo/registry/index took up about 200MB for three different checkouts (for some reason). After deleting that and running `cargo update`, only one of them is recreated, 104MB. Out of that, 57MB is the JSON files and 47MB is git history. But if I just concatenate all the JSON files, the result is only 33MB, and after gzipping, 3MB. Hypothetically, a non-GitHub-based Cargo could store only those 3MB (using binary deltas to avoid resending it on every update), or even 0MB if it just relied on the server to resolve dependencies.
- bluejekyll 9y agoThis is a great example of something else about software. As software grows in usage and use cases, it starts bumping up against edge conditions which need to be handled for various reasons. Cargo now is becoming stronger and more stable because of bugs like this being discovered. All software goes through this growth cycle. It's great to see these things worked out in the various projects that support Rust. There is another point here though; anytime the question comes up to just rewrite a piece of software, throw out all the technical debt, it's not as straightforward as it seems. Remember, together with that technical debt lies a lot of valuable learnings written into the code. I haven't worked on Windows directly in years, but I never knew that NUL was a reserved word as a file. I would, and probably still will make this mistake in the future. Which makes me wonder, has anyone written a file name validation crate that guarantees that you're not writing to any reserved words on a filesystem of the host OS? A quick search of crate.io doesn't turn anything up.
- pjc50 9y ago> I never knew that NUL was a reserved word as a file. I would, and probably still will make this mistake in the future While we're here: NUL, COM<n>, LPT<n> and AUX are reserved.
- akavel 9y agoAnd CON, as Macha mentions in a sister comment. An idiom I remember from old times in DOS, for quickly writing some contents into a file - equivalent to `cat > myfile.txt` on Linux: COPY CON MYFILE.TXT
- ConfucianNardin 9y agoYou can also do type con > myfile.txt
- cesarb 9y ago> While we're here: NUL, COM<n>, LPT<n> and AUX are reserved. Worse: they're reserved with any extension. Have a file in your repository called "aux.rs"? It will cause problems on Windows.
- alkonaut 9y agoIt's very tricky to do cross platform file handling stuff, and only the most mature projects have ironed out this. Just look at your pet project and see if it handles - Windows and unix line breaks in text files - Windows and unix path separators - BOM and non-BOM marked files if parsing UTF - Forbidden filenames such as in this article By "handling" I mean it should accept or fail nicely on unexpected input - e.g. say that line breaks should be unix style, or paths should be backslashes etc. Very few projects actually do this well. Even fewer will do even more complex things like handling too long paths with nice error messages etc.
- tannhaeuser 9y agoSince Windows 10 now comes with an official Linux subsystem, why not just use POSIX APIs and conventions everywhere, and not bother with Windows-specific code if possible?
- untog 9y agoFor one, because the Linux subsystem is an optional install. If you're making anything user-facing you can't rely on it being there - it's really a tool for developers, not end-users.
- alkonaut 9y agoDepends on what type of application you are making. For a library that can be used in a "real" graphical Windows application, you can't make a posix type shortcut. I think "if possible" is (at least still) very rarely the case that it is.
- yrashk 9y agoWouldn't it make sense for Cargo not to use crate names in file names, and use hexadecimally encoded hashes instead?
- brianberns 9y agoYes, or some other identifier that's unique to that crate. Assuming that the crate name is also a valid file name seems risky.
- nomercy400 9y agoOr have a prefix. Once cost me exam points trying to optimize a prefix away. The examinators were right.
- stirner 9y agoYou could just hex encode the crate name. No need to hash it too.
- garaetjjte 9y agoOther magic aliases include CON, PRN, AUX, COM1-9 and LPT1-9. They are aliased to respective devices in Win32 namespace "\\.\". COMs and LPTs above 9 don't have aliases in global namespace and must be accessed explictly in Win32 namespace, eg. "\\.\COM10" (which itself is symlink to NT native "\Device\Serial9") In fact, it is possible to create files named NUL, COM1, etc. using \\?\ (eg. "\\?\C:\NUL" is valid path) prefix which disables parsing arcane Win32 magic files. Unfortunately these files are causing strange behaviour in applications that don't use that prefix, Explorer included. source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#namespaces https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
- monochromatic 9y agoI still remember using "copy con foo.txt" and ending with ctrl-z to quickly create a file. It was years before I understood how that actually worked.
- slobotron 9y agoThere was a bug in Windows 95 (98 too?) where if you tried to open 'nul\nul' or 'con\con' etc, it would BSOD instantly. Provided lots of drive-by fun in computer labs... (got really good at typing win+r con\con)
- tonyarkles 9y agoFor more fun, you could also target other machines with SMB shares. \\thevictim\foo\nul\nul would BSOD that machine. Good times.
- cesarb 9y agoIIRC, you could also reference it in a HTML page, so the whole computer would crash when that page was viewed.
- johnfn 9y agoThis is like the 90s era "undefined is not a function." null is not a problem, but null.null, on the other hand...
- lsiebert 9y agoHmm... I feel like someone should stick the reserved names into a json somewhere for easy reference for the next package manager.
- roryisok 9y agoI was working on a video project for a local comics convention, and named the project file "con.proj". That file hung around until I upgraded my hard drive because no file manager could delete it.
- toabi 9y agoI tried `npm install nul` on my win7 VM and it created a folder called nul which I can't get rid of ¬_¬
- msimpson 9y agoWhile I know nothing of Rust, Diesel, or CrateDB, I do know that Windows uses a case-insensitive file system and this fix doesn't seem to take that into consideration. However, the author of the fix does note: > I believe crates.io's namespace is case insensitive let me know if that's wrong Someone should probably validate that.
- zyx321 9y agoNot quite. Windows uses a case insensitive API on top of a case sensitive file system. FUN FACT: As of 2017, Windows 10 is (partially) binary-compatible to Ubuntu Linux. Any application that was originally compiled for Linux will still be case sensitive when running on Windows 10.
- msimpson 9y agoRight. I forgot NTFS is in fact case sensitive. Thanks.