17 ms·
I Broke Rust's Package Manager for Windows Users
- ziikutv 10y agoWhat did you end up calling the new crate? Edit: I suggest "terminated"
- filleokus 10y agoThe .toml file in the master branch on Github seem to still call it "nul": https://github.com/SSheldon/nul/blob/master/Cargo.toml https://github.com/SSheldon/nul/blob/master/Cargo.toml I can't find it on crates.io though.
- ziikutv 10y agoI guess we can say, its To be Determined or entirely scrapped.
- sasheldon 10y agoI haven't published it again because I hadn't thought of a name (and nothing published is using it, so no urgent need). I like terminated! Good suggestion :D
- hmottestad 10y agoMakes me wonder if I can make a crate called "../.." and have it overwrite some user files.
- callumjones 10y agoIt looks like the files were managed by Git (see the output where the checkout errors out), so no that won't work.
- glandium 10y agoI thought git handled the windows special file names. Am I misremembering or is it a difference (and thus a bug) in libgit2? (used by cargo, afaik)
- kibwen 10y agoI believe that git chokes by default on special file names on Windows, but I think there's a config variable that you can set to fix it. I don't know if libgit2 differs here.
- steveklabnik 10y agoRegular old git will fail; I tried to check out MINIX's source code a month ago and the clone failed since a file was named COM.
- kibwen 10y agoCrate identifiers are required to be [a-zA-Z] for the first character and [a-zA-Z0-9_-] for the rest. So, no. :P
- aisofteng 10y agoMore succinctly: [A-z]+[A-z0-9_-]
- kibwen 10y agoIf you look closely at your comment, you'll realize the hilarious HN bug that prevents me from writing it as you suggest. :P
- majewsky 10y ago[a-zA-Z][a-zA-Z0-9_-]* FTFY
- deleted 10y ago[deleted]
- wruza 10y ago[\]^_` included, or is that a special case?
- jwilk 10y ago[A-z] is not the same as [A-Za-z] . The former includes a few punctuation characters that are between "Z" and "a".
- protomyth 10y agoFor those who don't use Windows and might need this info: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
- encryptThrow32 10y agoI recommend ':?', as it will work in POSIX, but not Windows.
- akerro 10y agoThis sounds like end of Rust for next month for me...
- joshu 10y agoRemember not to name anything pr#6 for either...
- HedleyLamar 10y agoDon't they have a continuous integration system where they run the unit tests on all platforms for all checkins to master?
- steveklabnik 10y agoYes. Why do you ask?
- akerro 10y agoObviously, he wanted to know.
- pvg 10y agoIn the Mac System 7-ish days, people used to earnestly warn each other not to name a file '.Sony' (a special name reserved for the floppy driver) as it supposedly trashed your HD. Although I've never heard of anyone reproducing it.
- Xylakant 10y agoTrying to name a folder "trash" led to the error message "The name 'trash' is reserved for the operating system."
- db48x 10y agoNow's your chance: https://archive.org/details/mac_MacOS_7.0.1_compilation https://archive.org/details/mac_MacOS_7.0.1_compilation I'd try it myself, but I've only got my phone with me.
- pvg 10y ago7.0.1 might be a little late, at least, for the supposed catastrophic results. It doesn't like the file at all but nothing dreadful seems to happen.
- db48x 10y agoI tried it in that and System 6 (https://archive.org/details/mac_MacOS_6.0.8); https://archive.org/details/mac_MacOS_6.0.8); System 6 actually didn't care at all. An interesting bug. I haven't actually done so, but earlier versions are available if you know where to look (https://archive.org/details/mac_Paint_2 https://archive.org/details/mac_Paint_2).
- tatterdemalion 10y agoAs the blog post mentioned, we solve the issue by deleting the crate from the package repository and reserving these problematic names. The incident lasted about 2 and a half hours. Crate names have to be one or more valid idents connected by hyphens, so no other clever names like `/home` would be possible to upload. We already had some crate names reserved and we just needed to add these to the list.
- kibwen 10y ago> The incident lasted about 2 and a half hours. And because it was a weekend, much of that time involved me trying to figure out who had the proper credentials for crates.io, and then texting those people until one of them responded. :)
- derefr 10y agoReserving just the crate names won't cover your bases, though, no? I'm not clear on what exists as part of a crate—but if there's any user control over the filenames of the contents of the crate (e.g. if the crate's source code is in there) then any crate might contain a file named e.g. "nul.rs", triggering the same problem.
- kibwen 10y agoI think you're misunderstanding the problem described in the OP. When you build a project via cargo using the default settings, it fetches the git repository at https://github.com/rust-lang/crates.io-index https://github.com/rust-lang/crates.io-index to enable it to resolve dependencies locally. This git repository contains metadata for each library on crates.io, where the metadata for a given library is located in a file with the same name as the name of that library. When the OP uploaded a library whose name was an illegal filename on Windows, git unexpectedly choked when updating the local crate index repo, impacting all Windows users. It sounds like the concern you're describing is a different matter. It's likely true that if the source of a crate contains a file named "nul.rs", cargo on Windows will fail if it attempts to git-fetch the source (unless you're using Linux Subsystem for Windows, anyway). While this would indeed be a problem, it would only affect users who elect to use specific libraries, rather than serving as a denial-of-service for every Rust user on Windows.
- Strilanc 10y agoUrgh, this "nul" filename / reserved filename bug is probably in a lot of software.
- tannhaeuser 10y agoEvery MS-DOS programmer of old knows about nul, con, and the other reserved names. Those might come from CP/M actually (so are even older), and Atari TOS had them as well I believe.
- Sir_Cmpwn 10y agoSounds more like a problem with stupid Windows design choices than with anything you did.
- alkonaut 10y agoBecause there aren't any of those in posix...
- Sir_Cmpwn 10y agoAre you kidding? POSIX is perfect /s
- dagw 10y agoWindows is, for better or worse, fiercely proud of its backwards compatibility. So it's not so much a stupid Windows design choice as a 'stupid' DOS 1.0 design choice (and not even so much a choice as simply a quirk of how the DOS 1.0 file system worked) that Windows doesn't want to break backwards comparability with.
- wand3r 10y agoI agree with parent that it's a bit crazy; but I wouldn't be as critical. to your point; presumably even if they dropped DOS support something between DOS and now likely relies on that. It's a fine line.
- captn3m0 10y agoWhat I don't understand is why cargo fetches the entire crate list and create a directory for every crate (even if you never install it). Why not just have a single file with the entire list? The issue mentions they use a trie, but why use the filesystem as the trie store? Why not have a single file?
- nerdponx 10y agoAFAIK this is how the BSD Ports system works too.
- kibwen 10y agoThe original authors of cargo, wycats and carllerche, aren't around today to ask (it's a weekend!) though IRC attempted to answer regardless: <foo> to keep the number of files in a single directory down <foo> tools become unhappy with hundreds of thousands+ of things in a single dir <foo> as do filesytems <bar> why not just a flat file <bar> or sqlite or whatever <qux> right now it uses git's deduplication feature <qux> aka, when downloading updates you only download the objects that changed <qux> but it mostly works on a per file basis <qux> so git hashes each file and if the hash didnt change, it doesnt download an update <qux> but if it did, it treats it as completely new file, even if its just a little change
- kibwen 10y agoUpdate: <wycats> Because of this: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 <wycats> I ran some scenarios against huge repos when I first worked on cargo <wycats> Trying to minimize the cost of operations <wycats> I landed on the current strategy, and GitHub in the above thread more or less endorsed what we were already doing at that time <wycats> Also see https://github.com/rust-lang/cargo/issues/2452
- comex 10y agoIt's still fundamentally a waste of disk space. On my system, as of a minute ago, ~/.cargo/registry/index took up about 200MB for three different checkouts (for some reason). After deleting that and running `cargo update`, only one of them is recreated, 104MB. Out of that, 57MB is the JSON files and 47MB is git history. But if I just concatenate all the JSON files, the result is only 33MB, and after gzipping, 3MB. Hypothetically, a non-GitHub-based Cargo could store only those 3MB (using binary deltas to avoid resending it on every update), or even 0MB if it just relied on the server to resolve dependencies.
- bluejekyll 10y agoThis is a great example of something else about software. As software grows in usage and use cases, it starts bumping up against edge conditions which need to be handled for various reasons. Cargo now is becoming stronger and more stable because of bugs like this being discovered. All software goes through this growth cycle. It's great to see these things worked out in the various projects that support Rust. There is another point here though; anytime the question comes up to just rewrite a piece of software, throw out all the technical debt, it's not as straightforward as it seems. Remember, together with that technical debt lies a lot of valuable learnings written into the code. I haven't worked on Windows directly in years, but I never knew that NUL was a reserved word as a file. I would, and probably still will make this mistake in the future. Which makes me wonder, has anyone written a file name validation crate that guarantees that you're not writing to any reserved words on a filesystem of the host OS? A quick search of crate.io doesn't turn anything up.
- pjc50 10y ago> I never knew that NUL was a reserved word as a file. I would, and probably still will make this mistake in the future While we're here: NUL, COM<n>, LPT<n> and AUX are reserved.
- akavel 10y agoAnd CON, as Macha mentions in a sister comment. An idiom I remember from old times in DOS, for quickly writing some contents into a file - equivalent to `cat > myfile.txt` on Linux: COPY CON MYFILE.TXT
- ConfucianNardin 10y agoYou can also do type con > myfile.txt
- cesarb 10y ago> While we're here: NUL, COM<n>, LPT<n> and AUX are reserved. Worse: they're reserved with any extension. Have a file in your repository called "aux.rs"? It will cause problems on Windows.
- alkonaut 10y agoIt's very tricky to do cross platform file handling stuff, and only the most mature projects have ironed out this. Just look at your pet project and see if it handles - Windows and unix line breaks in text files - Windows and unix path separators - BOM and non-BOM marked files if parsing UTF - Forbidden filenames such as in this article By "handling" I mean it should accept or fail nicely on unexpected input - e.g. say that line breaks should be unix style, or paths should be backslashes etc. Very few projects actually do this well. Even fewer will do even more complex things like handling too long paths with nice error messages etc.
- tannhaeuser 10y agoSince Windows 10 now comes with an official Linux subsystem, why not just use POSIX APIs and conventions everywhere, and not bother with Windows-specific code if possible?
- untog 10y agoFor one, because the Linux subsystem is an optional install. If you're making anything user-facing you can't rely on it being there - it's really a tool for developers, not end-users.
- alkonaut 10y agoDepends on what type of application you are making. For a library that can be used in a "real" graphical Windows application, you can't make a posix type shortcut. I think "if possible" is (at least still) very rarely the case that it is.
- yrashk 10y agoWouldn't it make sense for Cargo not to use crate names in file names, and use hexadecimally encoded hashes instead?
- brianberns 10y agoYes, or some other identifier that's unique to that crate. Assuming that the crate name is also a valid file name seems risky.
- nomercy400 10y agoOr have a prefix. Once cost me exam points trying to optimize a prefix away. The examinators were right.
- stirner 10y agoYou could just hex encode the crate name. No need to hash it too.
- garaetjjte 10y agoOther magic aliases include CON, PRN, AUX, COM1-9 and LPT1-9. They are aliased to respective devices in Win32 namespace "\\.\". COMs and LPTs above 9 don't have aliases in global namespace and must be accessed explictly in Win32 namespace, eg. "\\.\COM10" (which itself is symlink to NT native "\Device\Serial9") In fact, it is possible to create files named NUL, COM1, etc. using \\?\ (eg. "\\?\C:\NUL" is valid path) prefix which disables parsing arcane Win32 magic files. Unfortunately these files are causing strange behaviour in applications that don't use that prefix, Explorer included. source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#namespaces https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
- monochromatic 10y agoI still remember using "copy con foo.txt" and ending with ctrl-z to quickly create a file. It was years before I understood how that actually worked.
- slobotron 10y agoThere was a bug in Windows 95 (98 too?) where if you tried to open 'nul\nul' or 'con\con' etc, it would BSOD instantly. Provided lots of drive-by fun in computer labs... (got really good at typing win+r con\con)
- tonyarkles 10y agoFor more fun, you could also target other machines with SMB shares. \\thevictim\foo\nul\nul would BSOD that machine. Good times.
- cesarb 10y agoIIRC, you could also reference it in a HTML page, so the whole computer would crash when that page was viewed.
- johnfn 10y agoThis is like the 90s era "undefined is not a function." null is not a problem, but null.null, on the other hand...
- lsiebert 10y agoHmm... I feel like someone should stick the reserved names into a json somewhere for easy reference for the next package manager.
- roryisok 10y agoI was working on a video project for a local comics convention, and named the project file "con.proj". That file hung around until I upgraded my hard drive because no file manager could delete it.
- toabi 10y agoI tried `npm install nul` on my win7 VM and it created a folder called nul which I can't get rid of ¬_¬
- msimpson 10y agoWhile I know nothing of Rust, Diesel, or CrateDB, I do know that Windows uses a case-insensitive file system and this fix doesn't seem to take that into consideration. However, the author of the fix does note: > I believe crates.io's namespace is case insensitive let me know if that's wrong Someone should probably validate that.
- zyx321 10y agoNot quite. Windows uses a case insensitive API on top of a case sensitive file system. FUN FACT: As of 2017, Windows 10 is (partially) binary-compatible to Ubuntu Linux. Any application that was originally compiled for Linux will still be case sensitive when running on Windows 10.
- msimpson 10y agoRight. I forgot NTFS is in fact case sensitive. Thanks.