4 ms·
Funny(???) warstory: 1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, excep
by jabl 9y ago
Funny(???) warstory:
1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc.
2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaults. Most people ignored this, of course, but files with now invalid utf-8 were mostly fine, just with the occasional "?"'s in the names.
3. Switch to NFSv4. Invisible to end users. NFSv4 per se requires that paths are UTF-8 encoded, but in practice the Linux NFS server and client just pass along a bag of bytes, so invalid UTF-8 just worked as fine as it did previously.
4. Switch from a Linux NFS server to a netapp.
5. User complains that files are missing. Initial comparison with the old Linux NFS server, which was still online, shows no problems. Problem occurs only on user workstation, not on admin box which has both the old Linux NFS and netapp directory trees mounted. Investigation on users workstation shows that in some cases lots of files appear to be missing, including ones which plain ASCII names.
- Turns out that the admin box had the netapp mounted with NFSv3, and thus everything appeared Ok there, including the rsync from Linux NFS -> netapp in the first place.
- However, when mounted using NFSv4, netapp follows the spec and does not like non-utf8 paths. Does it report an error then? Hell no, the NFS READDIR (READDIRPLUS?) message reply just stops returning directory entries when it hits the first one with invalid UTF-8. And thus you get a partial directory listing. GAAAH!
- So the solution was to run convmv centrally (from the admin box which had the netapp mounted with NFSv3) for the entire directory tree which had been moved.
- 0x0 9y agoA similar thing happens with Java's file and directory APIs on Linux. IIRC in Java filenames are Strings. If your vm is configured with utf8 as "file.encoding" and you have non-utf8-compliant filenames on your filesystem, those files are completely inaccessible to Java!
- Avamander 9y agoThis is kind-of why I think there should be heavy push towards not supporting any other encoding than UTF-8.
- ensiferum 9y agoUseless non-sense like this is the main reason why i desperately want to move away from software engineering.
- icebraining 9y agoCareful, though. Plain human bureaucracy can be just as bad - and often harder to debug :)
- wazoox 9y agoAh yes, had the same fun problem at a customer's facility last week. Moving 350 TB of data from an old DDP storage server to a Linux one. Mounting with CIFS (no other option available), an copying using "cp -a". The file names look OK after the copy on the Linux machine. However, when exporting the directory through Samba, the Macs Finder doesn't display files with accents in the names (though they appear correctly with "ls", weird...). So the user copies the files again, using the Finder. Now I have files with exactly the same name (uhhhhh???): # ls -l Mmo-1. -rw-rw-rw- 1 root root 8417218 6 sept. 2013 Mémo-1.aif -rwxr--r-- 1 test test 8417218 6 sept. 2013 Mémo-1.aif -rw-rw-rw- 1 root root 363175 6 sept. 2013 Mémo-1.m4a -rwxr--r-- 1 test test 363175 6 sept. 2013 Mémo-1.m4a Yes, it looks like two files have exactly the same name, but actually they're different: one as "é" encoded as 0xCC81, and the other one (the "good one") as 0xC3A9. Why is that? Why does one work with the Finder, and the other doesn't? who knows.
- throwaway7767 9y agoMost likely it's different normalization. I've seen this before with Mac systems. Renaming the files to use NFKC normalization fixed it. In python, you could loop through the files and do something like: os.rename(originalfilename, unicodedata.normalize('NFKC', originalfilename.decode('utf8'))) EDIT: You'll probably need to do this on a non-Mac system, linux for example should work.