16 ms·

My experience with using cp to copy 432 million files (39 TB)

• jrochkind1 12y ago
  So it was all the files in one go, presumably with `cp -r`? What about doing something with find/xargs/i-dunno to copy all the files, but break em into batches so you aren't asking cp to do it's bookkeeping for so many files in one process? Would that work better? Or worse in other ways?
  • dangero 12y ago
    The idea that you would need to do this seems fundamentally broken to me.
  • pedrocr 12y ago
    It should work much better provided you can easily divide it into batches where you know they don't have any shared hardlinked files. Otherwise you will duplicate those files wasting some space. That may be fine as well though.
  • xchg_ax_ax 12y ago
    This page may be useful: http://unix.stackexchange.com/questions/44247/how-to-copy-directories-with-preserving-hardlinks http://unix.stackexchange.com/questions/44247/how-to-copy-di... The main issue is that there's no api to get the list of files hard linked together: the only way is to check all the existing files and compare inodes. If you're doing a plain copy over 2 fs, you cannot choose which number the target inode will be, so you need to keep a map between inode numbers, or between inodes and file names ("cp" does the later).
    • sounds 12y ago
      pedrocr's comment above suggests a good solution: 1. Copy each file from the source volume to a single directory (e.g. /tmp) on the target volume, named for the source volume inode number. (edit: I suggest using a hierarchy of dirs to avoid the "too many dentry's" slowdown) 2. If the file has already been copied, it will already exist in /tmp - looking up the inode is a vanilla directory lookup 3. Create a hard link from /tmp to the actual path of the file 4. When all the files have been created on the target volume, delete the inode numbers in /tmp
  • fspeech 12y ago
    How do you keep hard links?
• gaius 12y ago
  I would probably have used tar|tar for this, or rsync.
  • mturmon 12y ago
    About rsync: if you just use -a, it does not copy hard links correctly: -a, --archive [...] Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H. If you do specify -H, rsync does keep track of hard links, but presumably at the cost of keeping a data structure similar to cp: -H, --hard-links This tells rsync to look for hard-linked files in the source and link together the corresponding files on the destination. Without this option, hard-linked files in the source are treated as though they were separate files. [...] Of course, rsync could be more efficient at keeping track of the hard links than cp, but there's no reason to believe a priori that it would be.
    • kazinator 12y ago
      Darn, thanks! I've been using rsync for incremental backups (themselves hard-link-based) for years, but neglecting to do this. That is to say, snapshot N+1 shares structure with backup N to save space (via hard links) via --link-dest. But files may be replicated because -H (missing) is orthogonal of --link-dest.
      • ars 12y ago
        You can use fdupes (or various others similar to it) to find and automatically hardlink the duplicates.
  • __david__ 12y ago
    Rsync has similar issues when you are copying hard links. It has to keep a table of all the inodes/filename pairs it sees so it can detect hard links and relink them on the destination. This can be big if you have a ton of hard links (say in a backup situation). Rsync at least acts reasonably idempotent, so you can just run it again if it gets interrupted, which is usually why I use it for large copies. I don't remember off the top of my head if tar handles hard links—it may be inappropriate for this usage.
• maaku 12y ago
  Another lesson: routinely scrub your RAID arrays.
  • jewel 12y ago
    On debian-based systems, /etc/cron.d/mdadm will already do this on the first Sunday of the month.
• mturmon 12y ago
  The later replies regarding the size of the data structures cp is using are also worth reading. This is a case where pushing the command farther can make you think harder about the computations being done.
• 0x0 12y ago
  I wonder how well rsync would have fared here.
  • sitkack 12y ago
    Rsync can die just from scanning the whole directory tree of files first.
    • chadcatlett 12y ago
      The incremental option(enabled by default) introduced in rsync 3.0 greatly reduces the need for scanning the whole directory structure.
    • nailer 12y ago
      If it's changing? You'd snapshot (assuming either LVM or BtrFS) then rsync of the snapshot.
• IvyMike 12y ago
  Interesting. In Windows-land, the default copy is pretty anemic, so probably most people avoid it for serious work. I'd probably use robocopy from the command line. And if I was being lazy, I'd use the Teracopy GUI. I think my limit for a single copy command has been around 4TB with robocopy--and that was a bunch of large media files, instead of smaller more numerous files. Maybe there's a limit I haven't hit.
  • RyJones 12y ago
    Once you get above about a million files on NTFS, life becomes very slow. If you use a DVCS and check out a bunch of projects with a bunch of history, you can easily get to a point where you can't do too much with the drive. This was true as of 7/2008, perhaps 8/2012 have different/better tunings.
    • ygra 12y ago
      Number of files on the drive shouldn't matter that much, actually. In a single folder is a different matter, of course (for that you definitely want to disable 8.3 compatibility). But the only way I can imagine right now where the number of files on the drive makes a difference would be where you crammed the drive almost full, resulting in a very fragmented MFT (normally the file system tries to keep space for growing the MFT, but not if you try to fill that space with data, too).
  • noinsight 12y ago
    > Teracopy I've used FastCopy for GUI based larger transfers, it's open source and can handle larger datasets well in my experience. It also doesn't choke on >MAX_PATH paths. Haven't had problems with it. Supposedly it's the fastest tool around... The only slight issue is that the author is Japanese so the English translations aren't perfect plus the comments in the source are in Japanese.
    • gizmo686 12y ago
      ">MAX_PATH paths" How does this happen?
• dbbolton 12y ago
  >We use XFS Why?
  • WestCoastJustin 12y ago
    Most likely because it would take a looong time to fsck 40TB using ext3/4.
  • Sanddancer 12y ago
    XFS really shines when you're dealing with large volumes, especially with RAID. There are certain features, like striped allocation, that mean you end up getting faster performance, because the filesystem has a better idea of how blocks are laid out on the disks.
  • cnvogel 12y ago
    I personally still consider XFS a very mature and reliable filesystem. Both in terms of utility programs and kernel implementation. If I remember correctly, it was ported to linux from SGI/Irix where it was used for decades. It also was the default fs for RedHat/centos for a long time, so it might still have stuck at many shops. Heres my anecdotal datapoint on which I base my personal believe: From about 10-6 years ago, when I was doing sysadmin-work at university building storage-systems from commodity parts for experimental bulk data, we first had a load of not-reliably working early raid/sata(?) adapters, and those made ext3 and reiserfs (I think...) oops the kernel when the on-disk structure went bad. Whereas XFS just put a "XFS: remounted FS readonly due to errors" in the kernel logfile. That experience made XFS my default filesystem up to recently when I started to switch to btrfs. (of course, we fixed the hardware-errors, too... :-) ) Also, from that time, I got to use xfsdump/xfsrestore for backups and storage of fs-images which not even once failed on me.
    • Eiriksmal 12y ago
      As a blithe, new-Linux user (3.5 years), I was bumfuzzled when I saw RHEL/CentOS 7 switched from ext4 to XFS, figuring it to be some young upstart stealing the crown from the king. Then I did some Googling and figured out that XFS is as old as ext2! I'm looking forward to discovering how tools like xfs* can make my life easier.
• sitkack 12y ago
  I appreciate that he had the foresight to install more ram and configure more swap. I would hate to be days into a transfer and have the OOM killer strike.
• sauere 12y ago
  > While rebuilding, the replacement disk failed, and in the meantime another disk had also failed. I feel the pain. I went thru the same hell a few months ago.
• lucb1e 12y ago
  > 20 years experience with various Unix variants > I browsed the net for other peoples' experience with copying many files and quickly decided that cp would do the job nicely. After 20 years you no longer google how to copy files. Edit: Reading on he talks about strace and even reading cp's source code which makes it even weirder that he had to google how to do this... Edit2: Comments! Took only ten downvotes before someone bothered to explain what I was doing wrong, but now there are three almost simultaneously. I guess those make a few good points. I'd still think cp ought to handle just about anything especially given its ubiquitousness and age, but I see the point. And to clarify: I'm not saying the author is stupid or anything. It's just weird to me that someone with that much experience would google something which on the surface sounds so trivial, even at 40TB.
  • sitkack 12y ago
    Because the man is wise. He also didn't kill a job that appeared to be hung, he started reading the code to figure out why and determined that it would in fact, complete.
  • jMyles 12y ago
    I'm not yet 20 years into this game, but the further I go, the more I'm finding I need to be willing to swallow my pride and check in with the current state of best practices.
  • jrochkind1 12y ago
    Because it turns out cp _barely_ managed to do the job, and he was almost wrong despite googling. (It did the job, I don't know about 'nicely') Just assuming that, oh, cp copies files, of course it can copy a few terrabytes no problem... is something that you might do, but not something someone with 20 years experience with unix would do.
  • Howitzer 12y ago
    Because the typical use case of copying a few files does not really compare to the use case of having to copy 43 terabyte of critical data that may or may not have corruptions. It's nothing but wise to double check.
  • 12y ago
• Andys 12y ago
  A problem with cp (and rsync, tar, and linux in general) is there is read-ahead within single files, but no read-ahead for the next file in the directory. So it doesn't make full use of the available IOPS capacity.
• vhost- 12y ago
  These are the types of stories I love. I just learned a boat load in 5 minutes.
  • 3rd3 12y ago
    Is there maybe an archive website dedicated to these kind of stories?
    • vhost- 12y ago
      If not, there should be.
    • breadbox 12y ago
      At one time there was; it was called the Internet. The archive still exists, but it's been made harder to browse through due to being jumbled up with javascript and cat gifs.
      • jayvanguard 12y ago
        It's true. We should have never let the public on the Internet. It has been downhill since then.
        • taeric 12y ago
          False choice, isn't it? I mean, the complaint isn't that the public now has sites with massive javascript and related technologies. The complaint is that it has muscled out useful sites that did not use those technologies. And it should be heavily noted that the heavy muscles that have pushed out many of these sites is not necessarily "the public."
          • thinkling 12y ago
            Kind of funny to say that on a text-only JS-free site that seems to be alive and well, linking to an article on an old-school mailing list archive site. :)
• mililani 12y ago
  This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild process if the RAID controller is smart enough. You just swap failed drives, and the RAID controller automatically utilizes the back up drive and then mirrors onto the new drive. Just much faster and much less prone to multiple drive failures killing the entire RAID.
  • LeoPanthera 12y ago
    How does the rebuild time of ZFS's raidz compare to RAID5/6?
    • XorNot 12y ago
      I have a 24TB RAIDZ3 array. Losing a disk out of that takes about 12-18 hours to rebuild.
    • benjohnson 12y ago
      It really depends on the load and if your raidz has 1,2, or 3 spare drives. From what I've experienced, re-slivering a the new drive in a raidz seems to happen at about 30% of the maximum ZFS throughput under medium load. And even better than many RAID5/6 implementations - it only re-slivers enough to store your data and not just the entire drive.
    • fiatmoney 12y ago
      It is generally superior because it can skip blank blocks. RAID is byte-by-byte.
    • craigyk 12y ago
      My largest ZFS pool is currently ~64TB ( 3 X 10 3TB (raidz2) ) The pool has ranged from 85%-95% full (it's mostly at 85% now and used mostly for reads). Resilvering one drive usually takes < 48 hours. Last time took ~32 hours. Something else cool: When I was planning this out I wrote a little script to calculate the chance of failure. With a drive APF of 10% (which is pretty high), and a rebuild speed of 50MB/sec (very low compared to what I typically get) I have a 1% chance of losing the entire pool over a 46.6 year period. If I add 4 more raidz2 10X3TB VDEVS that would drop to 3.75 years.
• gwern 12y ago
  > Wanting the buffers to be flushed so that I had a complete logfile, I gave cp more than a day to finish disassembling its hash table, before giving up and killing the process....Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. Does anyone know what the 'tear down' part is about? If it's about erasing the hashtable from memory, what takes so long? I would expect that to be very fast: you don't have to write zeros to it all, you just tell your GC or memory manager to mark it as free.
  • jrochkind1 12y ago
    I don't understand it, but the OP says this, implying the author agrees with you for any modern system, but not on 'old systems without working memory management' > And unless old systems without working memory management must be supported, I don't see any harm in simply removing the call to the forget_all function towards the end of cp.c.
    • MichaelGG 12y ago
      How would such a system even work? Any abnormal process termination or bugs would mean eventually the system becomes unusable.
      • gizmo686 12y ago
        The system would slowly leak memory until you reboot.
        • icedchai 12y ago
          Yep, I remember programming on such a system (Amiga) in the early 90's! Fun times.
  • maxhou 12y ago
    OP says the system was swapping. Once your hash table and its linked lists have been swapped to disk, each pointer dereference causes a random disk access, which are orders of magnitude slower than RAM.
• pedrocr 12y ago
  Unix could really use a way to get all the paths that point to a given inode. These days that shouldn't really cost all that much and this issue comes up a lot in copying/sync situations. Here's the git-annex bug report about this: https://git-annex.branchable.com/bugs/Hard_links_not_synced_in_direct_mode/ https://git-annex.branchable.com/bugs/Hard_links_not_synced_...
  • pedrocr 12y ago
    Seems like this is actually a feature in VxFS: https://sort.symantec.com/public/documents/sfha/6.0/linux/productguides/html/sf_admin/ch35s01.htm https://sort.symantec.com/public/documents/sfha/6.0/linux/pr... http://sfdoccentral.symantec.com/sf/5.0/aix/manpages/vxfs/vxfs_inotopath_gen_3.html http://sfdoccentral.symantec.com/sf/5.0/aix/manpages/vxfs/vx... I wonder if any modern filesystems (xfs/ext4/btrfs/zfs) can actually enable tracking this.
  • asveikau 12y ago
    Wow, it's not every day I hear about a filesystem feature that Windows has and Linux doesn't. (On a recent windows system: fsutil hardlink list <path> -- you can try any random exe or DLL in system32 for an example of a hard link.) I forget what the api for that looks like if I ever knew. Might be private. I am surprised, usually Linux is way ahead of Windows on shiny filesystem stuff.
    • ygra 12y ago
      Is that really a feature the file system provides? Or does that command simply walk the MFT and finds all hardlinks to the same file record? As far as I know NTFS does not store what hard links point to a file.
      • asveikau 12y ago
        I don't know. I do know there are tens of thousands of hardlinked files under system32, and the command I listed above can enumerate links pretty quickly.
    • 12y ago
• RexM 12y ago
  Is this where a new cp fork comes about called libracp?
• icedchai 12y ago
  For that many files I probably would've used rsync between local disks. shrug
  • ajross 12y ago
    And hopefully you would have written up a similar essay on the oddball experiences you had with rsync, which is even more stateful than cp and even more likely to have odd interactions when used outside its comfort zone. Ditto for tricks like:(cd $src; tar cf - .) | (cd $dst; tar xf -). Pretty much nothing is going to work in an obvious way in a regime like this. That's sort of the point of the article.
    • icedchai 12y ago
      Or maybe not. He mentions rsnapshot in the article, which uses rsync under the hood. This implies rsync would have a very good chance of handling a large number of hardlinks... since it created them in the first place.
      • bhaak 12y ago
        But rsnapshot created that large number of hardlinks incrementally. Without trying it there's no indication that rsync handles that better, especially as rsync with default settings doesn't preserve hardlinks as rsync builds a quite expensive data structure for them. I'm surprised that cp even worked. I wouldn't have had the balls to just use cp as it could just have failed. My strategy would probably have been to cp or rsync over the latest backup (so that new incremental backups could be immediately be created again) and then incrementally rsync'ing the previous backups with hardlink detection.
      • sophacles 12y ago
        That doesn't follow. If backups are for multiple machines to a big file server, the backup machine will have a much larger set of files than those that come from an individual machine. Further, each backup "image" compares the directory for the previous backup to the current live system. Generally it looks something like this: 1. Initial backup or "full backup" - copy the full targeted filesystem to the time indexed directory of the backup machine. 2. Sequential backups: a. on the backup machine, create a directory for the new time, create a mirror directory structure of the previous time. b. hard link the files in the new structure to those in the previous backup (which may be links themselves, back to the last full backup. c. rsync the files to the new backup directory. Anything that needs to be transfered results in rsync transfering the file to a new directory, the moving it into the proper place. This unlinks the filename from the previous version and replaces it with the full version. So yeah, the result of this system over a few machines and a long-timeframe backup system is way more links on the backup machine than any iteration of the backup will ever actually use.
• pedrocr 12y ago
  How about this for a better cp strategy to deal with hardlinks: 1. Calculate the hash of /sourcedir/some/path/to/file 2. Copy the file to /tempdir/$hash if it doesn't exist yet 3. Hard-link /destdir/some/path/to/file to /tempdir/$hash 4. Repeat until you run out of source files 5. Recursively delete /tempdir/ This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all the hashing. If you're smart about doing steps 1 and 2 together it shouldn't require any additional I/O (ignoring the extra file metadata). Edit: actually this won't recreate the same hardlink structure, it will deduplicate any identical files, which may not be what you want. Replacing the hashing with looking up the inode with stat() would actually do the right thing. And that would basically be an on-disk implementation of the hash table cp is setting up in memory.
  • sounds 12y ago
    Using the /sourcedir inode number sounds right. Be careful not to create too many files in /tempdir (create a hierarchy of subdirs). Filesystems slow way down when they get "too many" files in one dir. But take a good look at a tarpipe before applying 'cp' to such a large set of files.
    • pedrocr 12y ago
      >Be careful not to create too many files in /tempdir (create a hierarchy of subdirs). Filesystems slow way down when they get "too many" files in one dir. Yeah, you probably want something like /tempdir/srcfilesystem/NN/NN/NN/NN to handle the issue you mention and recursive copies that cross filesystem boundaries. Besides that tempdir will need to be on the same filesystem as destdir so the hard copy will work, and if destdir has been setup as a multiple filesystem target you'll need to keep a tempdir instance for each destination filesystem. Plenty of corner cases to think about.
  • derefr 12y ago
    If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined. Plan9's default filesystem is made up of two parts: Fossil, and Venti. - Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values. - Venti is a persistent graph database that holds what would today be called "inode metadata." It presents itself as a regular hierarchical filesystem. The "content" property of an inode simply holds a symbolic path, usually to an object in a mounted Fossil "bucket." When you write to Venti, it writes the object to its configured Fossil bucket, then creates an inode pointing to that key in that bucket. If the key already existed in Fossil, though, Fossil just returns the write as successful immediately, and Venti gets on with creating the inode. Honestly, I'm terribly confused why all filesystems haven't been broken into these two easily-separable layers. (Microsoft attempted this with WinFS, but mysteriously failed.) Is it just inertia? Why are we still creating new filesystems (e.g. btrfs) that don't follow this design?
• nraynaud 12y ago
  it reminds me of crash only software.
• angry_octet 12y ago
  The difficulty is that you are using a filesystem hierarchy to 'copy files' when you actually want to do a volume dump (block copy). Use XFS and xfsdump, or ZFS and zfs send, to achieve this. Copy with hard link preservation is essentially like running dedupe except that you know ahead of time how many dupes there are. Dedupe is often very memory intensive, and even well thought out implementations don't support keeping book keeping structures on disk.
  • steveh73 12y ago
    "Normally I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained the bad blocks."
    • angry_octet 12y ago
      I was simplifying... dump backs up inodes not blocks. Some inodes point to file data and some point to directory data. Hard links are references to the same inode in multiple directory entries, so when you run xfsrestore, the link count increments as the FS hierarchy is restored. xfsdump/zfs send are file system aware, unlike dd, and can detect fs corruption (ZFS especially having extensive checksums). In fact, any info cp sees about corruption comes from the FS code parsing the FS tree. However, except on zfs/btrfs, data block corruption will pass unnoticed. And in my experience, when you have bad blocks, you have millions of them -- too many to manually fix. As this causes a read hang, it is usually better to dd copy the fs to a clean disk, set to replace bad blocks with zeros, then fsck/xfs_repair when you mount, then xfsdump. dd conv=noerror,sync,notrunc bs=512 if=/dev/disk of=diskimg See Also: http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/xfs-repair.html http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US... http://xfs.org/index.php/Reliable_Detection_and_Repair_of_Metadata_Corruption http://xfs.org/index.php/Reliable_Detection_and_Repair_of_Me...
      • Rapzid 12y ago
        If the risk of keeping the system running while the array rebuilt was deemed to high, I would have just gone with a dd/ddrescue of the remaining disks onto new disks and then moved on from there. +1 for mentioning ZFS. It's really quite amazing. Almost like futuristic alien technology compared to the other freely available file systems.
• fintler 12y ago
  I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/hpc/libcircle https://github.com/hpc/libcircle that would gather metadata on a few hundred-million files in a few hours (we had to limit the speed so it wouldn't take down the filesystem). For a slimmed down version of that tool, take a look at https://github.com/hpc/libdftw https://github.com/hpc/libdftw
  • laymil 12y ago
    And it's interesting and useful for scientific computing where you already have an MPI environment and distributed/parallel filesystems. However, it's not really applicable to this workload, as the paper itself says. There is a provision in most file systems to use links (symlinks, hardlinks, etc.). Links can cause cycles in the file tree, which would result in a traversal algorithm going into an infinite loop. To prevent this from happening, we ignore links in the file tree during traversal. We note that the algorithms we propose in the paper will duplicate effort proportional to the number of hardlinks. However, in real world production systems, such as in LANL (and others), for simplicity, the parallel filesystems are generally not POSIX compliant, that is, they do not use hard links, inodes, and symlinks. So, our assumption holds. The reason this cp took such large amounts of time was the desire to preserve hardlinks and the resize of the hashtable used to track the device and inode of the source and destination files.
    • encoderer 12y ago
      Sure, but if you read that article you walk away with a sense of thats a lot of files to copy. And the GP built a tool for jobs 2-3 orders of magnitude larger?! Clearly there are tradeoffs forced on you at that size...
    • jlafon 12y ago
• limaoscarjuliet 12y ago
  Rsync seems a better tool for this. Can be run multiple times and it will just copy missing blocks.
• rwg 12y ago
  Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits. A wonderful trend I've noticed in Free/Open Source software lately is proudly claiming that a program is "Valgrind clean." It's a decent indication that the program won't doing anything silly with memory during normal use, like leak it. (There's also a notable upswing in the number of projects using static analyzers on their code and fixing legitimate problems that turn up, which is great, too!) While you can certainly just let the OS reclaim all of your process's allocated memory at exit time, you're technically (though intentionally) leaking memory. When it becomes too hard to separate the intentional leaks from the unintentional leaks, I'd wager most programmers will just stop looking at the Valgrind reports. (I suppose you could wrap free() calls in "#ifdef DEBUG ... #endif" blocks and only run Valgrind on debug builds, but that seems ugly.) A more elegant solution is to use an arena/region/zone allocator and place potentially large data structures (like cp's hard link/inode table) entirely in their own arenas. When the time comes to destroy one of these data structures, you can destroy its arena with a single function call instead of walking the data structure and free()ing it piece by piece. Unfortunately, like a lot of useful plumbing, there isn't a standard API for arena allocators, so actually doing this in a cross-platform way is painful: • Windows lets you create multiple heaps and allocate/free memory in them (HeapCreate(), HeapDestroy(), HeapAlloc(), HeapFree(), etc.). • OS X and iOS come with a zone allocator (malloc_create_zone(), malloc_destroy_zone(), malloc_zone_malloc(), malloc_zone_free(), etc.). • glibc doesn't have a user-facing way to create/destroy arenas (though it uses arenas internally), so you're stuck using a third-party allocator on Linux to get arena support. • IRIX used to come with an arena allocator (acreate(), adelete(), amalloc(), afree(), etc.), so if you're still developing on an SGI Octane because you can't get enough of that sexy terminal font, you're good to go.
  • _delirium 12y ago
    Adding some kind of arena-allocation library to both the build & runtime dependencies solely to keep valgrind happy, with no actual improvement in functionality or performance, doesn't seem like a great tradeoff on the software engineering front. I'd rather see work on improving the static analysis. For example if some memory is intended to be freed at program cleanup, Valgrind could have some way of being told, "this is intended to be freed at program cleanup". Inserting an explicit (and redundant) deallocation as the last line of the program just to make the static analyzer happy is a bit perverse. (That is, assuming that you don't need portability to odd systems that don't actually free memory on process exit.)
• ccleve 12y ago
  Maybe this is naive, but wouldn't it have made more sense to do a bunch of smaller cp commands? Like sweep through the directory structure and do one cp per directory? Or find some other way to limit the number of files copied per command?
  • caf 12y ago
    No, because then it wouldn't have replicated the hardlink structure of the original tree. That was the goal, and also the bit that causes the high resource consumption.
• minopret 12y ago
  In light of experience would it perhaps be helpful after all to use a block-level copy (such as Partclone, PartImage, or GNU ddrescue) and analyze later which files have the bad blocks? I see that the choice of a file-level copy was deliberate: "I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained the bad blocks."
  • fsniper 12y ago
    Also there is no mention of unrecoverable file analysis like error handling of cp operations in the article. And with this many files it would not be feasible without using an error log file. So going with a simple block copy should suffice IMHO.
    • rbh42 12y ago
      I'm the OP, so I can shred a bit of light on that: Dell's support suggested a file-level copy when I asked them what they recommended (but I'm not entirely sure they understood the implications). Also, time was not a big issue. I did keep a log file with the output from cp, and it clearly identified the filenames for the inodes with bad blocks. Actually, I'm not sure how dd would handle bad blocks.
      • fsniper 12y ago
        Thank you for clarification. I was about to bet on "read fail repeat skip" cycle for dd's behaviour but, looking into coreutil's source code at https://github.com/goj/coreutils/blob/master/src/dd.c https://github.com/goj/coreutils/blob/master/src/dd.c , if I'm not mistaken , dd does not try to be intelligent and just uses a zeroed out buffer so It would return 0's for unreadable blocks.
  • mnw21cam 12y ago
    A while back I had a hard drive die with an unrecoverable error on the superblock (ext3) and a few other places. I sent the hard drive to a professional recovery firm who were as incompetent as incompetent can be. I got the duff drive back and decided to have a go myself. Recovering the superblock was easy - there are extra copies of that. I then did a full map of the drive listing bad blocks, and did a reverse map to work out which files contained those blocks. Luckily, the only file that contained any bad blocks was a swap file, so I recovered absolutely everything. Other bad blocks were in unallocated space. Ext3 is actually a sane enough disc layout that it is possible to work out what is going on manually. I would have been stuffed if it had been a more complicated filesystem.
• grondilu 12y ago
  On Unix, isn't it considered bad practice to use cp in order to copy a large directory tree? IIRC, the use of tar is recommended. Something like: $ (cd $origin && tar cf - *) | (cd $destination && tar xvf - )
  • dmckeon 12y ago
    Use && there, not ; - consider the result if either of the cd commands fails.
    • grondilu 12y ago
      fixed
  • brazzy 12y ago
    Would have been even worse in this case due to the requirement to preserve hard links.
    • ccozan 12y ago
      tar can preserve hardlinks. In fact, tar is really the best solution in this case, using the streaming between two tar commands.
      • ygra 12y ago
        Wouldn't it somehow have to keep about the same data as cp for preserving hard links?
      • brazzy 12y ago
        As discussed in detail elswhere abive: the problem is not with the ability to preserve hardlinks, but the resources, namely RAM, required to do so. Two tar command would in fact be much worse, because they'd require twice as many resources.
        • grondilu 12y ago
• davidu 12y ago
  This is not, not, not how one should be using RAID. The math is clear that in sufficiently large disk systems, RAID5, RAID6, and friends, are all insufficient.
  • lysium 12y ago
    Can you elaborate?
    • davidu 12y ago
      http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162 http://www.zdnet.com/blog/storage/why-raid-5-stops-working-i...
      • lysium 12y ago
        Thanks for the link! The article says, that due to the read error rate and the size of today's disks, RAID 5 and RAID 6 have (kind of) lost their purpose.
        • davidu 12y ago
          Yep, mathematically no longer safe.
    • curun1r 12y ago
      I'm guessing he means that RAID is designed as a technology to increase the availability of your data and the person in this story is using it as a backup solution. Google 'RAID backup' to see why it's such a bad idea to use it for that purpose. He specifically used cp because he was worried about data loss from using something like dd. But if he'd been doing regular backups to tape, he could have just restored the backup from tape with no fuss or problems. It's only because he was using tools that aren't suited to what he was trying to do that he ran into problems.
• dspillett 12y ago
  > The number of hard drives flashing red is not the same as the number of hard drives with bad blocks. This is the real take-away. Monitor your drives. At very least enable SMART, and also regularly run a read on the full underlying drive (SMART won't see and log blocks that are on the way out so need retries for successful reads, unless you actually try to read those blocks). That won't completely make you safe, but it'll greatly reduce the risk of other drives failing during a rebuild by increasing the chance you get advanced warning that problems are building up.
  • rbh42 12y ago
    Glad someone noticed it (I'm the OP). Reading the drives systematically is called "Patrol Read" and is often enabled by default, but you can tweak the parameters.
• pmontra 12y ago
  Another lesson to be learnt is that it's nice to have the source code for the tools we are using.
• pixelbeat 12y ago
  I found an issue in cp that caused 350% extra mem usage for the original bug reporter, which fixing would have kept his working set at least within RAM. http://lists.gnu.org/archive/html/coreutils/2014-09/msg00014.html http://lists.gnu.org/archive/html/coreutils/2014-09/msg00014...
• calvins 12y ago
  I would usually use the tarpipe mentioned already by others for this sort of thing (although I probably wouldn't do 432 million files in one shot): (cd $SOURCE && tar cf - .) | (mkdir -p $DEST && cd $DEST && tar xf -) Another option which I just learned about through reading some links from this thread is pax (http://en.wikipedia.org/wiki/Pax_%28Unix%29 http://en.wikipedia.org/wiki/Pax_%28Unix%29), which can do it with just a single process: (mkdir -p $DEST && cd $SOURCE && pax -rw . $DEST) Both will handle hard links fine, but pax may have some advantages in terms of resource usage when processing huge numbers of files and tons of hard links.
  • tedunangst 12y ago
    You know how tar handles hardlinks, right? By creating a giant hash table of every file.
    • dredmorbius 12y ago
      How's that going to scale with memory? In-memory hash tables were the downfall of cp here.
      • tedunangst 12y ago
        It's going to scale just like you'd imagine it would. All the people saying "oh, tar was built for this" obviously haven't actually tried replicating the experiment using tar.
        • dredmorbius 12y ago
          Pretty much as I'd suspected.
    • calvins 12y ago
      I wasn't suggesting to blindly use a tar pipe for as many files as the OP was trying to address. My main point was that the tar pipe (locally or over ssh) is superior to cp/scp for the sorts of things that many of us do regularly (as discussed elsewhere in this thread), such as copying tens of thousands to low millions of mostly smallish files locally or remotely and wanting to preserve the structure of links and file metadata, and that pax looks like a simpler alternative to the tar pipe that accomplishes the same thing in a single command, although it does still use a hash table internally. I wonder if using something other than a hash table would be worthwhile for cases like the OP though, or if there is a tool that has that option -- the memory overhead of empty buckets and resizing seem like serious downsides when you're trying to handle as many files as the OP.
• brokentone 12y ago
  Feels like a similar situation to this: http://dis.4chan.org/read/prog/1109211978/21 http://dis.4chan.org/read/prog/1109211978/21
• pflanze 12y ago
  I've written a program that attempts to deal with the given situation gracefully: instead of using a hash table, it creates a temporary file with a list of inode/device/path entries, then sorts this according to inode/device, then uses the sorted list to perform the copying/hardlinking. The idea is that sorting should work well with much lower RAM requirements than the size of the file to be sorted (due to data locality, unless the random accesses with the hash, it will be able to work with big chunks, at least when done right (a bit hand-wavy, I know, this is called an "online algorithm" and I remember Knuth having written about those, haven't had the chance to recheck yet); the program is using the system sort command, which is hopefully implementing this well already). The program stupidly calls "cp" right now for every individual file copy (not the hard linking), just to get the script done quickly, it's easy to replace that with something that saves the fork/exec overhead; even so, it might be faster than the swapping hash table if the swap is on a spinning disk. Also read the notes in the --help text. I.e. this is a work in progress as a basis to test the idea, it will be easy to round off the corners if there's interest. https://github.com/pflanze/megacopy https://github.com/pflanze/megacopy PS. the idea of this is to make copying work well with the given situation on a single machine, unless the approach taken by the dcp program mentioned by fintler which seems to rely on a cluster of machines. There may also be some more discussion about this on the mailing list: http://lists.gnu.org/archive/html/coreutils/2014-09/msg00013.html http://lists.gnu.org/archive/html/coreutils/2014-09/msg00013...
• dredmorbius 12y ago
  The email states that file-based copy operations were used in favor of dd due to suspected block errors. Two questions come to mind: 1. I've not used dd on failing media, so I'm not sure of the behavior. Will it plow through a file with block-read failures or halt? 2. There's the ddrescue utility, which is specifically intended for reading from nonreliable storage. Seems that this could have offered another means for addressing Rasmus's problem. It can also fill in additional data on multiple runs across media, such that more complete restores might be achieved. https://www.gnu.org/software/ddrescue/ddrescue.html https://www.gnu.org/software/ddrescue/ddrescue.html
  • pflanze 12y ago
    OP said "I went for a file-level copy because then I'd know which files contained the bad blocks". When you copy the block device with ddrescue (dd doesn't have logic to work around the bad sectors and the only sensible action for it is thus to stop, but don't take my word for it), the result will just have zeroes in the places where bad blocks were, and, assuming the filesystem structure is good enough (you should run fsck on it), will give you files with zones of zeroes. But you won't know which files without either comparing them to a backup (which you won't have by definition if you're trying to recover) or with a program that verifies every file's structure (which won't exist for the general case). Whereas cp will issue error messages with the path of the file in question. So the OP's decision makes sense.
    • dredmorbius 12y ago
      I've played with ddrescue very lightly. From the GNU webpage linked above, it appears it creates logfiles which can be examined: Ddrescuelog is a tool that manipulates ddrescue logfiles, shows logfile contents, converts logfiles to/from other formats, compares logfiles, tests rescue status, and can delete a logfile if the rescue is done. Ddrescuelog operations can be restricted to one or several parts of the logfile if the domain setting options are used. That might allow for identification of files with bad sectors.
      • pflanze 12y ago