5 ms·
Unusual disk latency
- bprater 18y agoOne of those completely random discoveries that almost never happened in this instance of the universe. Sorta like vulcanized rubber. Or waffle cones.
- kogir 18y agoWhat's coolest is that the analytics tools were available to discover it. No other storage solution I've seen has that level of instrumentation.
- seiji 18y agoSun went out of their way to make an amazing storage product. Interestingly, they started with perl, then rewrote everything in JavaScript/C instead[1]. You can even download a VMware image to test their UI and provisioning: http://www.sun.com/storage/disk_systems/unified_storage/resources.jsp http://www.sun.com/storage/disk_systems/unified_storage/reso... NetApp should go out of business already. [1]: http://news.ycombinator.com/item?id=390000 http://news.ycombinator.com/item?id=390000 (yes, item 390,000)
- sh1mmer 18y agoI was at a conference with a Sun evangelist banging on about this stuff. I was pretty skeptical. One of our Hadoop grid sysadmins used to work at Sun. He was raving about things like D-Trace and a bunch of features of Solaris that other *nix don't have. I guess I'm not 100% sold on Open Solaris but they do have some pretty badass stuff in there.
- Andys 18y agoIts nice that they release the fruit of their labours in OpenSolaris. I've rolled my own hybrid storage and played with it in ZFS using a RAM disk and flash storage and it really does work - it can make a RAID of slow SATA disks look like a bigger RAID of fast 15krpm disks, provided all your workload fits in the various ramdisk/flash caches.
- sqs 18y agoHow much overhead does it take to record disk latency data in realtime? Does it slow down the disks? I read on the guy's blog that ZFS's write caching meant that end users didn't experience increased latency when the guy yelled at the disks, but surely there must be some performance cost to all this instrumentation.
- seiji 18y agoIt's possible thanks to the magic of dtrace. In a storage appliance, you wouldn't expect the load to be overwhelmingly CPU bound[1], so performing real time instrumentation isn't a noticeable performance hit: http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Overhead http://www.solarisinternals.com/wiki/index.php/DTrace_Topics... [1]: assuming you aren't using ZFS gzip compression: http://blogs.smugmug.com/don/2008/10/13/zfs-mysqlinnodb-compression-update/ http://blogs.smugmug.com/don/2008/10/13/zfs-mysqlinnodb-comp...
- sh1mmer 18y agoI wonder if the kind of "anti-noise" tech used in high end Japanese cars around the wheel arches would be useful in a data center.
- deleted 18y ago[deleted]
- rw 18y agoFor those of us unable to watch the video - what is the explanation?
- immad 18y agoHe screamed at the disks to recreate the graph. Disk vibrations led to latency.
- jbert 18y agoWhile shouting like that is unlikely, I wonder if some data centres are effected from traffic rumble or other sources of vibration. (Having your server farm performance dip when you switch to diesel generation might be a problem). I guess that might manifest as "slow racks" or "slow data centres". It also suggests that rubberised disk mounts (as are used for 'silent' desktop PCs) might sense on server systems too (or are they standard already?)
- sdragon 18y agohighly wild idea: If specific vibration causes specific disk latency, given a high precision intstrument, what are the chances of being able to eavesdrop sysadmins on a co-loc site? ;)
- kogir 18y agoSubstantially lower than if you put a mic inside one of the servers, preferably next to the concealed webcam ;) Also, to directly answer your question: The latency demonstrated is in response to high amplitude noise. For talking you'd need it to be sensitive to the frequency as well. Perhaps you could tell if things weren't going well (lots of yelling)?
- mosburger 18y agoI used to write the firmware that runs the tracking systems for disk drives (specifically, some of my stuff was in the Quantum Atlas III, IV, 10K, and 10K II). This does not surprise me at all. Cabinet vibrations were always a HUGE component of track mis-registration (i.e., the heads going off the tracks). It used to really annoy us when the cabinet and workstation manufacturers would stick our drives in flimsy, crappy components and wonder why the performance would suffer.
- Andys 18y agoEnterprise drive makers like Seagate and WD are now claiming to have features which limit the damage caused by cabinet vibrations. I was intrigued to note that these differences are done only with firmware. Any idea whats happening behind the scenes? I assume Seagate measured the common vibration frequencies and conteract them with head movements?
- gaius 18y agoYou know, I have been suffering from unusual disk latency recently. I'm going to send someone into the DC to check for this. Not howling lunatics mind, other potential sources.
- ShabbyDoo 18y agoDoes anybody make a vibration sensor that could be used to correlate cabinet vibration with latency?
- gaius 18y agoDunno. I plan to use a use a network engineer with a mobile phone and have him walk around the datacentre putting his hands on things while I kick off batch jobs. Tho' now that you mention it, a metal ruler held in his teeth would probably be better.
- spolsky 18y agoDon't hard drives use voice coils to move the arm back and forth?