9 ms·
Sysadmin mistakes start-ups make
- DrJokepu 17y agoHere's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.
- pquerna 17y agoGood lesson that RAID != Backup! RAID5 by chance?
- e40 17y agoARRRRRHGH. That is NOT THE LESSON. I'm sorry, every time there is a discussion of hard disks and RAID, someone posts this same stupid comment. The GP comment didn't suggest that RAID == Backup. Not in any way. Your reply suggests that they did. "RAID5 by chance?" How do you know they didn't use it? They might have had multiple failures too close together to rebuild the array. Sorry for the caps, but I just snapped seeing this comment for the nth time and with so many upvotes.
- michaelbuckbee 17y agoThat isn't something I'd ever thought about until now. Would it make sense to use drives from more than one company as they would very different failure characteristics?
- vidarh 17y agoPossibly. IBM had the infamous bad run of DeskStar (or was it another model?) drives about 8-9 years ago... We got a batch of them - in that case it wouldn't have helped you to buy them from different distributors or otherwise tried to get a different batch as the number of problematic drives was huge. At least they were extremely good about replacing them no questions asked and we got them all replaced before we lost data.
- astine 17y agoDeskStar (or was it another model?) DeskStar is right. They earned the nickname "DeathStar" because they failed so often.
- wmf 17y agoWould it make sense to use drives from more than one company as they would very different failure characteristics? This is probably difficult to take advantage of unless you setup your RAID layout very carefully.
- blasdel 17y agoIt's a terrible idea to mix and match drive models, because they'll have drastically different performance characteristics, and if you're lucky you'll only get a little worse than the lowest common denominator on all axes. What quality OEMs do is make sure they never ship you drives from the same manufacturing batch in one enclosure.
- DTrejo 17y agoHow do you make sure to buy from a quality OEM?
- jbert 17y agoI've thought this for a while. Ideally, you want to have drives of different ages, so they're at different pts of the 'bathtub curve'. If you're mirroring, you should only ever mirror a "fresh, unproven" disk with an "old stalwart" disk. Doing that also means that when your "old stalwart" becomes senile, it's paired with a younger disk. But doing that does mean rotating mirror sets when you buy a new tranch of disks. Which does put load on, which can trigger failure. Anyone have a good plan for doing this?
- tesseract 17y ago(In theory, as I don't run a storage farm) RAID6 might help here since if the disk rotation triggered a failure you'd have a second parity disk to fall back on. However ideally all the disks in an array would be from different lots. Or, I believe that on some RAID controllers it is possible to add a third disk to a RAID1 pair, which means you could build the new disk before removing either of the old ones, thus there is never a single point of failure even during the disk replacement operation.
- skorgu 17y agoLegend has it that one of the early ESS systems ran into something like this. Telephone switching systems have functionally zero downtime. They're designed to be fully modular, entirely hot swappable, the kind of thing Erlang was built for and give Z-series a run for their money. So you have this hulking room-sized[1] brute of a telephone switch which cannot ever fail and if it does so must always do so gracefully and with plenty of warning. And one day it just falls over. No warning, no graceful failover to a redundant system, just poof gone. After much wailing and gnashing of teeth the root cause is identified: n drives were capable of failing without issue, n+1 failed simultaneously. At this point, stories differ: this was either the beginning of a "no two drives from the same manufacturer" policy or the end of the career of a PHB who vetoed said policy on grounds of excessive cost. [1] http://www.montagar.com/~patj/phone-switches.htm http://www.montagar.com/~patj/phone-switches.htm
- joshu 17y agoYou frequently see RAID cascade failures. Drive A fails. You pop it out, put in a new disk. Machine starts rebuilding RAID array. This involves a ton of reads from other disks. Under the increased strain, Drive B, which was already on the edge, also fails. And so on. I'm no longer a believer in redundant disks. Redundant machines would be better.
- tptacek 17y agoThis was a great article, but I ended it wondering whether they either (a) knew what a system call was (until the end, I thought maybe they meant a system() shell-out) or (b) realize how many system calls a vanilla request/response cycle incurs.
- patio11 17y agoIf you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application. I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning. Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently making to system(big_command_to_invoke_imagemagick) ?
- polvi 17y agoThe solution to use an image processing library such as RMagick, http://rmagick.rubyforge.org/ http://rmagick.rubyforge.org/
- tptacek 17y agoCalling into RMagick/ImageMagick from inside the request/response cycle is probably even worse than shelling out, because ImageMagick does grievous damage to your runtime.
- polvi 17y agoI guess it all depends how you design it and what you are doing. I would have to agree with others, the out of request cycle image processing solutions are definitely the right way to go overall.
- Pistos2 17y agoLast I tried RMagick, it leaked significantly. Definitely not something I want to use in a long-lived process. I remember having to fork to use it, to work around the memory leak. If you don't need the fancier operations, there are lighter image manipulation gems out there that do just the basics, but without leaking. e.g. ImageScience http://seattlerb.rubyforge.org/ImageScience.html http://seattlerb.rubyforge.org/ImageScience.html
- boundlessdreamz 17y agoUse a queue. You should never be doing time consuming method calls inside a controller anyway.
- aristus 17y agoI disagree with 1.3. "Serving static content is the easiest possible task for any web server." Yes, but keeping connections open for slow clients (esp with KeepAlive on) is not a good use of your 500MB Mongrel process' time. On the other hand, KeepAlive is a handy thing to have. Using a proxy like nginx or varnish to serve static files (and even dynamic data) if you have the proper KeepAlive and Nagle bits flipped can save you a lot of server resources at the application layer.
- tptacek 17y agoApache disables Nagle by default, which is what you want for small static files, but I'd love to see data showing that Nagle is actually a significant performance issue for a realistic load.
- aristus 17y agoYou're right about Nagle; I mention it only because lighty or one of the others does not turn it off by default. Having a lightweight proxy that keeps connections alive on the client end but cuts them off between themselves and the application layer is the bigger win all round for many real-world web loads.
- staunch 17y agoIt's almost always a bad idea to use anything other than a non-blocking/async server to handle static content. I think it's simpler/easier (maybe faster) to serve content from a separate sub-domain (static.site.com or whatever). Using a reverse proxy works too, but unless you're caching dynamic content it's probably no benefit and it's less efficient.
- andrewtj 17y agoA good reverse proxy will buffer client and server side so that your heavy app can be available to serve the next request whilst the light proxy feeds the page back to a slow client. Under certain circumstances serving static files from separate hostnames can be beneficial as HTTP clients are supposed to limit the number of simultaneous connections per hostname.
- peterwwillis 17y agoThe memory use is not accurate unless you take shared pages into account. Copy-on-write will make it look like each apache child is using 40MB, when really it's only 10MB private RSS. Use a RSS-calculating script (http://psydev.syw4e.info/new/misc/meminfo.pl http://psydev.syw4e.info/new/misc/meminfo.pl) to determine the close-to-real memory use. If you don't calculate your maximum memory use correctly you will run into swap with traffic peaks. Also keep in mind that swap is a good thing. Is your app constantly cycling children? This isn't going to allow it to move unused/shared memory into swap. Don't ignore memory leaks by reducing your max requests per child. The forking thing is more of the same. Copy-on-write means it's not going to balloon your memory unless some function turns that shared rss into private. It isn't something that you want to do a lot of, though.
- Periodic 17y agoThis stood out to me as well. I like the script to actually calculate real usage. Modern operating systems are smarter than I am when it comes to memory management. What you don't want is to have anything you use more than once a minute in swap, and preferably only the stuff you don't plan on using for an hour (i.e. not any time soon). That probably means you want your main application and web server in memory all the time. If there are pieces of it that are unused and you're hitting a resource cap then you have something mis-configured. RAM is also dirt cheap right now, making it often easier to add RAM than to optimize slightly sloppy code.
- rythie 17y agoThis seems like an odd section of sysadmin mistakes - I would have thought there are some other ones being made more often.
- thwarted 17y agoEspecially since the third one is a developer mistake that, as a sys admin and developer, I've had to point out to developers not to do -- but for security reasons, not because fork is oh-so-super expensive (even though it can be). Also, there is no "system" system call. "system" is a library call that forks and execs a shell to evaluate and execute a string. Having a sys admin that doesn't know the difference may be the biggest sys admin mistake you could make. There are a lot of library wrappers for system calls, but these are documented in section 2 of the man pages as system calls.
- ajross 17y agoFTA: However, sqlite should never be used in production. It is important to remember that sqlite is single flat file, which means any operation requires a global lock I don't know jack about sqlite's locking architecture or scalability, but this statement is just silly. There are a conceptually infinite number of ways to make fine-grained locking work on a single file, both within a single process, a single host, or across a network. Maybe the author is thinking fcntl() locking is somehow the only option. I guess the corrolary to this article has to be "Don't let your startup's sysadmins diagnose development-side issues."
- pquerna 17y agoSQLite locking: http://www.sqlite.org/lockingv3.html http://www.sqlite.org/lockingv3.html """ An EXCLUSIVE lock is needed in order to write to the database file. Only one EXCLUSIVE lock is allowed on the file and no other locks of any kind are allowed to coexist with an EXCLUSIVE lock. In order to maximize concurrency, SQLite works to minimize the amount of time that EXCLUSIVE locks are held. """ But compared to something like MySQL w/ InnoDB (or postgres, or Cassandra, or BerkeleyDB), which all have something closer to Row Level or Page Level locking, SQLite's concurrency for server side applications is a serious deficiency. Yes, there are lots of ways to have fine grained locking, SQLite just doesn't do them.
- silentbicycle 17y agoLike many of SQLite's other quirks, this is because SQLite is designed to accommodate embedded usage.
- peterwwillis 17y agoI guess the corrolary to this article has to be "Don't let your startup's sysadmins diagnose development-side issues." You'll have to add "Make sure your developers can diagnose development-side issues" to the list as well. Most web app developers I have met do not know how to diagnose problems, or simply defer immediately to the sysadmins if there's no syntax errors or logs to refer to.
- lsb 17y ago
- michaelbuckbee 17y agoI'd guess the real number one mistake is insufficient paranoia about backups. I know lots of companies doing TDD but that have never done a full test restore from their backups.
- gaius 17y agoThe easiest way to do this is to make your backups the mechanism by which you refresh your Dev/QA environment from Production. It means your Ops team are very nearly doing a DR exercise every week.
- absconditus 17y agoHow are the last two system administration problems?
- SwellJoe 17y agoOne of the most common problems we see is DNS misconfiguration. It seems most folks just haven't read the grasshopper book. If you're doing anything on the Internet, you need a basic understanding of DNS. Once you grasp the fundamentals, most DNS problems become completely transparent, but I've seen people spend weeks trying to solve DNS problems due to lack of understanding.
- there 17y agodns is the cause of many seemingly unrelated problems. some services (like sshd) do reverse dns lookups on connecting ips. a misconfigured dns server somewhere (or improper delegation) along the path can make this initial connection take up to 30 seconds while waiting for dns timeouts. it may look like an extremely slow/busy server, but in reality it's just sitting there doing nothing waiting for a dns reply. tools like dnstracer (http://www.mavetju.org/unix/dnstracer.php http://www.mavetju.org/unix/dnstracer.php) and dig are very useful for diagnosing these issues, but you really need to understand the fundamentals of how dns and delegation work.
- SwellJoe 17y agoYes, I should have been emphatic that part of the problem with not understanding DNS is that if you don't understand it, you might not even realize that your problem is DNS-related. Whenever I hear, "X is slow!", my first response tends to be, "Are you sure it's not DNS instead of X?" About 50% of the time, DNS misconfiguration is at least a component of the problem if not the entirety of it. DNS touches every service on the Internet. If you get it wrong, you break every service, sometimes in subtle ways.
- anApple 17y agoThanks, we just got a fixed ip at work and I was wondering why the initial connection to our servers took longer than usual. Our new ip doesn't have a dns entry.
- Huppie 17y agoI'm not sure which book you are talking about but my guess would be "DNS and Bind". http://oreilly.com/catalog/9780596100575/ http://oreilly.com/catalog/9780596100575/
- toisanji 17y agohmm, I have a hard time understanding why anyone would try to use sqlite in production unless they explicitly wanted to?
- anApple 17y agoSimpler to use (no external program to start and monitor) and to backup (just copy the sqlite file)
- julio_the_squid 17y agoYep, #1 happened to me the other day. We hit our Apache server limit of 256 and the site slowed to a crawl. I'm not really sure what was causing the load to be like 50-90, but requests were quite delayed waiting for an open process (keepalive was at 5 secs). Indeed, my first idea was indeed to install nginx for images really quick. However, I have no experience with nginx. Thankfully, we had a spare server and I offloaded the images to there for now... Throwing more hardware at the problem usually works.
- jrockway 17y agoFork is actually a very fast system call. It never blocks, and (on Linux), only involves copying a very small amount of bookkeeping information. If you exec right after the fork, there is basically no overhead. However, forking a new shell to parse "mv foo bar" is more expensive than just using the rename system call. And it's easier to check for errors, and so on. SQLite is also not as slow as people think it is; you can easily handle 10s of millions of requests per day with it. If your application's semantics require table locks, MySQL and Postgres are not going to magically eliminate competition for locks. It's just that they both pick very weak locking levels by default. (They run fast, but make it easy to corrupt your data. Incidentally, I think they do this not for speed, but so that transactions never abort. Apparently that scares people, even though it's the whole point of transactions. </rant>.) Most of my production apps are SQLite or BerekelyDB, and they perform great. I am not Google, however.
- rosser 17y agoActually, PostgreSQL's MVCC architecture makes lock contention radically less likely than with most other DBMSes. For SELECTs, you're only ever taking "Access Share" locks on the table ("Hey, I'm using this table; you can't DROP it right now."). For DML queries (UPDATE, INSERT, DELETE), you'll see those, plus "Row Exclusive", which is just what it sounds like. There are a few other lock types you'll run into as well, but they're typically only seen in narrow, specific cases -- when you're performing maintenance (vacuuming, clustering, &c), indexing, DDL changes, or when you explicitly LOCK a table for whatever reason.
- teej 17y ago> SQLite is also not as slow as people think it is; you can easily handle 10s of millions of requests per day with it Scaling relational databases to the point of 10s of millions of requests is extremely non-trivial. Unless you can show me personally or can show me evidence otherwise, don't make this claim. You're doing a disservice to the people that have worked countless hours to eke every last millisecond of performance out of MySQL and Postgres.
- jrockway 17y ago
- abalashov 17y agoIn my experience of the most common mistakes is the failure to realise that on pretty much all Linux distros, services like Apache and MySQL come conservatively tuned. This is deliberate; it means a DoS or out-of-control process within one of those domains is unlikely to take out the entire server, because there's a hard limit on consumption of memory, CPU, child processes, threads, etc. However, this default configuration needs to be tuned to allow you to take advantage of the hardware - if you have generous hardware. Otherwise, you will wonder why your web sites are extremely unresponsive, yet the server load stands at something relatively unimpressive. I found this out the first time a blog post on one of my servers got digg'd.
- joshOiknine 17y agoPersonally I don't think we would ever run into those issues. A. we don't have other servers to switch over to B. We are using MySQL for testing and development and C. we don't like what happens when we make system calls for within a web app. forget about forking.
- bcl 17y agoI'd say their biggest mistake is usually not hiring a sysadmin who also has development experience (or developers without sysadmin experience). I've found that my knowledge in both realms has been invaluable in determining how to design the infrastructure and how to write the code.
- durana 17y agoTake #1 and generalize it to the mistake of trying to fix a problem without really understanding what the problem is. This has to be the most common mistake I've seen in the sysadmin world.
- imtotallygay 17y agoNIGGERS!!!!!!!!!!!!!!!!!!!!!!!
- c00p3r 17y agoIs this a example of a knowledge level of modern sysadmin? If so, we're in trouble. =) Sysadmin should be able to think in terms of data flows, which means memory management, data partitioning, and network stack usage, able to put different types of data into different kinds of storage, and understand the role of cache and how data should be access. Packages are just a tools.