5 ms·
How Lanyrd moved from AWS to SoftLayer and MySQL to PostgreSQL with no downtime
- Alex3917 14y agoWhile this is technically impressive, if you had gone down in the middle of the night for a few hours would it really have mattered? Apple.com takes down their entire site for several hours at a time every few months, and they don't seem to be hurting too much for it.
- simonw 14y agoFor us, it does matter. We have conferences running all over the world which use us for their schedule - for them, two hours of downtime in the middle of their event is a big problem. Since we're a global site we have to think about events across every timezone so there isn't really an obvious time to do this kind of thing. We might be able to schedule downtime for a period when no active events are going on (tricky considering the number of conferences happening around the world) but it's much easier for us to be able to make these kinds of changes without worrying about events that are using us to serve critical information. As it is, we still make sure to communicate planned read-only mode periods in advance so conference organisers have a chance to plan around them.
- mwsherman 14y agoWould like to know more about the hardware specs, and what Lanyrd is paying over at SoftLayer.
- simonw 14y agoSorry, no details - but we're using a variety of different server specs. We use their cheaper dedicated machines for our application servers (running Django, memcached and celery workers) and significantly meatier servers for our backends running PostgreSQL/Solr/Redis. We also have a couple of smaller things running on their cloud servers.
- neverm0re 14y agoThe hardware Softlayer is offering looks ancient, Xeons from 2006, only a paltry 5GB/mo bandwidth tier as initial offering -- this page looks stuck in time, like what you could get maybe three~four years ago in dedicated hosting. Even Cari.net, which I think prices a bit higher than others is offering more for the money. I've had several dozen machines with them since 2006 without issue, top notch support. I also use the really cheap and no-frills folks like Ubservers for when I want a bunch of disposable cheap dedicated machines. Their support is absolutely shit, but by god are they cheap and the bandwidth real. I've been through probably dozen and some change of these guys and that's really all that matters is cheap solid bandwidth and a vague sense of support.
- boundlessdreamz 14y agoWho are the other reliable providers offer value for money in US ?
- neverm0re 14y agoThere's new players all the time and a lot of mom-n-pop type shops out there that actually can reliably deliver 100Mbps and 1Gbps pipes, though some are certainly shady. I've named the ones I feel comfortable mentioning, but if you really want to know more about what's out there, there's a number of forums like Web Hosting Talk where the actual companies maintain presences to run promotional deals -- and there's a lot of public opinions aired about how said companies are doing. There's going to be some rather uninformed opinions aired, but most people can tell you when there's a real problem.
- staunch 14y agoMy startup does: https://uptano.com https://uptano.com
- X-Istence 14y agoWhat virtualisation tech do you guys use? Can I run FreeBSD on top of it?
- simonw 14y agoNot exactly no downtime... but all site content stayed available throughout the move thanks to our handy read-only mode. We added this early in the life of the site and it's proved extremely useful for a couple of major moves. It's a pretty simple implementation (based on our site-wide feature flags) - once in read-only mode all login cookies are ignored (so everyone gets the signed-out experience) and the sign-in button is disabled. We also return an error for any POST requests just in case someone has already loaded a page with a form before we turned read-only mode on. The moment we've gone in to read-only mode we can create a brand new instance of the site from a copy of the (now frozen) database, make any necessary changes to that, then switch traffic over to the new instance once we've tested that everything is working properly. If the new version of the site has a problem we can turn read-only mode off and continue to run on our original database. If you're running a more content-oriented site it's well worth taking the time to set this kind of thing up - it's not too hard to do, and it gives you an enormous amount of flexibility for maintenance further down the line.
- RakshaC 14y agoGreat tip about read-only mode, Simon! This should help us plan our migrations with zero downtime. I was wondering about disabling writes to databases that are critical to application to support read-only mode with application logic to gracefully handle write failures.
- simonw 14y agoI thought about disabling UPDATE/INSERT statements at the database cursor level (we have our own custom Django cursor wrapper class) but we're confident that it's not necessary with our application - we don't have any writes that we can't afford to lose which aren't triggered by signed in users going to our main application database.
- jballanc 14y agoThis is excellent advice! I can't tell you the number of times I've worked on projects where I wished just such a "read-only" mode existed. As an added benefit if you have a read-only version of your site, I know at least Akamai (and possibly their competitors) has a service where even if your origin server disappears, they will continue to serve the latest version of your read-only site until the origin servers reappear.
- professorplumb 14y agoIf you follow the link to the more detailed blog post, Andrew shows how he was able to speed up the data transformation script by a factor of 60 by using ALTER TABLE statements to cast column types after the fact rather than rewriting every single INSERT. A little more dangerous perhaps, but still an impressive improvement.
- whalesalad 14y agoDid you guys consider Heroku PostgreSQL? It's inside of AWS. If so, can you shed some light on why you opted against it? I don't want to sound like a fanboy of Heroku (I am not) but I am just interested in whether or not you guys considered it.
- simonw 14y agoWe didn't - it's a fantastic product (I'm really impressed with how easy they've made it to set up followers and forks) but we were ready to move to dedicated hardware. It's also a pretty expensive option.
- whalesalad 14y agoThanks for the response. I used to agree with you on the price until I realized that with Heroku you're essentially paying for a dedicated server with that much RAM for each plan. Reason being, every single database is "hot" and the entire thing lives in memory to avoid I/O. On second glance ... you're right. It is kind of pricey. Speaking of moving to dedicated.. I'd love to hear some of your general feedback on how AWS has been at some point. Essentially where you'd use it again or where you'd go directly to dedicated (or a diet option like a VPS) if you were starting on a new project.
- gsibble 14y agoI've worked in similar environments and Heroku is crazy expensive and slow compared to dedicated hardware.
- jaequery 14y agosad thing is, most people don't realize how slow they are. once they move off to dedicated w/ SSD, their site starts to run blazing fast (plus at a fraction of the cost) and say to themselves why they haven't done this sooner.
- gsibble 14y agoIndeed. Most companies don't have the operational experience to optimize for speed until they are later stage. Also, they will develop on something like Heroku and get addicted to the ease of use.
- acangiano 14y agoI've been using Softlayer for years now and have only good things to say about them and their support team.
- nigma 14y agoHave you considered using PostgreSQL foreign data wrapper (mysql_fdw) to pull data directly out of MySQL instead of the dump/load process? It would be great to know more about your new setup, i.e. do you use streaming replication and some resource manager like Pacemaker?
- simonw 14y agoNo, we didn't look at mysql_fdw. We're using standard PostgreSQL 9 streaming replication.
- bfirsh 14y agoWe're looking for hosting which has both dedicated and virtual servers on the same network. SoftLayer and Rackspace seem to be the two big players who do this. Does anybody have any experience with them both?
- bchen 14y agoI don’t have experience with Rackspace, but, like the other commenter acangiano said, I only have good things to say about SoftLayer’s support for the past 4 years. Their customer service reps are quite responsive and can be reached either using tickets or phone. My issues were mostly addressed within an hour or two, even when it’s not during the normal office hours.
- gsibble 14y agoI've used both. SoftLayer in my opinion has faster and more price efficient offerings. Always gone with them and never regret it.
- larrywright 14y agoI only have experience with Rackspace (though I've heard consistently good things about SoftLayer), but my experience with them has been fantastic.
- misterbwong 14y agoRackspace user here. Be prepared to pay $$$$-their hybrid environment is not cheap, especially for servers with above normal specs. There are also (industry standard) restrictions on the network side of things with their VM's, which we weren't prepared for when we started. Lastly, if you're ever the target of DDoS attacks...good luck.
- druiid 14y agoPrevious Rackspace user here (we got fed up with Rackspace and finally got our own equipment for a song compared to our monthly hosting costs with them)... Yeah, good luck if you get a very common DDoS attack while on their network. Apparently their data-centers in general are large DDoS targets. We moved and have not had an attack since (and it's a big deal since we do e-commerce). That said, out of any company I've ever been with their support was really good. Just understand you're paying like crazy for it.
- gsibble 14y agoI have used SoftLayer for years and have nothing but great things to say about them. Nothing but speed, stability, and excellent service. Thank you SL!
- knodi 14y agoWhat is the size of Lanyrd data? I'm curious to see what your think is considered "big data".
- rpicard 14y agoHe refers to "our tens of gigabytes of data" in the post on his site. [1] [1] http://www.aeracode.org/2012/11/13/one-change-not-enough/ http://www.aeracode.org/2012/11/13/one-change-not-enough/
- StavrosK 14y agoI remember a quote from some presentation: "If it fits on an iPod, it's not big data."
- genwin 14y ago> There's nothing wrong with AWS - indeed, we still run a staging environment there - but our database benefits greatly from the low latencies of physical disks... Doesn't AWS have physical disks for RDS? What am I missing?
- friism 14y agoAWS RDS uses EBS for storage[1]. Depending on RDS instance type, I think they use multiple raid'ed EBS volumes. EBS relies on physical disks, of course, but they're accessed as SAN devices over the network and that means higher latency than accessing disks that are hardware-attached to the local instance. EBS has historically also had somewhat variable performance (on top of the extra network latency), but the new provisioned IOPS feature should help with that[2]. [1]: http://aws.amazon.com/message/65648/ http://aws.amazon.com/message/65648/ [2]: http://aws.amazon.com/about-aws/whats-new/2012/07/31/announcing-provisioned-iops-for-amazon-ebs/ http://aws.amazon.com/about-aws/whats-new/2012/07/31/announc...
- dangrossman 14y agoNo, RDS runs on a virtual RAID array of virtual disks over a network (EBS).