9 ms·
Joyent: 54+ hours of downtime
I am a customer of Joyent.
For the past two days, the server on which I host some customer websites and mail servers has been down. I did not receive any warning or explanation by email or telephone. When I filed a support ticket, I was given a link to their help site with some status updates.
Status updates (paraphrased):
2011-02-26 09:22:19 GMT: server offline
2011-02-26 14:38:38 GMT: no ETA
2011-02-26 17:09:42 GMT: estimated downtime 12+ hours
2011-02-27 22:23:08 GMT: estimated downtime 20+ hours
That final estimate of 20+ hours is in addition to what is now 37 hours of downtime. Joyent has offered no reparation or backup service (e.g. backup MX service to keep mail from bouncing).
For your consideration when choosing hosting providers.
- moe 16y agoOuch.
- sid_g 16y agoI remember that chunkhost has really aggressive pricing.. And I remember there service being pretty good...
- jasonhoffman 16y agoHe's currently at "free", so get any more aggressive than that.
- enduser 16y agoPaid for lifetime service in advance to help bootstrap your company, not "free".
- asuth 16y agoWe're Joyent customers as well, and their support has always been great, so this is definitely surprising. You should get on the phone with them (they provide an emergency number). What kinds of machines are you running? Is it just one, or many? It shouldn't be necessary, but message me if you're still not getting through and I can ping some contacts.
- jasonhoffman 16y agoThank you, we actually take our support very seriously and we're pretty accessible. He's talking about a different product than what you have.
- payingcust 16y agoNo you don't, I asked via a support ticket two days ago for an estimate, I can't even get a response from the support team. S I am frustrated enough to post here
- kylecordes 16y agoI generally like Joyent, and use them. So I offer this tip with a tinge of sadness: It looks like these message are from a problem with their Howe server. That is one of their old servers, which came over from the Textdrive merger years ago. I used to have some sites hosted on that same server, and was likewise not very happy with things. It appears to me that Joyent as a company doesn't care very much about that legacy equipment and line of business. The way to make the pain stop, is to stop using Joyent's old TextDrive servers. You can do that by leaving Joyent entirely, or by moving to their newer stuff. Either way it's a fair amount of work, depending on the complexity of what you are hosting. I took the path of moving to their newer stuff, and have been very happy with Joyent ever since.
- iskander 16y agoI had a lifetime account with TextDrive. I don't know if it was official policy at Joyent to neglect the old customers, but I certainly felt like I was being pressured to repurchase services I had already paid for. After several outages, data loss and extremely slow responses from tech support I switched to linode. No problems since.
- jefe78 16y ago+1 for Linode. I've literally never had a problem. I'm a huge fan and I'm picky as hell.
- jasonhoffman 16y agoNot comparable for this.
- jefe78 16y agoAgreed. They're better.
- jacques_chester 16y agoI had a server go down twice due to outages at their Fremont data centre, but otherwise I've been very pleased.
- jsdalton 16y agoI had some issues with a Textdrive server recently as well and migrated over to one of their smart machines (at a lower price than what I had been paying). I was a bit frustrated like you, but Joyent was extremely, extremely helpful in assisting me with the migration and now everything is running smooth. I would try to touch base with them again and see if they can help you out.
- jasonhoffman 16y agoAnd there's no reason that you needed to pay additional unless you bought a different product.
- itsnotvalid 16y agohttp://www.joyentcloud.com/about/policies/cloud-service-level-agreement/ http://www.joyentcloud.com/about/policies/cloud-service-leve... For clause 2: 2. Service Level Goal: Joyent’s goal is to achieve 100% Availability for all customers. Remedy: Subject to Sections 3 and 4 below, if the Availability of customer’s Grid Container is less than 100%, Joyent will credit the customer 5% of the monthly fee for each 30 minutes of downtime (up to 100% of customer’s monthly fee for the affected server). So probably you can get this month for very little (Takes only ten hours for getting full refund if SLA is signed) But at the same time their exceptions list are pretty long, things like emergency maintenance and upgrades are considered as exceptions.
- deleted 16y ago[deleted]
- jasonhoffman 16y agoWe do have a strong SLA for our IaaS and PaaS offerings. But this thread is about a 6 year shared hosting product, that we've grandfather'ed in and he doesn't even have to pay for anymore. There's nothing to refund.
- swombat 16y agoSounds like the OP is getting the service for free anyway...
- officemonkey 16y agoIt wasn't "free". We bought a "lifetime" package. I don't recall the details, but I ponied up a couple hundred dollars at a time when we didn't know textdrive was going to make it. I don't expect a lot of support, but I do expect uptime and courtesy.
- itsnotvalid 16y agoOh I see... then, I just wonder how they could eat up for life-time subscriptions for so long. After some Google searches, I've found that those plans are one off, and seems to cost $799 to sign up (http://hostingfu.com/article/joyent-lifetime-accelerator http://hostingfu.com/article/joyent-lifetime-accelerator). If the terms was not changed, it now cost $25 a month on JoyentCloud to have the same thing with the SLA I mentioned. So it takes 31 years to compensate the cost of the $799 lifetime subscription to pay off. But the fact is that now they have "10 TB Monthly Bandwidth" (or unlimited depending on which web page you are looking at), I am just blindly guessing that lifetime users would get that too, right?
- andrewvc 16y agoI remember trying to get a quote / get a few details about buying large quantities of bandwidth from Joyent a couple years back. They consistently took days to answer simple questions via email. I generally take it that if a company can't even get its sales people responding quickly there's a snowballs chance in hell it's techs will.
- jasonhoffman 16y agoIf it was a couple of years ago and for large quantities of bandwidth, then you would have been talking to me. We didn't have "sales people" a couple of years back. And I would have said "No" because we weren't in the business of selling "large quantities of bandwidth"
- netik 16y agojoyent had this sort of failure regularly about 4 years ago and continues to do so. Avoid.
- jasonhoffman 16y agoAlso incorrect. "About 4 years ago" we always did >99.99% uptimes even on the old textdrive stuff. I'd be happy to show all the old stats and graphs, we still have them.
- iskander 16y agoJason, you're experiencing a PR hiccup; it's not a big deal but it's also not clear that responding to every message in this thread is going to help you. Instead, why not (1) make things right for the OP and everyone else on Howe and (2) put a response on the blog? You Joyent folks will never get a subset of your customers to migrate off textdrive hardware (so it's no use to keep yelling "just migrate!"). However, if you let those older servers fall apart you'll only get more negative PR.
- jedsmith 16y agoI get having a passion for your product -- I really do. Some of the responses here from Jason are on the border line, though, and I think they are making the situation worse overall. I wish, for his sake, that he'd have quit while ahead. Without a doubt, Jason has the best intentions here, but I'm getting a really bad vibe from reading the entire thread -- particularly the by the way, you contributed to a ticket and I can look it up so I'm going to remind you so that you'll look stupid for bitching about us bit. The hostile you-are-wrong attitude, while maybe technically correct, is a disaster for outside perception.
- jonto 16y agoIf it were me? I likely would have EOL'ed those FreeBSD boxes a long time ago. Customers, even the life timers, should not have been given a choice. Why? Textdrive promised lifetime hosting. Where does it say they promised hosting on the same box for that entire lifetime? TxD/Joyent has been gracious to provide hosting on their new platform. One could argue that the 'lifetime' was referring to the actual box, and if the downtime is increasing, the lifetime of the server is ending. I say pull the plug and force the migration. Granted, it could be a small PR hit.
- frankwiles 16y agoI have one single client who refuses to move off Joyent. My experience with them has been less than stellar. Frequent problems with no explanations, ask a support question like "How does one do X on your system?" and often get "oh you want to do Y I see, here is a link to the support forum for that." and the link isn't at all related to X or even the Y. They aren't horrible, but I would never use or recommend them to anyone.
- jasonhoffman 16y agoIf you're on an old textdrive server, then you need to move, like we've been saying.
- deleted 16y ago[deleted]
- jasonhoffman 16y agoJust can't win. I am reading all the comments, and I'm not spamming the thread. I'm having a conversation on a site that I frequent everyday. This is called being responsive.
- deleted 16y ago[deleted]
- payingcust 16y agoTry answering your support tickets instead of this forum, might make me less disgruntled.
- amock 16y agoHave you considered offering to buy back the lifetime offers from people who won't migrate? It seems like that might be cheaper in the long run than having to support the old BSD servers forever.
- tylerritchie 16y ago
- jasonhoffman 16y agoI get the frustration. I really do and I apologize for any bump like this. I'm the founder of those old textdrive "share hosting" servers (they really have nothing to do with Joyent's current product line). These servers are ~6 years old and they're largely populated by "lifetime" customers who have tricked out various bits of their accounts enough that an automated migration has been basically impossible. We've been telling everyone for 3 years to really get off of them, we've been offering free upgrades for 4 years, including you btw, you got a free upgrade back in Nov 2007, and there's even support tickets detailing it (which you've participated on). So now we're piecing together everything that was left but those good ole FreeBSD UFS file systems take awhile.
- enduser 16y agoOffering a transition from FreeBSD to Solaris that requires the effort of the customer is not free. Asking a customer who knows FreeBSD to take on administration of a Solaris virtual server for your convenience is not reasonable. You are offering an unacceptable level of service to customers who have supported and recommended you from your early days. You were paid in advance for a lifetime of service. I took you on your word and offered lifetime hosting to friends with small websites for small businesses. If you can't make good on your agreements, I will eat the cost of moving these sites to Rackspace Cloud. I can't risk another 54 hours of downtime. With proper backups and a fresh server it should not take 54 hours to get back online.
- jasonhoffman 16y agoTo even call this "Joyent" is misleading. The packing system is the same, it's BSD userland. The paths are the same. Everything is the same. The kernel shouldn't matter. And if you had told support that you wanted us to just migrate you because nothing was out of the ordinary, what would they have done? They would have migrated you .... And this has been quite a good deal for you. The logic of moving from something that costs you nothing to something that does costs you something is interesting.
- veidr 16y ago
- kenmck 16y agoI have a lifetime account as well. Joyent's tech support has always been excellent. I take this as a wakeup call to get my stuff off of the old severs beyond that I can see little fault with Joyent.
- kenmck 16y agobtw. Elapsed time between opening the support request and an answer from someone at Joyent - 9 minutes... Elapsed time from opening support request to receiving account details for an account on a new machine about 26 minutes. That includes the time it took me to get back to them about some details of the account.
- soulclap 16y agoHaven't read all the comments but this really seems like a 'special case' being an exception and does definitely not apply to the experience and quality you'll usually get on Joyent. I have been with them for a couple of years and the only reason why I left was because the project I used to host there was put on hiatus and I don't need that kind of 'enterprise' hosting for my other stuff. I agree there shouldn't be downtimes on any of their machines but just going by the headline, you'd think Joyent is unreliable when indeed their own server plans are rock-solid. You make it sound like they are Dreamhost. That said, hope your problems will get solved soon.
- enduser 16y agoI will edit my original post as soon as they solve the problem. No response yet, other than excuses and denigration from Jason Hoffman (Joyent founder) in this thread. Edit: I can no longer edit the original post, but I will post a comment.
- oziumjinx 16y agoT minus 10 minutes before this hits TechCrunch
- shareme 16y agoThe lesson jasonhoffman could learn: If you have a small set of users who bought lifetime accounts to bet on a company you founded than even with new company I would bend over backwards to make sure their needs are met as a thank you. Otherwise some gets the bright idea to put up a joynet sucks web page with proper Google approved SEO and makes this little PR bump a big EFFING MOUNTAIN!' jasonhoffman, I imagine you have less than 12 hours before that happens..as TechCrunch may do it for 'free'..:)
- tedjdziuba 16y agoParse Error.
- jarin 16y agoThere seems to be some confusion in this thread about the difference between "free" and "paid for in advance". Cutco will re-sharpen your knives for free, no matter how old they are, even if you bought them on eBay, because they made a 'lifetime resharpening' promise with the original purchase. And they're basically a multi-level marketing scam.
- milod 16y agoIt sucks that your hosting is down, but this is a pretty malicious use of HN considering you paid $200 over 6 years ago for lifetime hosting. Less than $3 per month by now. What SLA were you guaranteed? The lesson for service hosts, NEVER offer lifetime anything.
- nphase 16y agoI'm sure I'll get down-modded for saying this, but on WHT people will tell you that you get what you pay for, which is in stark contrast with what most people here are saying (lack of gratitude for being a charter customer, etc). Sure, you paid $200. But if you've been a customer since 2007 then you've basically paid the same price as any cheap shared hosting provider. Contrast this experience with Layered Technologies that ran three price hikes and then when I still didn't leave the server I had with them essentially told me to fuck off once their harddrive failed. At least jasonhoffman had the courtesy to tell people their best option is to switch servers to avoid trouble. But, like they say, you get what you pay for. If lifetime support and stability is what you'll come to expect, don't pay bottom dollar prices up front. Disclaimer: I am not a Joyent customer in any way.
- Klinky 16y agoThe better option would be for Jason Hoffman to say that he stands by his offer & will ensure that those who took advantage of the TextDrive deal will continue to see the same level of service or better that they received back when they bought it. Advertise your new services fine, but don't try to upsell people anytime there is downtime & don't go around telling everyone that these people are whiners getting their service for "free".
- jaaron 16y agoOkay, I get that migrating costs time and that's definitely not free, but saying that Jason upselling is false. It's been repeated by a number of commenters here and it's starting to bother me. I'm a lifetime Joyent (TextDrive) member. When Joyent started offering newer shared hosting accounts, they allowed me and all other lifetime members the opportunity to migrate. No new fees, no hassle. Just a nice, shiny new space. To me, it's a testament of their support that they didn't try and keep all the lifetime members stuck on the old FreeBSD system. I also find it unrealistic to expect lifetime customers to never have to migrate from one server to another. Do you really expect the server to run for 20 years without ever having to do anything yourself? Seriously, take some responsibility for your site and move to the latest supported service.
- fingerprinter 16y agoFOR JASONHOFFMAN I see you posting quite a bit here...you keep saying the same thing: "take the free upgrade that is being offered". There also seems to be quite a few old textdrive lifetime members here. Perhaps the best approach would be for you to outline the steps for those textdrive folks to migrate off of those servers and take the free upgrade you are offering. I personally have no idea how...so if you could explain it, give a link or some concrete information it would be helpful. I for one would be happy to migrate if it means better servers, I just need to know how...I'm sure I'm not alone.
- tylerritchie 16y agoFrom one of the very many forum posts on the exact topic: "If you have not sent in a request and would like to get a jump on the migration, please send an email to support [at] joyent [dot] com with the following information: - Subject line: migration request + plan type if known (startup, plus, premier) - Full Name - User Name(s) / Primary Domain(s) (please note new info if you want to change it at this time) - Existing Server(s) If you wish your username or domain name to be changed, please include this in the email and we will process when we send your golden ticket. Waiting until after your account is set up will incur a $50 fee and you will receive a brand new account (old one will be blasted). Please do not send follow up requests as it only slows the process down. If you asked questions when you sent in your migration request, we will answer those when we send your golden ticket out. Once you receive your 'golden ticket', you will have 60 days to migrate to the OpenSolaris environment. If you need the second account for more than 60 days, regular monthly charges will apply."
- fingerprinter 16y agoThank you for that. I find it a tad bit odd that I received NO emails on migration. Frankly, there is no conceivable way that announcing in a Joyent forum and expecting users to become aware of it is a sound strategy. Pull vs Push and all that. Even so, I'm willing to migrate if I can find some additional information as to what is involved. Again, thanks for posting...I'll go digging through Joyent's forums now....
- desigooner 16y agoIs there any time that Mr. Hoffman's going to call it quits on this thread and actually take it offline in an attempt to resolve the OP's issues?! Frankly, the tone of most of his comments have been borderline denigrating and it's doing nothing but adding nuisance value to the thread.
- ericd 16y agoHe's probably getting exasperated by the virtual lynch mob which seems to be downvoting everything he says and upvoting every angry comment, which is ridiculous. He's actually saying quite a lot of reasonable and useful stuff which is sitting at 0 or below.
- pygorex1 16y agoYou get what you pay for. This isn't a glib statement. A business transaction is about more than coming to agreement about the price of a thing or service. It's also about creating and maintaining incentives for both sides. I pay slicehost for a VPS. They get $50/month from me in return for disk space, CPU cycles, uptime and support. Our incentives are aligned. With Joyent/Textdrive, every month you get hosting and every month they get nothing. The incentives are not aligned. In fact, from Joyent's perspective, the incentive to provide any meaningful service diminishes over time. No about of cajoling or complaining or threatening Joyent will change this imbalance. Sure, it's 54 hours of downtime today - but I can guarantee that a year from now the outages will be far longer and far more frequent. "But we had a deal!" Yes, you did. And you were a fool to enter into it. Here's the bottom line: Your customers don't care what's causing the downtime. It's still your fault. Website users don't care that the deal has soured. It's still your responsibility. Joyent doesn't care that they're providing cr@ppy service to Textdrive users, despite the postings on HN by Jason Hoffman (hey Jason, instead of wasting time posting on some internet forum, why not spend that time fixing the actual problem?) Seriously - Joyent doesn't give a sh?t about the lifetime accounts. If they did, then why are you experiencing 54 hours of downtime? If they do care, and are applying resources to solve the problem, then why is it taking so long? 54 HOURS TO FIX A SERVER. That, to me, is evidence of a shocking level of incompetence. In either case I would get the hell out of there.
- dasil003 16y agoFrankly, I'm surprised you are still on there. I too was an original VC and I left years ago due to the support failings. Look, we were investors in the truest sense. TextDrive was this revolutionary new shared hosting platform with a lot of promise (flexibility, no overselling, etc). Problem was that shared hosting is a commodity with no profit margins. Since TextDrive was started by people who want to build cool shit, rather than people who knew anything about operations in a customer service industry, the result is a massive series of pivots that left us out in the cold. I'm not making excuses for anyone here, personally I think Jason Hoffman spent way too much time in the forum sucking up the love, and not enough time figuring out how to run a support operation. So he burnt his bridge with me as a customer. I won't touch Joyent with a ten foot pole no matter how good it is. But here's the thing: that was their bridge to burn. It doesn't matter what 200 early customers think, because those 200 one-time payments were not going to build a successful company. The bottom line is that the market for a "premium" shared hosting service got squeezed by commodity hosting on one side, and VPSes on the other. Even if they had managed reasonable uptime (which they didn't because they totally underestimated the technical cost of giving people the freedom they did), it still would have been a doomed market, because VPSes got so cheap anyway. We can complain about what a bad deal we got, but it doesn't change the fact that the last salary we paid was in 2005, and the hosting world has moved on. They did what they had to.
- grandalf 16y agoI've had a very bad experience with Joyent. Abysmal performance, tech support in denial in spite of obvious tests. Just leave, your life will be better.
- visakhcr 16y agoDigging Techcrunch got me this story: Twitter and Joyent splits (Circa Jan 31, 2008) http://techcrunch.com/2008/01/31/twitter-and-joyent-split-amidst-downtime-travails/ http://techcrunch.com/2008/01/31/twitter-and-joyent-split-am... Twitter moved to Verio and here's Biz Stone's reply to Techcrunch: We’re still very much engaged in our efforts to bring solid reliability to Twitter. Achieving our goals is a sustained effort, not an overnight fix. Performance is our most important measure of success and we appreciate both the patience and frustration folks are sharing with us. With regard to discussing technical specifics about last night’s efforts, we’ll be more keen to do that once we have a chance to come up for air and cover it with some perspective.
- snapplez 16y agoYou must be on HOWE too. I am in the same boat - unhappy. send a tweet to CEO David Young @davidpaulyoung. (from @snapplez)
- dp111 16y agojoyent is enterprise grade and doesnt have this sort of downtime. its probably a break in your app.
- prodigal_erik 16y agohttp://help.joyent.com/index.php?pg=forums.posts&id=949&pc=6 http://help.joyent.com/index.php?pg=forums.posts&id=949&... acknowledges it's a real outage and talks of rebuilding the server after a significant hardware failure from two days ago, which implies they didn't have a disaster recovery plan.
- patrickgzill 16y agoGood catch; I just wonder how it can take more than 2 days, when you already have a bunch of employees as well as equipment and presumably spares, to build a replacement FreeBSD server and transfer data?
- payingcust 16y agoI will be moving from joyent. I am paying 15usd per month and have not been told to migrate. now I read the founder saying it's my fault that I am suffering so much downtime because I should have searched the forums and migrated years ago. Well I am going to migrate now.