8 ms·
OVH servers exploited through shellshock
- jgreen10 12y agoThis seems like the tip of the iceberg. Shellshock didn't generate nearly as much attention as heartbleed, but is far more devastating and easier to exploit. How many home routers have been taken over by now?
- vertex-four 12y agoHome routers run busybox, not bash.
- conradk 12y agoThat's pretty crazy, though with the hack publicly disclosed, it was bound to be exploited. What I don't understand is why this report incident comes so late. The hack has been public for a large amount of time. Could anyone explain?
- custardcream 12y agoProbably because they are a virtual and colo hoster so aren't generally responsible for the security of the machines they host. Also, a big chunk of people who buy these machines are less responsible than they should be. At best this is monitoring and playing whack-a-mole on their part which they've done a pretty good job of by the looks.
- seszett 12y agoThe involved machines are dedicated servers, according to the posts. Also, although I have many things to blame OVH for, they at least take a somewhat proactive stance regarding security, even for the very cheap dedicated servers. They usually contact you when they notice a traffic spike on unusual ports for example. But yeah... judging by the people I know in real life who own cheap OVH dedicated servers, I'm not surprised at all by these news.
- nraynaud 12y agothat's why I find crazy all those people wanting to have a dedicated server, I don't want to spend my life on call 24/7. I'm pretty sure 90% of the customers are not professionally trained in server maintenance. I am, and I don't want to do it anyways unless I have a good reason to do it.
- atmosx 12y agoI use a VPS for email (configured opensmtpd + IMAP over SSL), VPN server, HTTP Proxy and host 3-4 personal applications (nginx + mysql + ruby apps). The hard part was to set-up all these services, keeping them up-to-date, since no other party depends on them it is considerably easy and almost risk-free. Other than that is not time-consuming at all. Only 2 times in 3 years I had to re-configure a service. Most of the times I just run an upgrade-script on a tmux session and everything runs smoothly (I rarely use pre-compiled packages). Now, if I had a production-level web application, I would probably set-up as carefully as I could a new VPS and roll my application there, up to the point where I needed to scale. If it would be a one-man-show I'd probably go with Heroku (since I write sinatra/ruby/rails applications) or similar (possible cheaper) service to avoid spending time on sys-admin. But these service IMHO are still, expensive for projects with no income while a VPS can do all that and mode at once at a considerably lower price. ps. As a side note. Before I join HN, I though that all programmers were capable of sys-admin (UNIX servers). Then I realized that tasks that seem trivial to me, are considered somewhat difficult for others and vice-versa of course.
- custardcream 12y agoAgree entirely. That's why I go half-way and use Windows Azure's PaaS platform. Much less painful. The company I currently work for have to full racks in two data centres each, most of which is near idle and it's a full time job for 2 people. Could do away with all the machines and all the associated administrative staff, just use a devops team and shave 40% of capex and opex as well by moving to something else. But then again, they might not be getting nice lunches with HP and the DC vendor.
- devonkim 12y ago
- deleted 12y ago[deleted]
- atmosx 12y agoI manage 2 VPSs. I'm responsible for security patches and everything. If shit hits the fan, it's because of me not because of my VPS provider :-) That's what VPS is all about.
- deleted 12y ago[deleted]
- pavs 12y agoI have some vps with digitalocean, I haven't logged in to them in a long time, even after shellshock surfaced publicly. about aftter a month later I checked to see if I am vulnerable through shellshocker.net test and it was negative. I am guessing my provider did something to fix it.
- viraptor 12y ago> ... vps with digitalocean ... I am guessing my provider did something to fix it. I don't think DO offers any managed service. They wouldn't do anything to fix your servers. It's entirely on you to keep them updated. Edit: https://www.digitalocean.com/help/policy/ https://www.digitalocean.com/help/policy/ <- they really don't offer it
- pavs 12y agoI know they dont, I didn't say they do. But my non up-to-date Ubuntu server wasn't vulnerable when I checked about a month after shellshock vulnerable went public.
- tokenizerrr 12y agoHave you considered you may be using a software stack that is not (obviously) vulnerable? Not every software is vulnerable by default on all URLs, just the ones that wind up calling out to bash in a specific way.
- 12y ago
- Wilya 12y agoThe linked issue isn't "we need to patch Shellshock". It's "we have 800 hacked machines on the network, they are DDoSing like crazy, and it's saturating the internal network". Their problem isn't that servers have been hacked. It's that it's overloading the network. OVH isn't responsible for what people do with the servers. They provide the initial installation, networking, hardware monitoring and some management tools, but that's it. If they detect that a server is sending too much traffic, they can guess that it has been hacked, so what they usually do is disable it and notify the owner to fix it. But they don't do more, that's not their business. If you want a host that does more for you, it's managed hosting you want, not a dedicated server provider.
- aroch 12y agoThey actually tout their anti-hacking and anti-DDOS services when you sign up...
- funkyy 12y agoIts on network/hardware level. The issues with internal DDOS is that they are software related. OVH cannot access your server (or it shouldn't) so the only choice they have is to shut the server and wait till you will fix it. But if you receive DDOS attack to your server from outside, they can defend you using network resources.
- Wilya 12y agoTheir anti-DDOS system is mostly designed to protect against external attacks. It works at the network level, probably at the connection between their network and the outside world. Because that's the most efficient way: detect them and block them where you have the most bandwidth available. This is an internal attack, which requires different mitigation measures, and is seen less often in the wild (compromising 500 servers from a specific provider is more difficult than 500 random servers on the internet, and you're pretty much guaranteed that the provider will deactivate most of them after the first attack), so I guess their protection systems aren't as developped against it.
- dwild 12y ago
- ck2 12y agoTheir network was so saturated, we were only getting 1KB/sec inbound yesterday. Fun times. Their VAC protects external inbound but does nothing in their intranet which runs at full tilt 1gbps for most dedicated servers.
- SG- 12y agoThey've been having capacity issues for almost 2 weeks now at peak, you can see it at http://weathermap.ovh.net/usa http://weathermap.ovh.net/usa usually. Capacity inside their network between Montreal and Newark was at 100% until an upgrade was done last week (edit: the upgrade isn't finished yet, but so far it's helped): http://status.ovh.com/?do=details&id=8038 http://status.ovh.com/?do=details&id=8038 Tata transit in Montreal has been saturating at peak too. I'm hoping all this was just internal botnet/DDoS that was ramping up and the issues will go away as they clean things up.
- phreeza 12y agoCompletely forgot I had a server running there, I think it is actually a dedicated server, not VPS. It was vulnerable but seems to be uncompromised. Fixed it now.
- djm_ 12y agoI'm sure you're well aware but that server should not be trusted now for anything sensitive (or anything at all really) until it's been re-imaged.
- werid 12y agoJust having a vulnerable bash isn't enough to break in. You also need something that uses it. A CGI script available via webserver, or some other service (I hear OpenVPN had issues...)
- segmondy 12y agoVPS providers should take a little responsibility to protect their customers. I'm not talking about full on sys admin or managed services, but even a scan and report or taking off badly infected hosts offline till the owner upgrades it. This will save them much time in the future, save their own some time, and save other sites (that will be attached through the compromised hosts some time). Being a good citizen of the Internet boils down to taking some level of responsibility. I wont put grandma on the internet without malware bytes or avg, etc on her computer. So why should I give her a VPS and tell her she's on her own?
- M41K0 12y agoOVH is not a VPS Providers. OVH is a global service provider. This hack is on Dedicated Server. OVH do not have the right and the access to the system of thoses servers. By the way, all the clients have been informed Two weeks ago by a global mailling because of Shellshock. The responsability is all on the customer.
- spacefight 12y ago"The responsability is all on the customer." That changes quickly if links get saturated suddenly and other customers are impacted.
- mahouse 12y agoThen cut off the network connectivity of the compromised hosts because they have breached the contract.
- BuildTheRobots 12y ago> OVH do not have the right and the access to the system of thoses servers. You should check your TOS. They reserve the right to access your server and if you use any of the default install images it comes pre-configured with their SSH key.
- stevekemp 12y agoOne of the most command ways a virtual machine is compromised is by brute-forcing a root SSH password. Do you suggest that every hosting company run non-stop dictionary attacks against their clients? Or even nmapping open ports, non-stop, looking for backdoor services? 95% of the time virtual machines are running unmanaged, and the hosting company has no insight into what is running inside the machines, nor how up to date they are.
- concern2 12y agoDoes anybody know if any of the hubic.com services have been affected by this?
- sspiff 12y agoI run a server on OVH, but I figured as I don't run any dynamic web app on it, I was unaffected by shellshock - the only way in is through the web server (which basically hosts a bunch of static files), or SSH. From what I understood, shellshock was caused by poor sanitation of bash variable extension, and so only posed a threat to people who used bash from their public-facing interfaces, hosted bash CGI scripts, ... Was I wrong to assume that I was safe?
- SCHiM 12y agoYes. I've seen attempts to exploit bash via http headers (the headers them selves). Like so: GET / HTTP/1.1\r\n Host: () { :;}; echo vulnerable\r\n User-Agent: () () { :;}; echo vulnerable\r\n\r\n So even static servers could have been vulnerable. I'm not sure if your server was vulnerable or not, but I think you were definitely too soon with assuming you were safe.
- tokenizerrr 12y agoThis exploit only works if bash is called at all during the handling of the request. This is typically only the case with CGI scripts, or when the requested application does a system() call or directly calls out to bash
- SCHiM 12y agoThough it should certainly not happen, it's not uncommon for various applications to 'shell-out' certain tasks. The point I was trying to make is that you're not necessarily secure just because you don't use CGI or only host static content, and that exploitation via the web server could certainly have been possible.
- TazeTSchnitzel 12y agoAny time any user data is stored in an environment variable and bash is called somewhere down the line, you've got a problem. This means any CGI script is affected if bash is the default shell, for example.
- 12y ago
- Tepix 12y agoI have had a bad experience with the OVH abuse department. I sent them email twice (once about spam, once about network attacks) and never received anything, not even an automated email delivery confirmation.
- balladeer 12y agoOVH/Kimsufi has non-existent support. They claim to offer support (yes, every kind - from listing your passport verification request to tax exemption request) via a publicly accessible forum. Even that support is non existent. All you get there is few useless answers from few uselessly enthusiastic (sometimes just smartass) fellow customers.
- kristopherwong 12y agoI think the thread here is that we should look at having an option to do automated patching. The major CVE (heartbleed, SHellshock) are really detrimental to the community as a whole and think that the VPS provide could build some "Goodwill" if they offered and automated way to mitigate these Vulnerabilities. Worst case, is that they send you an email notifying you. Best case they offer you a 1 click (1 cmd line) to resolve the major issues.
- napsterbr 12y agoI feel it's my clue to ask a question I've been wondering lately: how good is OVH's dedicate server service? (Regarding network, uptime, system outages etc). I used one of their dedicate servers for six months and never had any problem with it, however I plan to buy (rent) dozens of big-data servers for my next project, and while my experience so far have been good, I'd like to hear from someone else. Despite of that, I had several downtime issues with VPSs hosted at OVH. I guess the difference is you can't oversell dedicated servers.
- Bedon292 12y agoI have been using OVH for a few years now, and had no issues to report. This is only a few low-mid level dedicated boxes and a half dozen VPS's though, so others may have differing results.
- thaumaturgy 12y agoOVH has been a frequent flyer in my abuse and spam logs, and I blackhole them regularly, fwiw.
- yc1010 12y agoI had a few expensive top end storage servers (36 drives), they worked fine for years, only 1-2 drive failures, no power failures or major network outages I can remember. Last year i switched to collocation instead, but still have 1 server with them, things have improved alot lately, now the servers come with ipmi and new ovh control panel is nice. Problem with OVH are noobs buying their servers trying to save every buck only to realise that the support will not fix anything but hardware issues (loose cable, broken drive etc). Obviously alot of people do not understand that unmanaged means unmanaged
- api 12y agoThe title is misleading -- it was customer servers on OVH that were exploited en masse and then proceeded to flood the network with DDOS and bot traffic, not OVH itself (apparently). This has likely happened to loads of VPS and dedicated server providers. One consequence of hosting such a service is that you typically end up with a lot of poorly administered crack houses on your network.
- teepo 12y agoThat's a pretty serious limitation of their automation if they are unable to just whack the ACLs on the private network to the affected customer nodes.
- Lanzaa 12y agoIt sucks that so many servers are still vulnerable to the shellshock issue. I am impressed with OVH's openness in regards to reporting this issue. I am amused and impressed by their final comment/conclusion; "Obviously it is time to go to 2x100G on the private network between Europe and Canada."
- cpsaltis 12y agoIt is sad that so many servers are still vulnerable on an issue that has been reported through all major mainstream news networks. We've witnessed several attacks many days after Shellshock was fixed, and chasing down the botnet scripts we saw hundreds of servers compromised. The story and scripts were published on a blog post in case anyone wants to check out a standard botnet attack: http://blog.mist.io/post/100582053116/anatomy-of-a-shellshock-botnet http://blog.mist.io/post/100582053116/anatomy-of-a-shellshoc...