6 ms·

OVH acknowledges back office security issue

58 points by kerneis 13y ago

aw3c2 13y ago
A pastebin link is not a source worth reading. Here is what I found as more official, the oles@ovh.net account is often used for official announcements: http://forums.ovh.net/showthread.php?p=548183 http://forums.ovh.net/showthread.php?p=548183 tl;dr: Bad random string generator in password change URLs, 3 customers affected, all Bitcoin related. edit: OVH, if you are reading this. The 4 character string for receipt protection is creeping me out ever since I saw it. Why not simply use a longer string?
- kerneis 13y ago
  Thanks for the link. I've been looking for the list archives on http://ml.ovh.net http://ml.ovh.net (as advertised in the email itself) but the domain doesn't seem to exist anymore; hence my (unreliable) copy on pastebin.
DoubleMalt 13y ago
Can someone proficient in French provide a tl;dr? I have been thinking about renting a server from then so this interests me greatly.
- jkat 13y ago
  21 character reset was generated via 3 separate functions each generating 7 characters each. 2 of these were old and weak, resulting in a URL that could be brute forced. Once discovered, they analyzed 3 years worth of logs to see who was exploited (not clear how they matched it..maybe scanning for brute forces?) and found 3 sites all related to bitcoin. The weakness in the algo has been resolved by using a 2 real random functions to generate 64 random characters
doe88 13y ago
From the post on the french forum: they were generating a 21 characters string from 3 different sources of random (each source contributing 7 characters to the final string) and it looks like that among these 3 sources, 2 of them were flawed and were not really random. They say they fixed this flaw by now generating 64 characters random strings from 2 sources of random which they think secure.
- MichaelGG 13y ago
  How hard is /dev/urandom? Or without that, read /dev/random every so often and mix it with SHA1?
adorable 13y ago
TL;DR: Here's what happened. When a password reset is submitted, the user receives a random URL by email containing a random 21 letter string. The algorithm used was bad, so the random string could be brut-forced. The OVH team researched 3 years of password changelogs and found 3 customers who had been brute-forced that way, all in the "bitcoin community"
coutud 13y ago
Translation from french : Hi, To change your password, you have to go on ovh website and ask for a change with a given id. An email is sent to the email address associated with the id which includes a unique url. This url is randomly generated with 21 characters, generated from 3 different algorithms for randomess generating each 7 characters. The client who receives the email can then click on the link and get a new password for his id. A confirmation email is sent announcing the change of password. In all emails sent, ovh includes the IP of the person who did the request. This is a procedure that was set up 7 years ago and hasn't changed since. On april 26th, we have discovered internely an issue with the function that generates the 21 characters. Two of the three random functions were generating a not-so-random string. It was hence possible to ask for a password change and then to brute-force the password reset url. This problem was found by one of our developpers on apr 26th at 11:03:14 and was fixed at 12:54:13. The origin of this problem was linked to the rand() function used in this part of the code that hasn't been patched to the same level as the rest of the code when we activated the script execution cache (?). We have replaced the old function returning 21 characters from 3 "random" functions with a new function using two really random functions each providing 64 characters. We have then started searching in our databases to check if the vulnerability had been exploited, and when it could have been. To do that, we have looked at 3 years of history of password resets. We are authorized by the CNIL (commission nationale informatique et liberté, french agency regulating databases containing personnal data) to archive and use all logs for the last 10 years, specifically for this kind of issue. We have found 3 active ids which had a password reset done by brute-force. These three occurences were targeted attacks against a part of the "bitcoin" community that uses ovh. The hacker seems to have found the flaw on apr 23rd at 22:00, and did a lot of testing for one hour to make his method work. At 23:00, his method was working and he has hacked the first id, then the two others ids the following day, still for bitcoin related sites. We have been in contact with these clients but the quality of our exchanges did not allow us to get enough information that would help reveal an vulnerability on our side. Our developpers have in a completely independant way found the issue mentionned above, and only then have we realised that there was a link between our flaw and these 3 customers. We certainly have something to learn here about the best way to ensure dialog with clients about this kind of issue. It took us some time to communicate on the subject, because we have quickly realised that the impact was very limited (3 customers) and we wanted to take enough time to verify everything in depth and make sure that this had not impacted any other customers. Having finished our search today going back three years, we can now assert that no other customers have seen this issue. We are going back 10 years now, but the probability that this happened ealier is null. I think that despite the small impact for our customers, it was our duty to inform everybody of this security incident that we had to deal with last week. We have added a code-review process on some very old code that hasn't been rewritten for years, to check that there are no other similar issues. We are also trying to see how we can improve communication with clients for this kind of incident, knowing that 2 of the 3 customers are actually customers of our affiliates/branches. TL;DR : Yes, we had a security vulnerability allowing a password change using some complex brute-force procedure. We advise all clients running critical services to limit access to the manager from some IP only. Yes, 3 clients of the bitcoin community have been impacted by this security flaw. It is very important to read the emails that ovh sends automatically, especially the notification of password change emails, when they aren't initiated by you. If you see a request for a password change that you haven't initiated, you should call our incident support 24/24 which will block your account until everything is clear. No, our clients database hasn't been compromised. No, there was no impact on other clients than these 3. We are deeply sorry for the three customers that have been impacted, and invite them to contact our commercial team (in french). Regards Octave
mikegogulski 13y ago
Vision of the future: Hosting companies which do not provide insurance against such losses, subject to intensive security auditing, justifiably lose business.
- rubinelli 13y ago
  If you decide to stash your life savings into your local swimming pool's locker, don't be surprised when they are taken away. For $40~$100/month, OVH offers adequate securit for your regular daily deals site or micro-SaaS. If you expect the level of security that comes with PCI/HIPAA compliance, expect to pay two orders of magnitude more. At some point, someone in the Bittcoin community will bite the bullet and pay the price, and that will be a huge competitive advantage.
  - gtaylor 13y ago
    This. OVH is a budget provider. While this doesn't excuse them from fault here, you definitely get what you pay for in this case.
laurent123456 13y ago
"oles" is Octave Klaba, the CEO of OVH. I'm quite impressed by how transparent they are in dealing with this issue, and the degree of technical details they are willing to provide.
mrb 13y ago
There we go. I suspected that Oles was misinforming/lying to the community when he initially denied the existence of a security vulnerability, and I was right: https://news.ycombinator.com/item?id=5627487 https://news.ycombinator.com/item?id=5627487 When you think about it, this whole situation with the hacks of Linode, OVH, etc, is completely insane. Bitcoin has become so successful and valuable, that bitcoins thieves are compromising core Internet infrastructure that so many companies rely on. People say "use more serious/professional hosters", but OVH is the world's largest server host and is very much a "professional" service. I bet that within 2-3 years we will see more and more stunning hacks of other hosters like Amazon EC2, etc.
- DigitalJack 13y ago
  The flip-side is now OVH is more secure than they have been in the past. Nothing like a pot of gold to motivate people to try to break your security. This is probably more effective at exposing weakness than paid pen-testing. Not that you ever want customers to lose money/data/whatever in the process, but it's not all down-side.
- ig1 13y ago
  It's not however a high-security host. If you need security you should at least be using a provider that meets the security provisions of PCI, HIPAA, FISMA, etc. If banks and government agencies aren't using it, then it probably doesn't have the security level needed for high-value bitcoin wallets. There's a reason you have specialist data centres that focus on sectors like finance.
sylvinus 13y ago
Not sure how relevant this is but there is a huge amount of spelling errors in this email, which makes it look all the more authentic but very unprofessional. Also surprising that this email (if the pastebin is authentic) appears to have been written by a mail client from 2006 suffering from at least one public GnuPG-related vulnerability.
- gizmogwai 13y ago
  Octave Kabla, aka Oles is native from Poland, his french has always been quite approximative (like my english, I guess).
thomc 13y ago
I'm 6 for 5 at the moment - of the 6 companies I've hosted pet projects with over the years, 5 have been compromised at the control panel / infrastructure level while I was a customer: * Layered Technologies, dedicated OpenBSD (Sep 17 2007) [0] * Hetzner, dedicated Linux (Oct 21 2010) [1] * Dreamhost, shared Linux (Jan 19 2012) [2] * Linode, VPS Linux (Apr 15 2013) [3] * OVH, VPS Linux via reseller (Apr 23 2013) [4] [0] http://what.entwinedstudios.com/?p=21 [1] http://mybroadband.co.za/news/internet/16021-Hackers-hit-Hetzner.html [2] http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/ [3] http://blog.linode.com/2013/04/16/security-incident-update/ [4] https://news.ycombinator.com/item?id=5632479 Food for thought.
luisivan 13y ago
Oles, what a CEO. I have had direct contact with him, as OVH is a partner of one of my projects, and it's not that easy to find someone so passionate, humble and responsive as he is.

58 points by kerneis 13y ago

aw3c2 13y ago

A pastebin link is not a source worth reading. Here is what I found as more official, the oles@ovh.net account is often used for official announcements: http://forums.ovh.net/showthread.php?p=548183 http://forums.ovh.net/showthread.php?p=548183 tl;dr: Bad random string generator in password change URLs, 3 customers affected, all Bitcoin related. edit: OVH, if you are reading this. The 4 character string for receipt protection is creeping me out ever since I saw it. Why not simply use a longer string?

kerneis 13y ago
Thanks for the link. I've been looking for the list archives on http://ml.ovh.net http://ml.ovh.net (as advertised in the email itself) but the domain doesn't seem to exist anymore; hence my (unreliable) copy on pastebin.

DoubleMalt 13y ago

Can someone proficient in French provide a tl;dr? I have been thinking about renting a server from then so this interests me greatly.

jkat 13y ago
21 character reset was generated via 3 separate functions each generating 7 characters each. 2 of these were old and weak, resulting in a URL that could be brute forced. Once discovered, they analyzed 3 years worth of logs to see who was exploited (not clear how they matched it..maybe scanning for brute forces?) and found 3 sites all related to bitcoin. The weakness in the algo has been resolved by using a 2 real random functions to generate 64 random characters

doe88 13y ago

From the post on the french forum: they were generating a 21 characters string from 3 different sources of random (each source contributing 7 characters to the final string) and it looks like that among these 3 sources, 2 of them were flawed and were not really random. They say they fixed this flaw by now generating 64 characters random strings from 2 sources of random which they think secure.

MichaelGG 13y ago
How hard is /dev/urandom? Or without that, read /dev/random every so often and mix it with SHA1?

adorable 13y ago

TL;DR: Here's what happened. When a password reset is submitted, the user receives a random URL by email containing a random 21 letter string. The algorithm used was bad, so the random string could be brut-forced. The OVH team researched 3 years of password changelogs and found 3 customers who had been brute-forced that way, all in the "bitcoin community"

coutud 13y ago

Translation from french : Hi, To change your password, you have to go on ovh website and ask for a change with a given id. An email is sent to the email address associated with the id which includes a unique url. This url is randomly generated with 21 characters, generated from 3 different algorithms for randomess generating each 7 characters. The client who receives the email can then click on the link and get a new password for his id. A confirmation email is sent announcing the change of password. In all emails sent, ovh includes the IP of the person who did the request. This is a procedure that was set up 7 years ago and hasn't changed since. On april 26th, we have discovered internely an issue with the function that generates the 21 characters. Two of the three random functions were generating a not-so-random string. It was hence possible to ask for a password change and then to brute-force the password reset url. This problem was found by one of our developpers on apr 26th at 11:03:14 and was fixed at 12:54:13. The origin of this problem was linked to the rand() function used in this part of the code that hasn't been patched to the same level as the rest of the code when we activated the script execution cache (?). We have replaced the old function returning 21 characters from 3 "random" functions with a new function using two really random functions each providing 64 characters. We have then started searching in our databases to check if the vulnerability had been exploited, and when it could have been. To do that, we have looked at 3 years of history of password resets. We are authorized by the CNIL (commission nationale informatique et liberté, french agency regulating databases containing personnal data) to archive and use all logs for the last 10 years, specifically for this kind of issue. We have found 3 active ids which had a password reset done by brute-force. These three occurences were targeted attacks against a part of the "bitcoin" community that uses ovh. The hacker seems to have found the flaw on apr 23rd at 22:00, and did a lot of testing for one hour to make his method work. At 23:00, his method was working and he has hacked the first id, then the two others ids the following day, still for bitcoin related sites. We have been in contact with these clients but the quality of our exchanges did not allow us to get enough information that would help reveal an vulnerability on our side. Our developpers have in a completely independant way found the issue mentionned above, and only then have we realised that there was a link between our flaw and these 3 customers. We certainly have something to learn here about the best way to ensure dialog with clients about this kind of issue. It took us some time to communicate on the subject, because we have quickly realised that the impact was very limited (3 customers) and we wanted to take enough time to verify everything in depth and make sure that this had not impacted any other customers. Having finished our search today going back three years, we can now assert that no other customers have seen this issue. We are going back 10 years now, but the probability that this happened ealier is null. I think that despite the small impact for our customers, it was our duty to inform everybody of this security incident that we had to deal with last week. We have added a code-review process on some very old code that hasn't been rewritten for years, to check that there are no other similar issues. We are also trying to see how we can improve communication with clients for this kind of incident, knowing that 2 of the 3 customers are actually customers of our affiliates/branches. TL;DR : Yes, we had a security vulnerability allowing a password change using some complex brute-force procedure. We advise all clients running critical services to limit access to the manager from some IP only. Yes, 3 clients of the bitcoin community have been impacted by this security flaw. It is very important to read the emails that ovh sends automatically, especially the notification of password change emails, when they aren't initiated by you. If you see a request for a password change that you haven't initiated, you should call our incident support 24/24 which will block your account until everything is clear. No, our clients database hasn't been compromised. No, there was no impact on other clients than these 3. We are deeply sorry for the three customers that have been impacted, and invite them to contact our commercial team (in french). Regards Octave

mikegogulski 13y ago

Vision of the future: Hosting companies which do not provide insurance against such losses, subject to intensive security auditing, justifiably lose business.

rubinelli 13y ago
If you decide to stash your life savings into your local swimming pool's locker, don't be surprised when they are taken away. For $40~$100/month, OVH offers adequate securit for your regular daily deals site or micro-SaaS. If you expect the level of security that comes with PCI/HIPAA compliance, expect to pay two orders of magnitude more. At some point, someone in the Bittcoin community will bite the bullet and pay the price, and that will be a huge competitive advantage.
- gtaylor 13y ago
  This. OVH is a budget provider. While this doesn't excuse them from fault here, you definitely get what you pay for in this case.

laurent123456 13y ago

"oles" is Octave Klaba, the CEO of OVH. I'm quite impressed by how transparent they are in dealing with this issue, and the degree of technical details they are willing to provide.

mrb 13y ago

There we go. I suspected that Oles was misinforming/lying to the community when he initially denied the existence of a security vulnerability, and I was right: https://news.ycombinator.com/item?id=5627487 https://news.ycombinator.com/item?id=5627487 When you think about it, this whole situation with the hacks of Linode, OVH, etc, is completely insane. Bitcoin has become so successful and valuable, that bitcoins thieves are compromising core Internet infrastructure that so many companies rely on. People say "use more serious/professional hosters", but OVH is the world's largest server host and is very much a "professional" service. I bet that within 2-3 years we will see more and more stunning hacks of other hosters like Amazon EC2, etc.

DigitalJack 13y ago
The flip-side is now OVH is more secure than they have been in the past. Nothing like a pot of gold to motivate people to try to break your security. This is probably more effective at exposing weakness than paid pen-testing. Not that you ever want customers to lose money/data/whatever in the process, but it's not all down-side.
ig1 13y ago
It's not however a high-security host. If you need security you should at least be using a provider that meets the security provisions of PCI, HIPAA, FISMA, etc. If banks and government agencies aren't using it, then it probably doesn't have the security level needed for high-value bitcoin wallets. There's a reason you have specialist data centres that focus on sectors like finance.

sylvinus 13y ago

Not sure how relevant this is but there is a huge amount of spelling errors in this email, which makes it look all the more authentic but very unprofessional. Also surprising that this email (if the pastebin is authentic) appears to have been written by a mail client from 2006 suffering from at least one public GnuPG-related vulnerability.

gizmogwai 13y ago
Octave Kabla, aka Oles is native from Poland, his french has always been quite approximative (like my english, I guess).

thomc 13y ago

I'm 6 for 5 at the moment - of the 6 companies I've hosted pet projects with over the years, 5 have been compromised at the control panel / infrastructure level while I was a customer: * Layered Technologies, dedicated OpenBSD (Sep 17 2007) [0] * Hetzner, dedicated Linux (Oct 21 2010) [1] * Dreamhost, shared Linux (Jan 19 2012) [2] * Linode, VPS Linux (Apr 15 2013) [3] * OVH, VPS Linux via reseller (Apr 23 2013) [4] [0] http://what.entwinedstudios.com/?p=21 [1] http://mybroadband.co.za/news/internet/16021-Hackers-hit-Hetzner.html [2] http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/ [3] http://blog.linode.com/2013/04/16/security-incident-update/ [4] https://news.ycombinator.com/item?id=5632479 Food for thought.

luisivan 13y ago

Oles, what a CEO. I have had direct contact with him, as OVH is a partner of one of my projects, and it's not that easy to find someone so passionate, humble and responsive as he is.