10 ms·
HipChat security notice
- ndrake 10y agoI wonder which "popular third-party library" caused the problem
- pavel_lishin 10y agoI assume they'll reveal that information once the library fixes the issue.
- SteveNuts 10y agoProbably left-pad
- reverted 10y agoMore importantly, I wonder how much they were paying for this library, or to what extent they were supporting it internally. Because if the answer is zero and they weren't, I would put a lot of the blame on HipChat engineering.
- lolsal 10y agoI'm not sure I understand you - You would blame the users of a third-party library if the library was found to have a vulnerability and it was exploited against the people using the library?
- eropple 10y agoI read it as "if it's open-source, a company of Atlassian's size should be being good stewards and taking care of things that are helping them make money."
- lolsal 10y agoOpen source code now carries a moral maintenance obligation? Do we say the same thing about any large company that uses openssl or any other open source libs that people use or depend on? That doesn't seem fair or reasonable.
- plange 10y agoNot only moral, but mostly legal. Warranties are not included. So it's a bit lame to blame "a popular third party library". The OP was trying to say a company of Atlassians size should dedicate the resources to vet (and fix) those libraries if they use them for these purposes.
- lolsal 10y agoI don't think anyone is trying to shift blame, just to explain what happened. I am not affiliated with Atlassian so I'm only guessing.
- plange 10y agoSorry, i wasn't trying to imply they shifted blame. But atlassian does bare the legal&moral burden of securing their product here.
- lolsal 10y agoI agree with you. I don't agree that they have a moral obligation to make pull requests to the open source library that had the issue. Hopefully they will, but there is no obligation there in my mind.
- sithadmin 10y ago>Open source code now carries a moral maintenance obligation? Many have always argued that it has. >Do we say the same thing about any large company that uses openssl or any other open source libs that people use or depend on? Many do. >That doesn't seem fair or reasonable. Many argue that any company that failing to contribute to the OSS projects they depend upon isn't fair or reasonable.
- cbhl 10y agoI think this is a part of the general meme that big companies are making money by taking open source without giving back, and that big for-profit companies could do more for the open-source community.
- falcolas 10y agoIMO, frankly, yes. If you use someone else's code, especially if you're not paying anything for it, you get what you put into it: nothing. The liability for this breach is ultimately owned by Atlassian, not the third party library writer. To quote the most permissive license out there: "THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED."
- lolsal 10y agoTo use an analogy, do you blame everyone that has ever used the linux kernel whenever bugs/vulnerabilities are discovered in the kernel?
- falcolas 10y agoI would certainly blame Google if their Android phones were backdoored, especially if they tried to foist the blame off on the Linux kernel developers - a much more apt analogy since they sell Android phones.
- algesten 10y agoI would be very surprised if something as complicated as Android phones didn't contain anything that can be back doored. Obviously that is Google's problem, but I haven't seen Google (nor Atlassian in this case) claim anyone else is to blame.
- geofft 10y agoIf they use the upstream kernel, yes. Linus Torvalds has been very clear that security is not a priority. "We have one rule in the kernel: don't break userspace. Everything else is kind of a guideline. The whole security thing? It's a guideline that we shouldn't do stupid shit. But people do stupid shit all the time and I don't get that upset." https://www.youtube.com/watch?v=1Mg5_gxNXTo#t=8m28 https://www.youtube.com/watch?v=1Mg5_gxNXTo#t=8m28 Imagine, Torvalds said, that terrorists exploited a flaw in the Linux kernel to cause a meltdown at a nuclear power plant, killing millions of people. “There is no way in hell the problem there is the kernel,” Torvalds said. “If you run a nuclear power plant that can kill millions of people, you don’t connect it to the Internet.” Or if you do, he continued, you build robust defenses such as firewalls and other protections beyond the operating system so that a bug in the Linux kernel is not enough to create a catastrophe. http://www.washingtonpost.com/sf/business/2015/11/05/net-of-insecurity-the-kernel-of-the-argument/ http://www.washingtonpost.com/sf/business/2015/11/05/net-of-... And the Linux kernel has a track record for not being the world's most secure piece of software. Which is fine, it's a project he started for fun. It's certainly possible to pay people for a kernel that they'll stand behind commercially. If you're using, say, a RHEL kernel and a bug or vulnerability is discovered that impacts you, by all means get upset at Red Hat. If you're using a Fedora kernel, though, you made the choice to use it. It's completely unfair for you to get the benefits of running a kernel you put neither time nor money into, and not also the risk of running that kernel.
- GrinningFool 10y agoWhen you use any third party library, you're responsible for its behaviors (or misbehaviors) on your customer's machine. How is it possible to view this in any other way?
- lolsal 10y agoI 100% agree with you. Atlassian is 100% responsible. I'm not sure I would not say they are at 'fault' though. Maybe I'm just quibbling over semantics. I don't know the details of the vulnerability - I would say they were at fault if they did not update/patch a fixed vulnerability.
- throwaway91111 10y agoThis really depends on the use case of the library. It's entirely possible to find bugs that are outside your expertise to fix. I can't speak to this case, obviously.
- geofft 10y agoIf it's free software? Yes, absolutely. To do otherwise is a chilling effect against hobbyist free-software authors in favor of large companies that have the ability to take on that liability. I, as an individual, want to be able to write code in my free time, put it on GitHub, and let it get popular without worrying that maybe it has a bug in it. If people want to start holding me responsible for it, I'm just not going to release it. Maybe you can pay my employer for a commercial license if we think that the cost of auditing it and taking on the liability for bugs is commercially reasonable, but it's also unlikely my employer will be interested.
- lolsal 10y agoI am absolutely not advocating that the library author be held liable.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- ej_campbell 10y agoOne of these: http://sources.hipchat.com/ http://sources.hipchat.com/
- masterleep 10y agoIt might be: https://bugs.ghostscript.com/show_bug.cgi?id=697808 https://bugs.ghostscript.com/show_bug.cgi?id=697808
- notum 10y agoNeedless to say their (login) servers crashed from the pressure of people resetting their credentials. "Hey, you know what might be a good idea? Let's email all of the accounts at the same time using an Appriver blast!" Atlassian. I hate to hate you.
- rubyn00bie 10y agoWhile I can see your point in this case I think it was the appropriate action, their ops team should've just beefed up their resources in conjunction with the email blast. Only emailing a rolling amount of your customers becomes a shit show of support, who do you email first? Who do you email last? How long do you wait between groups? For who is security important, your biggest customers, highest paying, most security conscious? It's a real shit show to know, and one you'd absolutely get wrong, letting everyone know as fast as possible is the only acceptable solution to a security breach.
- notum 10y agoThe servers should have definitely been prepared for the increased load. Perhaps I'm overly optimistically using the plural form in this case. Truth to be told I can only assume this was done in a short burst, given my limited sample of (hopefully ever narrowing) circle of people who use Atlassian products. But would distributing the bulk-mail over an hour (two, three) using a randomized sample of their customer base really made a significant impact to security or their support? I wonder how I'd do it, really, if let's say, beefing up my infrastructure for some reason isn't an option.
- throwaway8473 10y agothey dont have an ops team anymore it was all moved to a terrible new product that they hope will make people love them again source:former employee
- 925dk 10y agoWhy are they force resetting everyone's password if they are bcrypt'ed?
- yarper 10y agothe most popular passwords are love, secret, sex and god
- jacquesm 10y agoDoes HipChat hash client side or server side?
- cypherg 10y agoit's considered best security practice to do so
- 925dk 10y agoBy whom?
- Godel_unicode 10y agoScience. You might find the below numbers interesting. Note that this performance is only one workstation with 8x gtx980. Even the mighty bcrypt (sidebar, look at the sha512 #s) won't save you if your password is bad. Now consider social media mining to enhance the word list. Now consider that (anecdotally) I have never done a hashcat audit and not had to have a conversation with someone about choosing better passwords: Hashtype: bcrypt, Blowfish(OpenBSD) Workload: 32 loops, 2 accel Speed.GPU.#1.: 6398 H/s Speed.GPU.#2.: 6507 H/s Speed.GPU.#3.: 6513 H/s Speed.GPU.#4.: 6643 H/s Speed.GPU.#5.: 6534 H/s Speed.GPU.#6.: 6512 H/s Speed.GPU.#7.: 6689 H/s Speed.GPU.#8.: 6542 H/s Speed.GPU.#*.: 52338 H/s https://gist.github.com/epixoip/c0b92196a33b902ec5f3 https://gist.github.com/epixoip/c0b92196a33b902ec5f3
- Jach 10y agoBcrypt isn't magic. It will help slow down a full crack against everyone when every pw takes some tens of ms to check (though even if they had a hefty work factor of 1 second on a beefy ec2 instance, you can check a million accounts for 'password' in 11.5 days on a single machine, much less when you can spin up many more instances / leverage a botnet of many if less powerful machines). And if you want to target an individual user, you can try a million different PWs on their account over the same period. That's why it's best practice for everyone to rotate, though if your PW is complex and you're not a particularly juicy target you can probably get away with not doing it right now.
- pram 10y agoIs this implying their database was leaked?
- plange 10y agoYes, possibly.
- jordache 10y agoWTF? I got the email from HipChat. It includes this sentence. Without additional non-techy context "HipChat hashes passwords using bcrypt with a random salt." This is a good example of how not to do mass e-mails targeting the general population.
- balls187 10y agoSerious question: Do you need the non-techy explanation?
- jordache 10y agono.. but I could be a non IT user using hipchat. This sentence is likely meaningless to me.
- balls187 10y agoSure. I get your point. Just asking if YOU specifically needed the explanation, which I'm sure a number of folks would be happy to provide.
- plange 10y agoThen you would be fine with the standard boilerplate "your data is secure with us"? I'm happy they say this.. now i'd like some sort of proof :-)
- jordache 10y agoA follow up sentence that re-iterates the same sentiment in non technical term will accommodate none technical recipients. Poor UX is my point of this
- bjoernw 10y agoGoogling bcrypt should do the trick and most people know how to do that. </sarcasm>
- Retr0spectrum 10y agoThe HipChat desktop client had a trivial MITM vulnerability which took them several months to fix after I reported it. They never made any kind of public notice about it, so I'm almost surprised to see them talking about security here.
- j_s 10y agoWhere does that vulnerability report fit in with the Atlassian acquisition? (circa spring 2012)
- Retr0spectrum 10y agoIt was first reported around this time last year, so "after".
- problems 10y agoDoubt this will be a popular view around here, but using a 3rd party service for internal business communications is just a bad idea. I've seen companies posting root passwords, ssh keys, salaries, internal financial details, etc in Slack and HipChat. Just waiting for a disaster to strike, adding value for every additional company to the target. Maybe this breach won't be the last straw, but it's a consistent risk. You can run your own MatterMost or XMPP server quite easily and even lock it down to behind VPN only to minimize security risks almost completely.
- throwaway91111 10y agoIn my experience, using irc or xmpp mostly results in people not using it unless a) the team is largely technical or b) there's a common, easy interface like gchat used to be.
- deaddodo 10y agoWhy are you giving your employees a choice in the matter of something so important? Set up an XMPP server, tell them that's what is used for internal communication. Period. And if they're too lazy/dumb/entitled to download Adium/Pidgin and enter their email address+password; well, you should probably find better employees.
- wilg 10y agoOh for God's sake.
- detaro 10y agoYou probably should not let all your (non-technical) employees install random software they download.
- Xylakant 10y agoFriends don't let friends use libpurple based messengers. Sadly, Adium development is pretty stale and unresponsive to even major security issues such as https://threatpost.com/code-execution-vulnerability-found-in-libpurple-im-library/124448/ https://threatpost.com/code-execution-vulnerability-found-in...
- chime 10y agoMy recent password update policy: Wait for a service to be hacked and then change the password to a long hash generated by Keepass. I was thinking of spending a whole day updating all the passwords for all the services I have accounts on but at the rate sites are getting hacked, it won't be long before I have created unique hash passwords for all the sites.
- yarper 10y agoI miss IRC. All it needs is a few tweaks to bring it to 2017..
- vurpo 10y agoThen you might like Matrix. From the IRC point of view, it's basically IRC, but updated to become a 2017 protocol.
- tbomb 10y agoDo they really need a captcha on BOTH username AND password input? I get that they are different submit pages, so they are likely querying the database on each page, but is that really necessary? I don't see any benefit from it, as a user, while trying to log in to my account.
- 925dk 10y ago"This weekend our Security Intelligence Team detected" ... "Security Intelligence Team" ... yeah, because that team actually exists.
- 925dk 10y agoIt's sitting right next to the emoji team.
- cypherg 10y agowell written blog post imho
- 925dk 10y agoBy the way, HipChat still has no two-factor authentication.
- higon 10y agoJust got hit by this. Everybody in my team was using HipChat as a primary online communication tool. So, was nice to see nobody in the room for a while. Fine, but I wonder why they didn't reset the API tokens while resetting password immediately. Are they managed by the different servers/services?
- Sephr 10y agoI understand not disclosing the vulnerability itself, but if they won't even disclose the affected library's name then they are being grossly irresponsible or are under an NSL. If they are under an NSL and not just being irresponsible that would mean the vulnerability is part of one of the stolen NSA exploit kits. If I use the library, and it is non-essential for my business, then I should know what it is so that I can remove it.
- blauditore 10y ago> If you are a user of HipChat.com and have not received an email from our Security Team with these instructions, we have found no evidence that you are affected by this incident. Well, I didn't receive anything, but couldn't log in either. I had to google this to find out what the hell is going on - error messages on the login page are not helpful either, they just refuse login even after resetting the password.