11 ms·
Heroku: We’ve Heard Your Feedback
- cmckn 4y agoPosting this at 5pm on a Friday sounds about right, yeah?
- deleted 4y ago[deleted]
- nixgeek 4y agoThat isn’t a very charitable take. It’s been 24x7 for them, the acknowledgement is most welcome, in my opinion. I’m not sure waiting until Monday 9AM Pacific just to avoid these type of comments would be the right choice. The internet is 24x7 after all.
- deleted 4y ago[deleted]
- bpeebles 4y agoThere doesn't seem to be anything different now than 6 hours ago. So posting it after 8pm EDT/5pm PDT Friday seems very intentional.
- Sebb767 4y agoWith what intention? Honest question, I don't see a strategic advantage in posting this on a Friday. More of a disadvantage, actually, with less visibility.
- endisneigh 4y ago> With what intention? Honest question, I don't see a strategic advantage in posting this on a Friday. More of a disadvantage, actually, with less visibility. The intention presumably is to be able to say they communicated this while (intentionally) limiting the amount of awareness that's spread about this.
- bpeebles 4y agoThe point is less visibility/criticism. Since this is confirming pretty bad news (the attackers had access to the environment variables.) The concept of a Friday night news dump is a fairly well-worn, but basically true, trope.
- deleted 4y ago[deleted]
- orangepurple 4y agoWith any hope it will slide in the news cycle and be buried by any developments over the weekend.
- deleted 4y ago[deleted]
- sergiomattei 4y agoHeroku has always been the opposite of transparent. I still don't forget how they handled the Intelligent Routing fiasco from back in the day.
- nixgeek 4y agoMaybe that’ll change, there’s a new Sheriff in town as the saying goes? Bob was previously General Manager of AWS Elastic Kubernetes (EKS) which seemed to have a good reputation and to engage with its customers pretty well.
- mt42or 4y agoEKS was really bad in it's first days.
- deleted 4y ago[deleted]
- darknavi 4y ago> I started as Heroku GM a few weeks ago with intense enthusiasm to be a part of such a storied team. Wow talk about terrible timing.
- toyg 4y agoMaybe he's the guy who posted a couple of days ago about pulling a disk out of a RAID array on his first day in a leadership position...
- bigcat12345678 4y agoWell, it's a great timing. Fail it's because I am new; Succeed then it's my superior capability; plus, the learning rate during crisis is at least 10 times higher than the peaceful time, let alone the nerves and mental fortitude where peaceful time would never train one for.
- charcircuit 4y ago>The Heroku team and their colleagues have worked around the clock, including nights and weekends Can someone more familiar with an event like this tell me what they are working so hard on? I imagine securing the vulnerable service and resetting various credentials doesn't take that much work.
- nixgeek 4y agoOne quick example: combing through logs which can go back months or years can be expensive in time, especially if those are stored in something like Glacier and need to be pulled back into a system which permits analysis.
- riverdroid 4y agoBesides that, lots of incident response work to figure out the scope and impact of the breach, I'd imagine.
- jolux 4y agoYou have to make sure the attackers aren’t still in your network, you have to get them out if they are, you have to fully scope out what they messed with, and restore anything that’s plausibly connected. Even in the best case with all the right monitoring systems in place there is a lot of manual work involved from owners of the various different effected services and just managing the overall response adequately. I would say the amount of effort involved is comparable to managing a novel, ongoing SEV1.
- jonstewart 4y agoAdditionally there are legal concerns, for both regulatory compliance and for prepping for inevitable litigation. Those concerns aren’t necessarily a blocker for service restoration, but it really depends on the systems involved. If service could be restored by rebooting a system, for example, but that system also has data related to customers’ (and their customers’) PII and it might have been accessed by the attacker, then you need to make sure it’s all properly preserved forensically first, so that you can comply with regulations regarding breach notifications. The forensic analysis could then happen, but it’s definitely a “measure twice, cut once” situation with lots of lawyers involved (they won’t understand the systems, but they’ll make you explain everything so they can make decisions about risk; and they are in charge). Also, generally, it’s a “fog of war” scenario, where you can have so many unknowns to work through in a compressed time period, and sometimes there’s an active attacker and they get a vote, too.
- hthrowaway 4y agoAs a former herokai, communication was always the #1 point of discussion internally. This idea that they "can" do better is a half truth. They DID do better. I leave it as an exercise to the reader to determine what the limiting resource was here. By the way, how many senior devs and cofounders are left at Heroku Bob? Why doesn't it show up at dreamforce anymore?
- hthrowaway1 4y ago(Former) Salesforce employee here, long enough to have been there a couple of years before the Heroku acquisition. I can't let this stand without a comment. Heroku never seemed to want to integrate, but instead be the "cool kids", those who just do not have to worry about the enterprisy stuff such as automated backups, DR, high availability, enabling Java, etc. Everything they did was great, everything "Salesforce" sucked, yet they never made any money before the acquisition. The acquisition was a waste of money in first place (IMHO). The founders did not even try and so they left.
- glenngillen 4y agoFormer Heroku employee here: everything you list happened within the first year of acquisition. You’re correct that the lack of real integration with SFDC was a large part of the problem though. Though as far as SFDC acquisitions go Heroku was cheap. More expensive ones had far worse outcomes even with integration.
- throwawa3238273 4y agoTrying to integrate with SFDC was a mistake. The product never fit inside of the company. Of course their investment made Heroku successful over the first few years (and eventually highly profitable as I understand) but they should've just spun it off and let it succeed without trying to figure out how to "enterprise" it. What we knew how to do at Heroku was build a great platform for developers to launch apps. We never claimed to know how to make that model work for enterprise. In fact we were awful at trying.
- Aeolun 4y ago
- ec109685 4y agoCurious why they would need to keep the GitHub integration down for a while longer. Are they afraid someone could still grab credentials?
- jxi 4y agoThey were using a legacy GitHub integration. Maybe it's inherently unsafe and they need to rewrite things using the new API.
- djbusby 4y agoAudit?
- dogecoinbase 4y agoGitHub is not allowing them to turn it back on yet (source: multiple people on both sides)
- jhealy 4y agoGitHub should insist on it being converted to a GitHub App before they allow it to be re-enabled. That way org admins can see the requested permissions and control exactly which repos are permitted. GitHub OAuth apps are an absolute nightmare to audit or control.
- nomilk 4y ago> We’ve heard your feedback on our communications during this incident. You want more transparency, more in-depth information, and fewer “we are working on it” posts. Well, those, and: 1. Speed. It took days for heroku customers to be told about this. 2. Customers sign up at "heroku.com", the platform is called "Heroku", the CLI is "heroku", everything's heroku, so don't send emails from a parent company (Salesforce), send them from "Heroku". 3. Unambiguous info on what customers need to do. I had to guess based on HN comments whether config vars were accessed. Config vars are 100x more sensitive than code. Comms should be unambiguous and complete, and if incomplete for any reason, explain that (e.g. we don't know yet). 4. I still don't know whether having 1 Github Deploy on my Heroku account allowed unauthorized access to all heroku applications on my heroku account (i.e. those using other deploy methods, like `git push heroku main`). Were all my apps' repositories able to be accessed, or just the one(s) deployed via Github Deploys? 5. I still don't know whether unauthorized access was gained to all other GitHub respositories on my GitHub account, i.e. the repos that aren't heroku apps. These said, I still really appreciate that security incidents happen and aren't easy to deal with, and there's no obligation for anyone at a profitable company to actually care about semi-captive customers, so thanks to Heroku for the efforts; it's genuinely appreciated.
- ev1 4y agore: 2 interestingly, when i log in or reauthenticate, or log in through the heroku cli, I get sent to https://verify.salesforce.com/v1/verify/ https://verify.salesforce.com/v1/verify/
- izzygonzalez 4y agoWhen I saw the Salesforce email I thought I was being spammed or phished or getting a job offer.
- booi 4y agoThose are all the same thing
- ralston3 4y ago
- RileyJames 4y ago“I have a lifelong enthusiasm for developers and the experience they have building software together” And then drops a link to contact them, via LinkedIn… LinkedIn is the polar opposite of GitHub. It’s the worst example of social media, from its news feed, to spam invites. And it’s broken every rule in the “be a good netizen” play book, from constant spam, to slurping your email contacts and surveillance to the extreme. I struggle to imagine a developer saying “I’d like to contact xxx, and I’d love to do it via LinkedIn” Why not drop your email? Or a GitHub profile with a public email, and readme containing other contact methods, would have been more dev centric.
- mattl 4y agoEmail address is good but also an issue tracker or better yet IRC or something public would be good.
- RileyJames 4y agoAgreed, for the specific issue. But it seems like he’s asking ppl to contact him more generally. It does pose the question, what is the most developer friendly contact method?? GET /contacts POST /message Interested to hear ideas.
- fooqux 4y agoI guess it depends on what "developer friendly" means. If it means 'easiest method for a developer to contact me' that's email. It's ubiquitous. Everyone has at least one. It's easy. If it means 'easiest method for only developers to contact me' that would probably be some documented REST API. Either way, I agree the answer isn't Linkedin.
- ryanbrunner 4y agoEmail address and / or GitHub. No one's saying he needs to show off what a great programmer he is through his contact methods, just don't use a site that's infamously bad for dark patterns and recruiter spam.
- 4y ago
- nakovet 4y agoHeroku is a one of kind platform, many tried to replicate, none have come close, the ability to setup a whole app with a few clicks on dynos and add-ons is great. The price was always salty $50/month for 1gb RAM shared CPU, and in the few past years no newly released features comes to mind. As a customer I have no idea what this security issue impact, the communication has been poor as this post does not clarify much, you have to click a status post link to see a feed of what's going on.
- sergiotapia 4y agoSorry but Render at their lunch entirely. Try it. https://render.com/docs/deploy-rails https://render.com/docs/deploy-rails Want simple? Use a build.sh Want a bit complexity? Use a Dockerfile. Their docs are really copy paste for 99.9% applications. Whoever in that company made the decision to invest in docs was right on the money. It made switching to them a much easier choice. https://render.com https://render.com
- airstrike 4y ago> Whoever in that company made the decision to invest in docs was right on the money. Also the reason Django has always been so approachable (particularly in the early days, as it is orders of magnitude more complex now) and therefore popular Older still, IMHO also why mIRC scripting was so much fun despite its shortcomings. The help file had literally everything you could want to learn about the language and it almost begged to be read. It's how I got into coding
- skolsuper 4y ago+1, Django docs were gold standard when I last used it. Also DRF. I still miss them, 5 years later.
- spacemaiv 4y agoI hadn't heard of Render prior to this incident, but now I see it recommended in almost every related thread. Of course, Heroku's free tier had always worked well enough for my hobby apps, so I had no reason to search for alternatives. Looking forward to giving Render a try.
- sergiotapia 4y agoIf you're on Heroku how do you deploy now? It says it's still weeks away before you can deploy - did I read this right?
- ojame 4y agoStill weeks away from enabling the GitHub integration again, you can still manually deploy via the heroku CLI. Our team realised heroku was going to take weeks and we’ve replicated our “review app” development workflow with GitHub actions that clone apps on PR, push code and rebuild them on push and destroy them on PR close. It’s not as seamless as the heroku GitHub integration but it’s good enough for now.
- mepiethree 4y agoWoah… this would be an amazing thing to open source
- jrochkind1 4y agoYou don't even need the "heroku cli" to deploy -- just git. That was heroku's original famous innovation, you may recall!
- ceejayoz 4y agoYou can deploy, you just can't trigger the builds automatically by pushing to Github. (This also means the Review Apps system is non-functional.)
- umangsh 4y agoDeploy using heroku CLI: https://devcenter.heroku.com/articles/git#deploy-your-code https://devcenter.heroku.com/articles/git#deploy-your-code Unfortunately, git lfs isn't supported for CLI pushes, haven't found a way that works yet.
- ab-dm 4y agoSame way we’ve done for years, via our CD (circle CI), which uses the Heroku CLI We’ve been lucky that this entire event has, theoretically, not touched us as we never connected GitHub. That may change as more information comes to light. We’re still strongly considering moving to AWS, and are in the process of getting quotes from vendors.
- ezekg 4y ago> Additionally, we have no evidence that the attacker has accessed any customer accounts or decrypted customers’ environment variables. Now the attackers had access to encrypted environment variables?
- ezekg 4y agoMore context, from the latest email notification: > We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets.
- BeefWellington 4y ago> and the threat actor did not access the encryption key necessary to decrypt config var secrets. If the threat actor had access to any of the systems that use the key, they may not have needed to. Even this statement isn't clear that they couldn't have done it, but suggests that they don't think it's true... This is really bad incident response messaging.
- remram 4y agoWhere do I find those notifications? I only got a single email, stating I should "reset my user password as mentioned in our previous notification". Nothing prior, nothing since, and their password reset form shows nothing but "Internal Server Error". I've been getting some details from HN but I think I'm missing a lot of the picture since they obviously don't send everyone every email.
- psaux 4y agoLet’s give Bob a chance. I worked with him at AWS and prior. He is very committed to the customer, hence his post. We all try to communicate the best we can, so please try to support him in his new role. I launched several startups on Heroku over the last decade +, and feel they have gravity on trust. Some of the best devs I have worked with.
- steffanA 4y agoSorry, but this is a joke of a response. When they started sending out password reset emails, they should have explained why. Not only when people started complaining, and the media picked up on the lack of transparency.
- bin_bash 4y agoIt's ok: Bob says "Trust as our #1 value".
- tomatowurst 4y agocan somebody fill me in what happened? is this related to the oAuth vulnerability through github a while back?
- bin_bash 4y agoThey got hacked about a month ago and have been extremely cagey about what exactly was hacked ever since. Turns out the master database with encrypted username/passwords got leaked and encrypted environment variables were also leaked but it was like pulling teeth to get them to answer whether or not these happened or even admit that it might have been possible. Presumably more than this was also leaked but so far they haven't said anything on that. Env vars were the biggest concern on everyone's mind. They gave the absolute least amount of information over the longest period they could muster. The problem wasn't the hack really, it was the lack of transparency in the response. See: https://twitter.com/jacobian/status/1522782890957819906 https://twitter.com/jacobian/status/1522782890957819906
- tomatowurst 4y ago> Turns out the master database with encrypted username/passwords got leaked and encrypted environment variables were also leaked WTF!!!! That alone is disastrous enough, they should be reprimanded for this. Are there I'm sure, class action lawsuits happening? How much of an impact will this have on Salesforce? I mean imagine the data from that alone would be immensely valuable.
- mirrormaster 4y agoToo little, too late
- midislack 4y agoSo I looked at their product offerings and the Wikipedia page about Heroku and I'm still left scratching my head as to what it actually even is. They basically rent out other cloud computer providers' time at a higher price and make you use some kind of shitty middleware? Is that all this actually is? Please help me to understand what in the hell this even is.
- bin_bash 4y agoYou broadly have it right, but it's not "shitty" middleware, despite their problems the actual process of launching an app on Heroku is stellar and was especially groundbreaking 10 years ago. It would provision a server, install dependencies, configure, then launch your app just by pushing a git repository to them. They were the gold standard for rails deployments back in the day when rails was popular—which was important since rails was actually pretty hard to host on a server compared to say PHP.
- mdoms 4y agoJust cut the fucking marketing corpospeak, please, we're begging you. This post is just saturated with it.
- mt42or 4y ago> we have no evidence It means they don't know. They must say "we are confident" otherwise.
- benced 4y ago
- benced 4y agoThe author seems like they really care and are competent. They should go work for render.com
- nikon 4y agoI moved over to Render and cancelled Heroku. I didn't mind it before because it was reliable (if not expensive) but this Github fiasco made me move. Happy to get rid.