11 ms·

LogJ4 Security Inquiry – Response Required

323 points by elsombrero 5y ago

perihelions 5y ago
>"Thank you for your reply. Are you saying that we are not a customer of your organization?" Isn't this the sort of question you'd ask your own side, first?
- jaywalk 5y ago
  In a Fortune 500 company, I'd imagine it could be quite difficult to definitively prove that they are not a customer of any one organization. The company I work for is not Fortune 500, but we have several Fortune 500 customers. The amount of inane bullshit we have to deal with as a result is mind-boggling.
  - mrtksn 5y ago
    I recall an incident of large company paying whatever bill they receive and only to find out that they never had a contract with some of the companies and never receiving any service.
    - yjftsjthsd-h 5y ago
      If it makes you feel better, it goes both ways; I once signed up for a utility, got service, and discovered months later that they somehow completed the paperwork to give me service but not to actually bill me.
- Bedon292 5y ago
  I would bet this was sent out to a list that was put together that contained all of their "partners" which was in turn compiled from various other spreadsheets. Including one that had a 'support contact' column, or something like that and they assumed any that had that value was a partner. Over the course of the 6 years that the sheet has been cut and paste in various formats, they completely lost where any of it came from in the first place.
softwarebeware 5y ago
"...The level of ignorance and incompetence shown in this single email is mind-boggling...no code I’ve ever been involved with or have my copyright use log4j and any rookie or better engineer could easily verify that..." Yeah, well, I've been quite shocked how rookie some F500 devs can be and how dysfunctional large corporations can also be. Probably what happened here is someone wrote a script that compiled the dependencies of all projects they have and they sent this same email to all of them (!) regardless of any actual or potential use of log4j.
- boring_twenties 5y ago
  Let's hope they apply a similar amount of due diligence when the author responds with an offer to look into it for $800/hr with a 20 hour minimum.
  - hotpotamus 5y ago
    You're thinking small potatoes https://www.ign.com/articles/2019/03/26/man-steals-122-million-from-google-and-facebook-by-just-asking-them-for-money https://www.ign.com/articles/2019/03/26/man-steals-122-milli...
    - boring_twenties 5y ago
      I'm not even mad.
    - rootusrootus 5y ago
      We used to joke about doing this at my last company. We knew for a fact that our accounts payable folks frequently paid invoices without doing any verification that they were valid. I'm willing to guess this happens a lot more than people realize. I doubt we were the only people joking about it. People joke, other people hear, some of those follow up with action. The smart ones keep quiet and stop well before getting to $122M.
- dagw 5y ago
  When I worked at a large, but not F500, company I had to once every 6-12 month or so fill in a spreadsheet with all third-party dependencies, with their licenses and some other info, the project I was working on used. I then emailed this to a mystery person and never heard anything back ever. I can easily see someone pulling out these spreadsheets and just emailing away without any developer, rookie or otherwise, being aware of what was happening.
aero-glide2 5y ago
Reads like an automated email they sent to many people.
kzrdude 5y ago
I wonder what their reply is about. They probably have no idea what/who they are really talking to, and it's probably not some kind of legal trap.
- travoltaj 5y ago
  It's a reply to David/Daniels email to the F500 org. The dev didn't post a screenshot of their reply, but they mentioned this - "I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed."
  - Bedon292 5y ago
    There is a reply from the company at the bottom of the post.
    - kzrdude 5y ago
      and it's short and a bit befuddling
BiteCode_dev 5y ago
It's actually fantastic to receive such email. You can answer: "We are happy to provide you with support regarding this issue for $5000/day" Then if they accept, proceed to do nothing for 10 days, then reply you find none of your code is impacted and they are safe then bill them $50k.
- kube-system 5y ago
  It's fraud to bill someone T&M for time that wasn't actually spent. You're better off quoting it fixed-fee. :)
  - evan_ 5y ago
    Bill hourly with an 80 hour minimum. Then you can give them an invoice for 5 minutes to type the email and bill them for 80 hours.
  - trevormcneal 5y ago
    well said, and these companies have way too many lawyers with free time to keep suing you, even if you are right and the judge solves the case in your favour, not always it is required to cover legal expenses and the amount of legal fees burned on it won't worth. Fixed fee or monthly "support contract", with minimum of 1year.
  - nsoqo 5y ago
    “I had Martin explain to me three times what he got arrested for because it sounds an awful lot like what I do here every day.”
- csdvrx 5y ago
  > proceed to do nothing for 10 days That would be fraud. No, start grep on the source code and a few things like that, then provide the results: "a detailed audit found no reference to log4js, so another audit was started which found no reference to any java code in the C source; it was repeated 5 times to confirm these promising results. Another audit followed the Boltzman brain hypothesis to check if the affected log4js binary code could not be spontaneously generated during compilation, by following a Monte Carlo simulation to check for various length of binary data that would match the log4j binary code. (...) Finally, to avoid this extremely remote risk, the code changed to switch to reproducible builts, which can guarantee this will not happen"
rmoriz 5y ago
As far as I learned, a couple of big companies are sending this kind of mail to every provider, partner or copyright owner of code that they could find. I assume some developer/supplier used curl and provided a list of third party code and licenses they use. In the aftermath of the log4j incident, companies now target everyone about this issue partly to learn about potential exposure that they are not aware yet, eg exploited infrastructure of depending services like newsletter or analytics services. Yes, it's annoying and pointless to spam this mails to open source projects. But at least someone is now behind auditing the supply chain.
- cryptonector 5y ago
  It's a really dumb approach to vulnerability management for CYA. Spray and pray that the regulators are assuaged. It might even work as far as that goes. But obviously, it's not a sound approach to actual vulnerability management.
- ericcholis 5y ago
  I've read speculation that this is to cover their own asses with various regulations. Not sure if there's any weight behind this.
trevormcneal 5y ago
The first email looks like someone who had zero idea of what they were doing, just did some dependency scanning and got your name/email there, probably these emails were sent to everything that they could find. Quite well handled, not arrogant, not bending over and doing whatever they say, but being honest. If curl is impacted or not, may not really matters for them, usually these companies go after compliance and someone who they can blame when things go wrong.
protomyth 5y ago
Reminds me a bit of "Attack of the repo man", a classic case of this behavior http://acme.com/software/thttpd/repo.html http://acme.com/software/thttpd/repo.html
cryptonector 5y ago
> Are you saying that we are not a customer of your organization? LOL.
- gred 5y ago
  The cherry on top (if I'm reading the follow-up email correctly), is that he gets his name wrong (David vs Daniel).
- tomjen3 5y ago
  I work for a much small company, but I can't say that we are not a customer of X organization, because somebody else may be buying from them.
  - cryptonector 5y ago
    Yes, I know. It's still funny.
zokier 5y ago
I think it is pretty easy to see how this sort of thing happens: 1. Someone decides that we need inventory of all the libraries used (iirc requirement for some certifications and generally not a bad practice) 2. A system (/excel sheet) is enrolled where you have fields like $our_product, $library_used, $vendor_email 3. A dev, not quite understanding the point, dutifully fills in the data for the project they are working on 4. No-one reviews the data 5. Crisis strikes, so mass-send email to all vendors how they are handling it Problem here is around point 4.; for the process to work, someone should have reviewed the data to check that the used libraries are from vendors with some sort of support arrangement. I think the reply they provided is pretty promising, it makes it sound like they wanted to be a customer but are not only due an oversight.
- djbusby 5y ago
  I've been responsible for parts 1,2,3,4 and hand off the work to owners/managers/investors. There is a large time gap between 4 and 5 - and it seems everyone forgets who they hired for that supply chain "analysis" many moons ago.
verytrivial 5y ago
For everyone boggling at the tone of the email, stop for a moment and have a guess at how many different sources of software they think the average large corp has on their books let alone on their infra. It can literally be hundreds or thousands of different sources. And each of those will have their own topology. This is clearly a scatter-gun survey because they're realised they really have no idea of their exposure. (And before you re-boggle at that, there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues.)
- hotpotamus 5y ago
  Software eats the world just like a black hole.
- devadvance 5y ago
  Generally this is an accurate take. I'd add two things: > ...because they're realised they really have no idea of their exposure. This is partially because it is often non-engineers being asked to figure this out. The "information security analysts" at F500s are asked to do a lot of unfair work, such as analyze risks related to decades-old software they didn't build. > ...there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues. The first part (answering "what dependencies does my software have") isn't inherently bad. I'd emphasize the underinvestment in the second part more.
  - zerkten 5y ago
    >> The "information security analysts" at F500s are asked to do a lot of unfair work, such as analyze risks related to decades-old software they didn't build. I think that's putting it mildly. When it comes to responding, they'll look around and find that they only have a small number of full-time employees with the skills to partake in a response. Most of the IT organization will be dependent on vendors who struggle during the best times while their leadership has the ear of the CIO because IT is only viewed as cost. The full-time employees will frequently be the real heroes, but when the incident passes this won't be recognized. Things will repeat themselves with the next major vulnerability discovered, but the organization may find that they have even fewer employees at that point to lead a response.
notyourwork 5y ago
I find it a bit sad that a tech literate group is bashing a non-literate group fo people. The entire reason your salary is much larger than many other career paths is because of your ability to deal with technology. The premise that when the less educated and informed try to question something they don't understand only to be left with pandering and jabs is disingenuous. The questions although perhaps better phrased by someone with a more tech focused background are fine questions for a business to ask. Stop being douchebags and grow up.
- commandlinefan 5y ago
  I had the same thought - the e-mail really wasn't that unreasonable, coming from the perspective of somebody who didn't realize there was no support contract in place (and maybe didn't even understand how that could happen). Haxx's response seems similarly reasonable - we don't have a support contract, let's get one in place and then move forward from there. This really seems to be an object lesson that if you're depending on somebody for business-critical infrastructure, make sure they have a reason to support your business.
  - notyourwork 5y ago
    I agree, the response was reasonable. My frustration is with this hackernews thread and the constant judgement and snarky attitude we give to less tech literate folks. If everyone understood tech, we wouldn't be paid nearly the salaries we are for what we do.
- kahrl 5y ago
  I find it sad that the security department of a Fortune 500 company is sending out emails demanding OSS maintainers respond within 24 hours or else. You can feel sorry for the poor sap that was forced to embarrass himself, but it doesn't change the fact that everyone here feels like that company can get bent.
  - notyourwork 5y ago
    Why should the company get bent? Because some executive caught wind of a critical zero day and decided to have their company mitigate damage the same as any other company. Do you really think the security department in this specific company would not find this email dumb? In many cases, when things are reacted to hastily and in parallel its easy to take one action and generalize it to the whole company and not realize this is one of many actions the company took. No need to get bent out of shape over this and say this entire fortune 500 company is equally incompetent. If you think that you are not living in reality.
deltree7 5y ago
OK, a large corporation legal team doesn't understand the nuance of ownership of open-source software. Do we mock every single open source guy who displays the same amount of cluelessness about the inner workings of a business because I see plenty of that displayed here and everywhere else.
- Mvandenbergh 5y ago
  Understanding the nuances of ownership and who is responsible for what is quite an important skill for corporate lawyers.
  - deltree7 5y ago
    If you are dealing with 1000s of cases, you can't apply nuances to every single of them. You are all are supposed to be smart software engineers. Probably know about pre-mature optimization and efficient path. Here's a secret about communications -- Mass emailing works and is very efficient. I'm sure you are the same person who rants about a recruiter reaching out to you even though you are the creator of Python. Reading through everyone's resume and tailoring a message is a waste of time and has the worst ROI for any salesperson. "But Ha Ha Ha, you guys are clueless about not knowing operational efficiency of an mass communications. Ha Ha Ha" Yeah, that's exactly how this sounds if the other side mocks HN/Engineers the same way you mock Sales and other "mass-outreach programs"
    - Hackbraten 5y ago
      > Here's a secret about communications -- Mass emailing works and is very efficient. Especially for the sender.
- deleted 5y ago
  [deleted]
- gautamdivgi 5y ago
  Ummm... I think the curl license is displayed pretty publicly. So, yes - this email deserves to be mocked roundly.
0xbadcafebee 5y ago
Welcome to Corporate Life. Somebody at the top says "Make sure we find out from all vendors what their log4j impact is", and that trickles down until some poor sap in InfoSec is told to do it. And of course "all vendors" includes "open source vendors", aka some dude named Carl in Uzbekistan who wrote a Node.js module. Since InfoSec sap shouldn't even have been tasked with this ridiculous ask, and he's got 10,000 of them to send, he sends a form letter.
jandrese 5y ago
I have a feeling that some security automaton at a major corporation is about to have their mind blown when they discover the world of Open Source Software. They had absolutely no idea that non-commercial software was even a thing.
0x500x79 5y ago
Many organizations document their 3rd party vendors and libraries and it doesn't surprise me that an automated email reached Daniel. Most likely someone mis-documented using one of Daniel's projects in a spreadsheet. I am personally a bit surprised about the responses here. It is completely reasonable for this email to reach Daniel and is most likely an artifact of bad documentation by engineers in the company. At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on. The response is as simple as "What library/product does this email pertain to?", "Please see the licenses for the libraries or products in question.", and what Daniel responded with as well: "I would be willing the dig in further for specific questions with a support contract.".
- MrStonedOne 5y ago
- matheusmoreira 5y ago
  > At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on. That alone is extremely disrespectful, it means they couldn't care less about the time of open source software maintainers. To say nothing of their "request" for review.
  - 0x500x79 5y ago
    It's not about open source maintainers. This isn't an "open source" problem further than the fact that Daniel's software is used in a product they are using. Daniel could take a couple of seconds to ignore this email and there was very little time wasted. The real "disrespect" should be whatever engineer put Daniel's name into the spreadsheet that blasted out these emails. Someone didn't do their job and is checking a box. How is the (possibly non-technical) person that is required for managing 100s of vendors and thousands of open source libraries supposed to verify all of that information? I'm personally happy to hear that this company is trying to do SOMETHING to make sure that Log4j is patched even if it's a bit incompetent in it's implementation. There is not malice here.
sandworm101 5y ago
Not every open source project is run by the little guy. I want to see a a security vulnerability in something like AES. Then the complaint emails demanding answers in 24 hours would be going to nsa.gov addresses. Anyone leading a shareholder action would love to see these emails. They are basic admissions that the company doesn't know how or from where it gets essential software.
deleted 5y ago
[deleted]
dudeinjapan 5y ago
More accurately "a clueless IT lackey at a Fortune 500 company" sent the mail. I doubt the chairman was pounding the board table and barking "We demand answers from Haxx!"
- daenney 5y ago
  It didn’t come from IT/engineering. This is legal/compliance.
  - NoboruWataya 5y ago
    It's signed off "Information Security". The template email might have been drafted by legal/compliance, but it is surely for the IT guys to figure out what code they use in their tech stack and lead those discussions.
  - dudeinjapan 5y ago
    Even more clueless then!
zeroesandones 5y ago
seems like it's automated, they must be aware it's oss
josephcsible 5y ago
Why black out the company name? Confidentiality notices at the bottom of emails aren't legally binding, especially when it's an unsolicited email from a company you have no relationship with.
- Nicksil 5y ago
  >Why black out the company name? Confidentiality notices at the bottom of emails aren't legally binding, especially when it's an unsolicited email from a company you have no relationship with. It's explained in the article.
- jffry 5y ago
  This was addressed directly within the linked blog post: > In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.)
louissan 5y ago
https://xkcd.com/2347/ https://xkcd.com/2347/
phendrenad2 5y ago
You can learn a lot from this. This is how efficient companies operate. No one who knows the difference between C and Java was involved in the sending of this letter. If they were, that would be a waste of resources.
MrStonedOne 5y ago
1970-01-01 5y ago
>>I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed. This made my day. If a wealthy individual takes your tools and then calls for help while fixing-up their shed with said tools, do not move a muscle until you agree on the fee.
- t0mas88 5y ago
  I expect they'll gladly sign a support contract with Daniel. As a commercial SaaS vendor we received these same emails from all of the major banks / insurance companies. It's interesting to see that we got some on the Monday after the issue was discovered and some a few weeks later, with some showing a clear understanding of the risk in the context of our product and some looking like a standard copy/paste. Gives you a rare behind-the-scenes view of the information security practices of these companies.
stavros 5y ago
I haven't seen anyone here comment on this, but I loved "Hi David" in response to Daniel's reply.
devit 5y ago
The document uses a monospace font, and the redacted name can be seen to be 10 characters long. Based on the 2019 Fortune 500 list, that gives these possible candidates: Activision, Alaska Air, Albertsons, Altice USA, Amazon.com, Ameriprise, AutoNation, BB&T Corp., Bed Bath &, Blackstone, Booz Allen, BorgWarner, Burlington, CBRE Group, Chesapeake, CMS Energy, CVS Health, Dean Foods, DTE Energy, Enterprise, Eversource, Expeditors, Fannie Mae, First Data, Ford Motor, Home Depot, Huntington, JM Smucker, Jones Lang, Laboratory, Mastercard, McDonald's, Murphy USA, Nationwide, News Corp., NGL Energy, NRG Energy, Occidental, PBF Energy, Prudential, PulteGroup, S&P Global, State Farm, Unum Group, US Bancorp, WEC Energy, Windstream, World Fuel, WR Berkley, Yum Brands
- knob 5y ago
  This analysis is beautiful. Thank you for doing the math! :-)
- aerovistae 5y ago
  "In the picture version of the email I padded the name fields to better anonymize the sender, and in the text below I replaced them with NNNN."
- deleted 5y ago
  [deleted]
- Miner49er 5y ago
  From the article: "The email comes from a fortune-500 multi-billion dollar company that apparently might be using a product that contains my code, or maybe they have customers who do. Who knows?" The "or maybe they have customers who do" makes me think that this company must provide services to other companies, so probably not a Mcdonald's or Albertson's or something like that.
  - merb 5y ago
    McDonalds provides services to other companies? that's the whole point of McDonalds?
gkoberger 5y ago
I don't want to defend this company, but my company (a dev tool used by many other companies) receives a handful of these a day. It's almost the exact same email, and they're just mass-sending them. It's not personal, and it's pretty standard. The tone feels off if you assume a human wrote it. But that's only because it's a form letter their legal department wrote for them to send off. They probably collected "dependencies" from the entire company (and someone wrote "curl"), and sent a mass email. If you just reply with a simple "We're unaffected!" (or ignore them), you'll never hear from them again.
- throw8932894 5y ago
  Better to reply "yes, we are affected. Your support contract has expired, please renew at XYZ".
  - gkoberger 5y ago
    So... lie to someone who is making an effort to protect customer data in order to steal money?
  - Jiro 5y ago
    If there's no expired support contract, that would be making a false statement of fact in order to get someone to sign a contract and pay me money. It's plausible that that would be fraud. Of course it's also plausible that that's not fraud at all. But I have no way to know for sure unless I ask a lawyer, which needless to say I wouldn't do. And if it turns out that it is fraud, well, the legal department of Fortune 500 companies tends to be pretty humorless.
    - throw8932894 5y ago
      Opensource license is a form of contract. I provide free 5 minute support to new users. And good luck suing me if I am not even US/EU based. Departments (small managers) are authorized to spend small money without approval, lets say up to 200 euro/month. If they send this type of emails, someone ass is on fire. They will DO spend it just to get legal green light. Anyway, I do not see reason to hold back, just because I am open source developer.
bastardoperator 5y ago
Versus asking for a support contract because I don't really want to support anyone like this long term, I would have sent an invoice... If it gets paid, I answer the questions, if it doesn't everyone knows where everyone stands. I also think it's easier to get an invoice paid versus trying to negotiate a support contract.
- daenney 5y ago
  He’s not trying to negotiate a support contract. It’s a polite “fuck off”.
  - executesorder66 5y ago
    Yes he is. > In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.) If he wasn't trying to land a contract, then he would have posted the company name.
rdiddly 5y ago
A tangential point, and granted it's been around and rightly ridiculed since probably the 90s, but how 'bout that classic signature about CONFIDENTIALity? It's like the icing on the don't-know-how-computers-work cake!
alkonaut 5y ago
If you want a company to change their behavior, give them a reason to do so. Daniel has quite a platform being a well known maintainer, but instead of using that platform to shame the company in question, he politely emails back to the person sitting with an outdated excel sheet of 500 suppliers. That person didn't decide that the "email everyone on this list demanding info" strategy was a good idea. To actually make a difference when you have a platform, use it. Tweet-shame them so that the fallout actually reaches the manager in question. This is just complaining about a behahavior while at the same time more or less doing everything possible to encourage that behavior.
bartread 5y ago
This is golden and characterises a surprising amount of my experience of communications with large corporations: >> Thank you for your reply. Are you saying that we are not a customer of your organization? It's just so beautifully orthogonal. Oh, and they got his name wrong in the salutation.
specialist 5y ago
Tangent: Wasn't Java's SecurityManager stuff supposed to prevent these kinds of exploits? I haven't used log4j for ages, so I didn't know offhand. Somewhat curious, I gleened that none of the enterprisey stacks use SecurityManager. I guess I kinda understand; SecurityManager was fashioned and pitched for an ecosystem of applets, agents, and sandboxes. Further, I then gleened there's a JSR to outright remove SecurityManager. With no apparent replacement, just some vague advice to roll your own capabilities based system. So, however we got here, what's then plan? Run JVMs on top of something like OpenBSD's pledge?
Bedon292 5y ago
It reads like this was a form letter that was to be sent out to all their actual paid partners, and after the massive game of telephone that is corporate hierarchy, it somehow became all dependencies they had contacts for. And somewhere someone filled out a form, probably years ago, with his email on it as the maintainer of a dependency they use (because leaving it blank isn't allowed). And he got caught up in that mass email, totally dumb, but also could easily see how it can happen.
aahortwwy 5y ago
Is there a product out there that makes it easy for open source maintainers to offer enterprise support services? I think support is probably the best way of making money from open source, but a lot of maintainers are unlikely to have everything set up to do so (business entities, contracts, ways to receive payment, probably a dozen other things that you'd never think of, etc.). Like Stripe Atlas for open source consulting?
slim 5y ago
I think daniel's reaction is appropriate and well thought. One can suppose thousands of these emails asking for free work have been sent. There is close to zero chance his demand for a support contract would get past the first filter. Whereas making a blog post about it makes for a good story and also has more chances to get the attention of the right people at this company. Even if it's slightly aggressive.

323 points by elsombrero 5y ago

>"Thank you for your reply. Are you saying that we are not a customer of your organization?" Isn't this the sort of question you'd ask your own side, first?

jaywalk 5y ago
In a Fortune 500 company, I'd imagine it could be quite difficult to definitively prove that they are not a customer of any one organization. The company I work for is not Fortune 500, but we have several Fortune 500 customers. The amount of inane bullshit we have to deal with as a result is mind-boggling.
- mrtksn 5y ago
  I recall an incident of large company paying whatever bill they receive and only to find out that they never had a contract with some of the companies and never receiving any service.
  - yjftsjthsd-h 5y ago
    If it makes you feel better, it goes both ways; I once signed up for a utility, got service, and discovered months later that they somehow completed the paperwork to give me service but not to actually bill me.
Bedon292 5y ago
I would bet this was sent out to a list that was put together that contained all of their "partners" which was in turn compiled from various other spreadsheets. Including one that had a 'support contact' column, or something like that and they assumed any that had that value was a partner. Over the course of the 6 years that the sheet has been cut and paste in various formats, they completely lost where any of it came from in the first place.

softwarebeware 5y ago

"...The level of ignorance and incompetence shown in this single email is mind-boggling...no code I’ve ever been involved with or have my copyright use log4j and any rookie or better engineer could easily verify that..." Yeah, well, I've been quite shocked how rookie some F500 devs can be and how dysfunctional large corporations can also be. Probably what happened here is someone wrote a script that compiled the dependencies of all projects they have and they sent this same email to all of them (!) regardless of any actual or potential use of log4j.

boring_twenties 5y ago
Let's hope they apply a similar amount of due diligence when the author responds with an offer to look into it for $800/hr with a 20 hour minimum.
- hotpotamus 5y ago
  You're thinking small potatoes https://www.ign.com/articles/2019/03/26/man-steals-122-million-from-google-and-facebook-by-just-asking-them-for-money https://www.ign.com/articles/2019/03/26/man-steals-122-milli...
  - boring_twenties 5y ago
    I'm not even mad.
  - rootusrootus 5y ago
    We used to joke about doing this at my last company. We knew for a fact that our accounts payable folks frequently paid invoices without doing any verification that they were valid. I'm willing to guess this happens a lot more than people realize. I doubt we were the only people joking about it. People joke, other people hear, some of those follow up with action. The smart ones keep quiet and stop well before getting to $122M.
dagw 5y ago
When I worked at a large, but not F500, company I had to once every 6-12 month or so fill in a spreadsheet with all third-party dependencies, with their licenses and some other info, the project I was working on used. I then emailed this to a mystery person and never heard anything back ever. I can easily see someone pulling out these spreadsheets and just emailing away without any developer, rookie or otherwise, being aware of what was happening.

aero-glide2 5y ago

Reads like an automated email they sent to many people.

kzrdude 5y ago

I wonder what their reply is about. They probably have no idea what/who they are really talking to, and it's probably not some kind of legal trap.

travoltaj 5y ago
It's a reply to David/Daniels email to the F500 org. The dev didn't post a screenshot of their reply, but they mentioned this - "I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed."
- Bedon292 5y ago
  There is a reply from the company at the bottom of the post.
  - kzrdude 5y ago
    and it's short and a bit befuddling

BiteCode_dev 5y ago

It's actually fantastic to receive such email. You can answer: "We are happy to provide you with support regarding this issue for $5000/day" Then if they accept, proceed to do nothing for 10 days, then reply you find none of your code is impacted and they are safe then bill them $50k.

kube-system 5y ago
It's fraud to bill someone T&M for time that wasn't actually spent. You're better off quoting it fixed-fee. :)
- evan_ 5y ago
  Bill hourly with an 80 hour minimum. Then you can give them an invoice for 5 minutes to type the email and bill them for 80 hours.
- trevormcneal 5y ago
  well said, and these companies have way too many lawyers with free time to keep suing you, even if you are right and the judge solves the case in your favour, not always it is required to cover legal expenses and the amount of legal fees burned on it won't worth. Fixed fee or monthly "support contract", with minimum of 1year.
- nsoqo 5y ago
  “I had Martin explain to me three times what he got arrested for because it sounds an awful lot like what I do here every day.”
csdvrx 5y ago
> proceed to do nothing for 10 days That would be fraud. No, start grep on the source code and a few things like that, then provide the results: "a detailed audit found no reference to log4js, so another audit was started which found no reference to any java code in the C source; it was repeated 5 times to confirm these promising results. Another audit followed the Boltzman brain hypothesis to check if the affected log4js binary code could not be spontaneously generated during compilation, by following a Monte Carlo simulation to check for various length of binary data that would match the log4j binary code. (...) Finally, to avoid this extremely remote risk, the code changed to switch to reproducible builts, which can guarantee this will not happen"

rmoriz 5y ago

As far as I learned, a couple of big companies are sending this kind of mail to every provider, partner or copyright owner of code that they could find. I assume some developer/supplier used curl and provided a list of third party code and licenses they use. In the aftermath of the log4j incident, companies now target everyone about this issue partly to learn about potential exposure that they are not aware yet, eg exploited infrastructure of depending services like newsletter or analytics services. Yes, it's annoying and pointless to spam this mails to open source projects. But at least someone is now behind auditing the supply chain.

cryptonector 5y ago
It's a really dumb approach to vulnerability management for CYA. Spray and pray that the regulators are assuaged. It might even work as far as that goes. But obviously, it's not a sound approach to actual vulnerability management.
ericcholis 5y ago
I've read speculation that this is to cover their own asses with various regulations. Not sure if there's any weight behind this.

trevormcneal 5y ago

The first email looks like someone who had zero idea of what they were doing, just did some dependency scanning and got your name/email there, probably these emails were sent to everything that they could find. Quite well handled, not arrogant, not bending over and doing whatever they say, but being honest. If curl is impacted or not, may not really matters for them, usually these companies go after compliance and someone who they can blame when things go wrong.

protomyth 5y ago

Reminds me a bit of "Attack of the repo man", a classic case of this behavior http://acme.com/software/thttpd/repo.html http://acme.com/software/thttpd/repo.html

cryptonector 5y ago

> Are you saying that we are not a customer of your organization? LOL.

gred 5y ago
The cherry on top (if I'm reading the follow-up email correctly), is that he gets his name wrong (David vs Daniel).
tomjen3 5y ago
I work for a much small company, but I can't say that we are not a customer of X organization, because somebody else may be buying from them.
- cryptonector 5y ago
  Yes, I know. It's still funny.

zokier 5y ago

I think it is pretty easy to see how this sort of thing happens: 1. Someone decides that we need inventory of all the libraries used (iirc requirement for some certifications and generally not a bad practice) 2. A system (/excel sheet) is enrolled where you have fields like $our_product, $library_used, $vendor_email 3. A dev, not quite understanding the point, dutifully fills in the data for the project they are working on 4. No-one reviews the data 5. Crisis strikes, so mass-send email to all vendors how they are handling it Problem here is around point 4.; for the process to work, someone should have reviewed the data to check that the used libraries are from vendors with some sort of support arrangement. I think the reply they provided is pretty promising, it makes it sound like they wanted to be a customer but are not only due an oversight.

djbusby 5y ago
I've been responsible for parts 1,2,3,4 and hand off the work to owners/managers/investors. There is a large time gap between 4 and 5 - and it seems everyone forgets who they hired for that supply chain "analysis" many moons ago.

verytrivial 5y ago

For everyone boggling at the tone of the email, stop for a moment and have a guess at how many different sources of software they think the average large corp has on their books let alone on their infra. It can literally be hundreds or thousands of different sources. And each of those will have their own topology. This is clearly a scatter-gun survey because they're realised they really have no idea of their exposure. (And before you re-boggle at that, there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues.)

hotpotamus 5y ago
Software eats the world just like a black hole.
devadvance 5y ago
Generally this is an accurate take. I'd add two things: > ...because they're realised they really have no idea of their exposure. This is partially because it is often non-engineers being asked to figure this out. The "information security analysts" at F500s are asked to do a lot of unfair work, such as analyze risks related to decades-old software they didn't build. > ...there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues. The first part (answering "what dependencies does my software have") isn't inherently bad. I'd emphasize the underinvestment in the second part more.
- zerkten 5y ago
  >> The "information security analysts" at F500s are asked to do a lot of unfair work, such as analyze risks related to decades-old software they didn't build. I think that's putting it mildly. When it comes to responding, they'll look around and find that they only have a small number of full-time employees with the skills to partake in a response. Most of the IT organization will be dependent on vendors who struggle during the best times while their leadership has the ear of the CIO because IT is only viewed as cost. The full-time employees will frequently be the real heroes, but when the incident passes this won't be recognized. Things will repeat themselves with the next major vulnerability discovered, but the organization may find that they have even fewer employees at that point to lead a response.

notyourwork 5y ago

I find it a bit sad that a tech literate group is bashing a non-literate group fo people. The entire reason your salary is much larger than many other career paths is because of your ability to deal with technology. The premise that when the less educated and informed try to question something they don't understand only to be left with pandering and jabs is disingenuous. The questions although perhaps better phrased by someone with a more tech focused background are fine questions for a business to ask. Stop being douchebags and grow up.

commandlinefan 5y ago
I had the same thought - the e-mail really wasn't that unreasonable, coming from the perspective of somebody who didn't realize there was no support contract in place (and maybe didn't even understand how that could happen). Haxx's response seems similarly reasonable - we don't have a support contract, let's get one in place and then move forward from there. This really seems to be an object lesson that if you're depending on somebody for business-critical infrastructure, make sure they have a reason to support your business.
- notyourwork 5y ago
  I agree, the response was reasonable. My frustration is with this hackernews thread and the constant judgement and snarky attitude we give to less tech literate folks. If everyone understood tech, we wouldn't be paid nearly the salaries we are for what we do.
kahrl 5y ago
I find it sad that the security department of a Fortune 500 company is sending out emails demanding OSS maintainers respond within 24 hours or else. You can feel sorry for the poor sap that was forced to embarrass himself, but it doesn't change the fact that everyone here feels like that company can get bent.
- notyourwork 5y ago
  Why should the company get bent? Because some executive caught wind of a critical zero day and decided to have their company mitigate damage the same as any other company. Do you really think the security department in this specific company would not find this email dumb? In many cases, when things are reacted to hastily and in parallel its easy to take one action and generalize it to the whole company and not realize this is one of many actions the company took. No need to get bent out of shape over this and say this entire fortune 500 company is equally incompetent. If you think that you are not living in reality.

deltree7 5y ago

OK, a large corporation legal team doesn't understand the nuance of ownership of open-source software. Do we mock every single open source guy who displays the same amount of cluelessness about the inner workings of a business because I see plenty of that displayed here and everywhere else.

Mvandenbergh 5y ago
Understanding the nuances of ownership and who is responsible for what is quite an important skill for corporate lawyers.
- deltree7 5y ago
  If you are dealing with 1000s of cases, you can't apply nuances to every single of them. You are all are supposed to be smart software engineers. Probably know about pre-mature optimization and efficient path. Here's a secret about communications -- Mass emailing works and is very efficient. I'm sure you are the same person who rants about a recruiter reaching out to you even though you are the creator of Python. Reading through everyone's resume and tailoring a message is a waste of time and has the worst ROI for any salesperson. "But Ha Ha Ha, you guys are clueless about not knowing operational efficiency of an mass communications. Ha Ha Ha" Yeah, that's exactly how this sounds if the other side mocks HN/Engineers the same way you mock Sales and other "mass-outreach programs"
  - Hackbraten 5y ago
    > Here's a secret about communications -- Mass emailing works and is very efficient. Especially for the sender.
deleted 5y ago
[deleted]
gautamdivgi 5y ago
Ummm... I think the curl license is displayed pretty publicly. So, yes - this email deserves to be mocked roundly.

0xbadcafebee 5y ago

Welcome to Corporate Life. Somebody at the top says "Make sure we find out from all vendors what their log4j impact is", and that trickles down until some poor sap in InfoSec is told to do it. And of course "all vendors" includes "open source vendors", aka some dude named Carl in Uzbekistan who wrote a Node.js module. Since InfoSec sap shouldn't even have been tasked with this ridiculous ask, and he's got 10,000 of them to send, he sends a form letter.

jandrese 5y ago

I have a feeling that some security automaton at a major corporation is about to have their mind blown when they discover the world of Open Source Software. They had absolutely no idea that non-commercial software was even a thing.

0x500x79 5y ago

Many organizations document their 3rd party vendors and libraries and it doesn't surprise me that an automated email reached Daniel. Most likely someone mis-documented using one of Daniel's projects in a spreadsheet. I am personally a bit surprised about the responses here. It is completely reasonable for this email to reach Daniel and is most likely an artifact of bad documentation by engineers in the company. At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on. The response is as simple as "What library/product does this email pertain to?", "Please see the licenses for the libraries or products in question.", and what Daniel responded with as well: "I would be willing the dig in further for specific questions with a support contract.".

MrStonedOne 5y ago
matheusmoreira 5y ago
> At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on. That alone is extremely disrespectful, it means they couldn't care less about the time of open source software maintainers. To say nothing of their "request" for review.
- 0x500x79 5y ago
  It's not about open source maintainers. This isn't an "open source" problem further than the fact that Daniel's software is used in a product they are using. Daniel could take a couple of seconds to ignore this email and there was very little time wasted. The real "disrespect" should be whatever engineer put Daniel's name into the spreadsheet that blasted out these emails. Someone didn't do their job and is checking a box. How is the (possibly non-technical) person that is required for managing 100s of vendors and thousands of open source libraries supposed to verify all of that information? I'm personally happy to hear that this company is trying to do SOMETHING to make sure that Log4j is patched even if it's a bit incompetent in it's implementation. There is not malice here.

sandworm101 5y ago

Not every open source project is run by the little guy. I want to see a a security vulnerability in something like AES. Then the complaint emails demanding answers in 24 hours would be going to nsa.gov addresses. Anyone leading a shareholder action would love to see these emails. They are basic admissions that the company doesn't know how or from where it gets essential software.

deleted 5y ago

[deleted]

dudeinjapan 5y ago

More accurately "a clueless IT lackey at a Fortune 500 company" sent the mail. I doubt the chairman was pounding the board table and barking "We demand answers from Haxx!"

daenney 5y ago
It didn’t come from IT/engineering. This is legal/compliance.
- NoboruWataya 5y ago
  It's signed off "Information Security". The template email might have been drafted by legal/compliance, but it is surely for the IT guys to figure out what code they use in their tech stack and lead those discussions.
- dudeinjapan 5y ago
  Even more clueless then!

zeroesandones 5y ago

seems like it's automated, they must be aware it's oss

josephcsible 5y ago

Why black out the company name? Confidentiality notices at the bottom of emails aren't legally binding, especially when it's an unsolicited email from a company you have no relationship with.

Nicksil 5y ago
>Why black out the company name? Confidentiality notices at the bottom of emails aren't legally binding, especially when it's an unsolicited email from a company you have no relationship with. It's explained in the article.
jffry 5y ago
This was addressed directly within the linked blog post: > In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.)

louissan 5y ago

https://xkcd.com/2347/ https://xkcd.com/2347/

phendrenad2 5y ago

You can learn a lot from this. This is how efficient companies operate. No one who knows the difference between C and Java was involved in the sending of this letter. If they were, that would be a waste of resources.

MrStonedOne 5y ago

1970-01-01 5y ago

>>I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed. This made my day. If a wealthy individual takes your tools and then calls for help while fixing-up their shed with said tools, do not move a muscle until you agree on the fee.

t0mas88 5y ago
I expect they'll gladly sign a support contract with Daniel. As a commercial SaaS vendor we received these same emails from all of the major banks / insurance companies. It's interesting to see that we got some on the Monday after the issue was discovered and some a few weeks later, with some showing a clear understanding of the risk in the context of our product and some looking like a standard copy/paste. Gives you a rare behind-the-scenes view of the information security practices of these companies.

stavros 5y ago

I haven't seen anyone here comment on this, but I loved "Hi David" in response to Daniel's reply.

devit 5y ago

The document uses a monospace font, and the redacted name can be seen to be 10 characters long. Based on the 2019 Fortune 500 list, that gives these possible candidates: Activision, Alaska Air, Albertsons, Altice USA, Amazon.com, Ameriprise, AutoNation, BB&T Corp., Bed Bath &, Blackstone, Booz Allen, BorgWarner, Burlington, CBRE Group, Chesapeake, CMS Energy, CVS Health, Dean Foods, DTE Energy, Enterprise, Eversource, Expeditors, Fannie Mae, First Data, Ford Motor, Home Depot, Huntington, JM Smucker, Jones Lang, Laboratory, Mastercard, McDonald's, Murphy USA, Nationwide, News Corp., NGL Energy, NRG Energy, Occidental, PBF Energy, Prudential, PulteGroup, S&P Global, State Farm, Unum Group, US Bancorp, WEC Energy, Windstream, World Fuel, WR Berkley, Yum Brands

knob 5y ago
This analysis is beautiful. Thank you for doing the math! :-)
aerovistae 5y ago
"In the picture version of the email I padded the name fields to better anonymize the sender, and in the text below I replaced them with NNNN."
deleted 5y ago
[deleted]
Miner49er 5y ago
From the article: "The email comes from a fortune-500 multi-billion dollar company that apparently might be using a product that contains my code, or maybe they have customers who do. Who knows?" The "or maybe they have customers who do" makes me think that this company must provide services to other companies, so probably not a Mcdonald's or Albertson's or something like that.
- merb 5y ago
  McDonalds provides services to other companies? that's the whole point of McDonalds?

gkoberger 5y ago

I don't want to defend this company, but my company (a dev tool used by many other companies) receives a handful of these a day. It's almost the exact same email, and they're just mass-sending them. It's not personal, and it's pretty standard. The tone feels off if you assume a human wrote it. But that's only because it's a form letter their legal department wrote for them to send off. They probably collected "dependencies" from the entire company (and someone wrote "curl"), and sent a mass email. If you just reply with a simple "We're unaffected!" (or ignore them), you'll never hear from them again.

throw8932894 5y ago
Better to reply "yes, we are affected. Your support contract has expired, please renew at XYZ".
- gkoberger 5y ago
  So... lie to someone who is making an effort to protect customer data in order to steal money?
- Jiro 5y ago
  If there's no expired support contract, that would be making a false statement of fact in order to get someone to sign a contract and pay me money. It's plausible that that would be fraud. Of course it's also plausible that that's not fraud at all. But I have no way to know for sure unless I ask a lawyer, which needless to say I wouldn't do. And if it turns out that it is fraud, well, the legal department of Fortune 500 companies tends to be pretty humorless.
  - throw8932894 5y ago
    Opensource license is a form of contract. I provide free 5 minute support to new users. And good luck suing me if I am not even US/EU based. Departments (small managers) are authorized to spend small money without approval, lets say up to 200 euro/month. If they send this type of emails, someone ass is on fire. They will DO spend it just to get legal green light. Anyway, I do not see reason to hold back, just because I am open source developer.

bastardoperator 5y ago

Versus asking for a support contract because I don't really want to support anyone like this long term, I would have sent an invoice... If it gets paid, I answer the questions, if it doesn't everyone knows where everyone stands. I also think it's easier to get an invoice paid versus trying to negotiate a support contract.

daenney 5y ago
He’s not trying to negotiate a support contract. It’s a polite “fuck off”.
- executesorder66 5y ago
  Yes he is. > In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.) If he wasn't trying to land a contract, then he would have posted the company name.

rdiddly 5y ago

A tangential point, and granted it's been around and rightly ridiculed since probably the 90s, but how 'bout that classic signature about CONFIDENTIALity? It's like the icing on the don't-know-how-computers-work cake!

alkonaut 5y ago

If you want a company to change their behavior, give them a reason to do so. Daniel has quite a platform being a well known maintainer, but instead of using that platform to shame the company in question, he politely emails back to the person sitting with an outdated excel sheet of 500 suppliers. That person didn't decide that the "email everyone on this list demanding info" strategy was a good idea. To actually make a difference when you have a platform, use it. Tweet-shame them so that the fallout actually reaches the manager in question. This is just complaining about a behahavior while at the same time more or less doing everything possible to encourage that behavior.

bartread 5y ago

This is golden and characterises a surprising amount of my experience of communications with large corporations: >> Thank you for your reply. Are you saying that we are not a customer of your organization? It's just so beautifully orthogonal. Oh, and they got his name wrong in the salutation.

specialist 5y ago

Tangent: Wasn't Java's SecurityManager stuff supposed to prevent these kinds of exploits? I haven't used log4j for ages, so I didn't know offhand. Somewhat curious, I gleened that none of the enterprisey stacks use SecurityManager. I guess I kinda understand; SecurityManager was fashioned and pitched for an ecosystem of applets, agents, and sandboxes. Further, I then gleened there's a JSR to outright remove SecurityManager. With no apparent replacement, just some vague advice to roll your own capabilities based system. So, however we got here, what's then plan? Run JVMs on top of something like OpenBSD's pledge?

Bedon292 5y ago

It reads like this was a form letter that was to be sent out to all their actual paid partners, and after the massive game of telephone that is corporate hierarchy, it somehow became all dependencies they had contacts for. And somewhere someone filled out a form, probably years ago, with his email on it as the maintainer of a dependency they use (because leaving it blank isn't allowed). And he got caught up in that mass email, totally dumb, but also could easily see how it can happen.

aahortwwy 5y ago

Is there a product out there that makes it easy for open source maintainers to offer enterprise support services? I think support is probably the best way of making money from open source, but a lot of maintainers are unlikely to have everything set up to do so (business entities, contracts, ways to receive payment, probably a dozen other things that you'd never think of, etc.). Like Stripe Atlas for open source consulting?

slim 5y ago

I think daniel's reaction is appropriate and well thought. One can suppose thousands of these emails asking for free work have been sent. There is close to zero chance his demand for a support contract would get past the first filter. Whereas making a blog post about it makes for a good story and also has more chances to get the attention of the right people at this company. Even if it's slightly aggressive.