4 ms·
The source code is not what is deployed. There is typically no link back from a deployed system to its source, even if you compiled the binary within your orga
by jiggawatts 5y ago
The source code is not what is deployed.
There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.
If it came from an external organisation, things are exponentially harder.
Modern deployment systems are largely "one-way", with no way to trigger a full recompile from the deployment end of things. If you have a VM with "SomeRandomBinary.exe" running on it that has a vulnerability in it, you don't have many options.
IMHO, typical IT as done in the wild suffers from this write-once attitude, where the chain of provenance is too easily broken.
Companies like Google have a very non-typical setup that can be copied only by very small orgs like startups. Typical enterprises look nothing like a startup or a FAANG.
- staticassertion 5y agoThis sounds like an ops problem. It shouldn't be hard to tell what's running in your production environment and tie that back to a specific commit.
- jethro_tell 5y agoIt shouldn't be hard for a dev to update a dependency which should trigger an auto build and roll your pipeline. Shouldn't be hard for an upstream vendor to patch and have that pulled to kick off a new build. IDK where this idea that no one can know what's in a dependency tree is coming from but is not ops best practice for more than a decade.
- bigiain 5y ago> IDK where this idea that no one can know what's in a dependency tree is coming from I’d guess people running commercial non source available binaries in production. And outside startups and unicorns, that’s almost everybody. How do you find the dependancy tree for your on prem Oracle db, or your self hosted Atlassion stuff, or your non cloud ServiceNow or PeopleSoft stuff, or your Huawei network management stuff, or or or… And what can you do for any of your cloud hosted saas stuff, beyond telling legal “I dunno, their website hasn’t said whether they’re vulnerable to exploit de jour yet”?
- staticassertion 5y agoPresumably you would go ask Oracle, and then patch it when they provide a patch.
- lazide 5y agoHope you’ve got a pretty sweet support contract or you’re going to be waiting awhile. Depending on the product, maybe forever.
- staticassertion 5y agoMy point is that patching third party binaries is generally not something companies take responsibility for. They wait for vendors to patch the binaries and ship them.
- jiggawatts 5y ago> generally not something companies take responsibility for I have some bad news for you: viruses and hackers don't care about your support contracts and the delays they cause. Actually, I tell a lie: the hackers love them.
- staticassertion 5y agoThat's not relevant? The original premise here is "compiled languages will make things worse because they're harder to patch" except: a) It's very rare to patch things manually b) It's very rare to develop patches yourself (for 3rd party software) I'm sure some companies out there with many decades old deployments have to do stupid shit like that, but it's hardly the standard.
- lazide 5y agoHaving been on the receiving end of waiting a long time to get a patch for a actively being exploited zero day - the point is, even if you want to fix it yourself (which I desperately did), you can’t. If it’s some important piece of software you also can’t stop or the business stops - well, you and everyone else is just going to take it in the pants. Many enterprises are in exactly this bind. And support contracts only get you so far.
- rocqua 5y agoIt is! But a lot of ops departments have problems! OPs point is that a lot of deployments out there make it really hard to remediate this kind of vuln. Moreover he is arguing that java, as opposed to fat binaries with potentially stripped symbols, made things solvable. He fears that many companies would be much worse off with a big vulnerability in e.g. a rust crate.
- staticassertion 5y agoYes, some places suck. They're going to suck either way. If their system is so backwards that they need to manually patch services because everything is EOL, and compiled code will break their workflow, they have plenty of other shit to deal with. I don't think we should optimize for garbage companies with garbage practices.
- rocqua 5y agoI'd say places lie on a spectrum of sucking. With a lot of places being very important and far from perfect. Those places really should get good security. That means designing for security in the common situation rather than the situation of 'best practices'. I think we should optimize security for say the 80th percentile company as far as not following best practices (i.e. 80% of companies have better practices than what I suggest targeting).
- staticassertion 5y agoIn no way is manually writing a patch and then live patching the common situation. The vast majority of cases, and this has been the case for decades, will be that you take responsibility for your own dependencies, and you let your vendors take responsibility for theirs.
- yawaramin 5y agoGo 1.18 onwards will solve this problem: https://utcc.utoronto.ca/~cks/space/blog/programming/GoVersionOfYourSource https://utcc.utoronto.ca/~cks/space/blog/programming/GoVersi... > The go command now embeds version control information in binaries including the currently checked-out revision and a flag indicating whether edited or untracked files are present.... Additionally, the go command embeds information about the build including build and tool tags (set with -tags), compiler, assembler, and linker flags (like -gcflags), whether cgo was enabled, and if it was, the values of the cgo environment variables (like CGO_CFLAGS).
- nefitty 5y agoMy list of languages to practice were Scala and Rust. Maybe I'll do Scala and Go considering this single point.
- ncmncm 5y agoIn other words, you are not actually interested in programming.
- xxgreg 5y agoThere's also ongoing work to build a dependency vulnerability database. This is proposed to be surfaced via a built in `go audit` command. https://go.googlesource.com/proposal/+/master/design/draft-vulndb.md https://go.googlesource.com/proposal/+/master/design/draft-v...
- derefr 5y agoOn Windows, the Microsoft compilers and linkers have inserted such toolchain/library version metadata into PE executables for decades now (the "RICH" header — https://bytepointer.com/articles/the_microsoft_rich_header.htm https://bytepointer.com/articles/the_microsoft_rich_header.h...). At the time that Microsoft started doing it, paranoiacs thought it was some kind of anti-piracy software fingerprinting mechanism, embedding the compiling computer's MAC address or something. But really, it serves exactly the same purpose that Golang's effort does here: to let you map compiled-artifact back to inputs.
- 5y ago
- FridgeSeal 5y ago> The source code is not what is deployed. There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation. Do you not have this? Our docker images are tagged with the git hash they were built from, so at any point, for any of our envs, I can pull up the lock file of that build. Our deployment config also describes everything that is running the relevant env, and nobody has machine access- going through the pipeline is the only way to get access, so there’s nothing deployed that’s a mystery.
- rockwotj 5y agoYeah our org does the same thing and I'd say this is the gold standard (or at least mine) but sadly I'm assuming most companies don't do this, not everyone runs on docker nor has a build pipeline like this.
- yjftsjthsd-h 5y agoSure, but everyone should be able to have their built process embed that info into build artifacts, whether it's by passing GCC a DEFINE flag that ends up in --version output, passing nasm a data structure embedded at X bytes into the binary, a version.txt in the output zip, an autogenerated comment inlined in each HTML/JS/CSS file, whatever... somehow, you can embed build info.
- jiggawatts 5y agoThis happened last week, for real with ${dayjob} customer. Put yourself in their shoes for a second. Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive. Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1] So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent. Now what? No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago. A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server. There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use. Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on. [1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir. [2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)
- wbl 5y agoDo another release. You do have a way to track releases right?