15 ms·

Gitlab servers are being exploited in DDoS attacks

312 points by intunderflow 5y ago

im3w1l 5y ago
> GitLab Workhorse could check if the file is a valid TIFF of JPEG before passing it to ExifTool This approach doesn't work in general. An attacker could craft a polyglot file - and in that case it's a matter of which format is tried first. Valid tiff's could potentially be processed as something entirely different.
- setpatchaddress 5y ago
  Yup. PoC||GTFO article (one of many, IIRC) on crafting a polyglot PDF / JPEG file: https://github.com/angea/pocorgtfo/blob/master/contents/articles/03-03.pdf https://github.com/angea/pocorgtfo/blob/master/contents/arti...
  - deleted 5y ago
    [deleted]
buildbuildbuild 5y ago
This happened to all Gitlab instances that I manage around 2 days ago. Good to see publicity, I’m still dealing with not-so-understanding abuse departments at my hosting providers. Sure, my fault for not keeping it up to date. But there is much noise to filter through in the many tools we juggle these days, especially if an organization prefers to self-host.
- SV_BubbleTime 5y ago
  Article said GitLab patched back in April. Safe to say you didn’t deploy these patches? No judgment. I’m paid to make things, not apply patches. This is however why I don’t use self-hosted, pros and cons, etc.
  - charcircuit 5y ago
    Not him, but how are you supposed to know about the update? Do you need to check some page every day if there's an update? Why can't security updates just autoupdate like apps on phones or at least email the admin saying there is an important update.
    - Gigachad 5y ago
      Yes, you are meant to be paying attention to the security news for all tools you self host. That's part of self hosting. If you can't manage that (which is completely fair), then you shouldn't be self hosting.
      - justin_oaks 5y ago
        What's the best way to keep up on security news for a set of tools? I once had a CVE RSS feed, but it was mostly noise even after I filtered it to only tools/libraries we used.
    - nightfly 5y ago
      Depending on how you have it installed you could have your package management system automatically install the updates. They aren't always perfect though, I've had at least one Gitlab update that required manually running migrations commands since the ones in the update script failed for some reason. I wouldn't trust doing it automatically. And I'm not sure about gitlab, but their are often mailing lists for security updates for major software packages.
      - buildbuildbuild 5y ago
        Thanks for the productive response. I was using Watchtower in Docker Compose on all instances, but I had it misconfigured. My fault for assuming auto updates were working without verifying, lesson learned.
      - gnramires 5y ago
        I highly recommend debian's auto update feature (works on derivatives): https://www.linode.com/docs/guides/how-to-configure-automated-security-updates-debian/ https://www.linode.com/docs/guides/how-to-configure-automate... I've never had any problems with it, although I just run a couple of servers :). If everyone enabled this one thing I'm sure the Internet would be significantly safer. Obs: You can select security updates only which I believe are unlikely to break anything!
- sofixa 5y ago
  Honestly GitLab is so stable and foolproof to update you cn have it autoupdate on a schedule like 1-2 weeks after their monthly eelease.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- Someone1234 5y ago
  > But there is much noise to filter through in the many tools we juggle these days, especially if an organization prefers to self-host. If an organization is too overloaded to patch for six months, maybe they should re-evaluate if self-hosting is the best course of action. Seems like this is a foot-gun of your own creation.
  - Gigachad 5y ago
    There is very little reason to even self host Gitlab unless you are insanely paranoid or for philosophy reasons like debian/gnome.
    - pm90 5y ago
      Compliance often forces the hand.
      - deleted 5y ago
        [deleted]
    - sslalready 5y ago
      Reasons for self-hosting: GitLab.com's not-so-great availability [1], slow code searches (compared to self-hosted with GitLab Advanced Search) and the fact that you can keep your code and resources off the public Internet. The fact that you do need to upgrade it yourself regularly is indeed a drawback. On the other hand, an Omnibus upgrade has only failed me twice in the last five years or so, so there's little reason to not do automatic upgrades at night and fire off an alert in case something doesn't work as expected afterwards. Their releases are typically solid, so kudos to the team. [1] https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
    - mdoms 5y ago
      Sorry but this comment strikes me as incredibly ignorant. There are lots of good reasons to self host, chief among them reasons which can't be worked around - compliance and data sovereignty.
codegeek 5y ago
"..Bowling said he discovered a way to abuse how ExifTool handles uploads for DjVu file format used for scanned documents to gain control over the entire underlying GitLab web server" Ah, the good old "File upload vulnerability". File uploads remain one of the hardest problems to solve when it comes to security.
- tata71 5y ago
  Is that because people use hackjob dependencies to handle it more often than not?
  - djbusby 5y ago
    ExifTool is hackjob? I think not. But also, file uploads should be handled in a jail or box of some type - and never let their analysis make network calls.
    - jeffbee 5y ago
      It's a perl program that evaluates untrusted strings it finds in user files. What exactly is your standard for "is hackjob"? It appears to be a complete piece of shit.
      - rndgermandude 5y ago
        ExifTool is the tool to extract and write and fix metadata, bar none. I haven't seen anything that even comes remotely close at how good it is at handling just jpeg metadata (exif, XMP, IPTC, MarkerNotes, and all the fine bugs every vendor has when creating these), let alone other formats too. ExifTool is essentially for (image) metadata what ffmpeg is for video. It isn't a hack job, but it started out as one, like so very many other things. Version 1.00 was released end of 2003, while the problematic code was added in 2008. Mind you, the problematic code does not just eval whatever it sees, it tries to make sure the input isn't dangerous first. That check failed spectacularly, defeated by a newline combined with how '$' in perl regex works without any special flags[0]. Using eval was a bad choice to begin with (but I am told something you would commonly see in perl software of the time, yes, even in 2008 still), it was the lazy choice of re-using perl to unescape C-strings instead of rolling your own unescaping code. So what do you suggest? Use libexif[1]? Exiv2[2]? Where would I run it? Can you suggest any operating system that never had a "stupid" RCE? >It appears to be a complete piece of shit. Let's see your code then. All the code you ever wrote that is possibly still in use somewhere. So if you never ever fucked up or got lazy, feel free to cast the first stone, otherwise I would suggest you dial down your rhetoric when it comes to taking massive steaming piles on other people's work. Yes, Phil Harvey had a "WTF?!"-class security bug here[4], shit happens, "goto fail", let me deRail your yaml and the Debian random number of the day is: 6. He patched it promptly compared to other vendors and projects (public release on April 13th, while April 7th was the initial bug report, to Gitlab not ExifTool, which Gitlab then passed along[3]). You can blame him for the bug, you can blame Gitlab for not running exiftool in some sandbox. But that half the gitlab instances remain unpatched some 7 months after patches became available, that you'll have to put on the people running these instances. [0] https://github.com/exiftool/exiftool/commit/cf0f4e7dcd024ca99615bfd1102a841a25dde031#diff-fa0d652d10dbcd246e6b1df16c1e992931d3bb717a7e36157596b76bdadb3800 https://github.com/exiftool/exiftool/commit/cf0f4e7dcd024ca9... [1] http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=libexif http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=libexif [2] http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=exiv2 http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=exiv2 [3] https://hackerone.com/reports/1154542 https://hackerone.com/reports/1154542 [4] I cannot be sure if he wrote it, or if somebody else contributed it, but at the very least he didn't catch it during review. Looks like he wrote it, tho.
deleted 5y ago
[deleted]
Eduard 5y ago
As the exploit requires uploading a file, is it required for the attacker to first have a user account with file upload permissions?
- nielsole 5y ago
  Anyone who can open issues in a repo I think
  - mattbee 5y ago
    I am confused by this right now. I built a self-hosted gitlab install years ago for my own use, turned off sign-up, no public project listings - and still it was compromised. The HackerOne PoC URL throws a sign-in redirect for me, so I'm still trying to work it out.
    - cjbprime 5y ago
      It was originally incorrectly thought to require authentication, but actually doesn't.
    - ceochronos 5y ago
      Same thing happened with my self-hosted gitlab. Besides I got a new user (gitlab) with admin privileges and somehow installed a miner script running under git user (system).
- cjbprime 5y ago
  No. The original submission described the vulnerability as requiring an authenticated user, but it was later discovered (recently) that it works unauthenticated too, and that's what kicked off this mass exploitation. No user account is required.
magicalhippo 5y ago
From the issue report[1] comments: > Thanks in no small part to your recent findings, [GitLab] are rolling back our policy about paying half-bounties for third party findings. These have great impact on GitLab and we want to continue to incentivize research for high+ severity issues in that area. At least they're taking these things a bit more seriously now. [1]: https://hackerone.com/reports/1154542 https://hackerone.com/reports/1154542
- Someone1234 5y ago
  Anyone have background on what a "third party finding" is? Are they talking about dependencies? Because, to me, this specific bug is both a first AND third party issue (e.g. GitLab not pre-verifying before calling ExifTool).
  - magicalhippo 5y ago
    > Because, to me, this specific bug is both a first AND third party issue Seems GitLab agreed on that one, they registered two CVEs: ExifTool https://nvd.nist.gov/vuln/detail/CVE-2021-22204 https://nvd.nist.gov/vuln/detail/CVE-2021-22204 GitLab https://nvd.nist.gov/vuln/detail/CVE-2021-22205 https://nvd.nist.gov/vuln/detail/CVE-2021-22205
  - PragmaticPulp 5y ago
    > Anyone have background on what a "third party finding" is? Are they talking about dependencies? Yes. I've had some experience with bug bounty programs. The situation with vulnerabilities in dependencies is complicated. On one hand you want to incentivize researchers to find vulnerabilities in your dependencies, which likely don't have their own bug bounty programs. On the other hand, you run into situations where researchers discover a bug in a 3rd-party library and, instead of reporting it ASAP, they keep it secret as long as possible while they build up an arsenal of bug bounty exploits to report to multiple companies that use the library. This is a weird disincentive to fix the underlying bug quickly because the researchers know they only have so much time to exploit it in bug bounty programs before one of the companies fixes it upstream. We also had a problem where amateurs would spam our bug bounty program any time a CVE came out for one of our dependencies, even if they couldn't exploit it. It was relatively easy to close these bug reports because they couldn't provide a proof of concept exploit, but it still wasted a lot of time arguing with them, especially when they'd try to blackmail us on social media for not paying them out (for a bug that didn't exist in our product). I do not miss my days of dealing with bug bounty programs. Met a few great researchers, but most of the (attempted) participants were trying harder to exploit the bug bounty program than find exploits in our code.
henning 5y ago
Would putting your Gitlab instance behind a VPN mitigate this issue and similar? At least, it would limit attackers to malicious people with VPN access.
- staunch 5y ago
  Yes, it would. Most people should put their internal cloud infrastructure behind a VPN, if at all possible. It's easy to do and dramatically reduces the surface area for an attacker. It's not a silver bullet but in many cases it will be the difference between getting hacked and not getting hacked. It is totally possible to design applications that can be safely exposed to the public internet but it requires some real effort.
- jodoherty 5y ago
  It would dramatically limit your attack surface to those who could gain access to your VPN. I prefer requiring TLS mutual authentication with a corporate PKI and issuing employees client certificates. Doing both wouldn't be a bad idea either.
  - relaunched 5y ago
    The number of software products, SaaS and on-prem, that don't support mutual tls is a disgrace.
    - tfigment 5y ago
      I frequently do mTLS with a reverse proxy (httpd, nginx, caddy, ...). Not perfect but you can tighten the connection security a lot without touching the other service. But by outsourcing it you lose some control.
      - PLG88 5y ago
        We put all our DevOps tools behind Open Ziti (ziti.dev) which ensures we do not need any public IPs (unlike a VPN or bastion) while giving granular access control for only trusted users.
DelightOne 5y ago
How much effort is it for a company like Gitlab to add kill switches to certain features and to trigger them incase they are exploited in the wild? Has to be pretty fine-grained though to disable analysis or upload on certain files, right? What are alternatives to automatic immediate updates and kill switches besides not exposing the service to the internet?
- bink 5y ago
  Kill switches sound great in theory until you disable a service used to host a billion dollar company, a hospital admission network, or your biggest paying customer. The solution here is to offer a security notification service, which GitLab does. It's up to the admins to maintain these systems. It's GitLab's job to give them the information they need to do so, which they have.
mattbee 5y ago
Yikes, it would have been nice to have a heads-up email from my Gitlab install that a new user had been created on September 3rd :-O I do occasionally get an email to say my account had been locked for security, but had assumed that was just noise from random login attempts, and the 2FA didn't make me look any further.
surfer7837 5y ago
How can you protect yourself from file upload threats? It's basically the worst possible threat model -- executing complex user input that conforms to a spec that was written 20 years ago by some proprietary company with no security. Executing everything on an isolated container with no permissions? Audit trial etc/good logging? If someone comes up with an RCE you're basically done for, you can only mitigate it but not completely stop it.
- JoshTriplett 5y ago
  If you have to process it at all, do it in a WebAssembly sandbox on the server. Or, alternatively, in a seccomp-secured sandbox that isn't allowed to make any system calls whatsoever, just read data from one file descriptor and write processed data to another.
  - wheresmycraisin 5y ago
    .... why webassembly?
    - JeremyNT 5y ago
      Yeah, I don't see the value here either. You don't need wasm or chrome or any of that stuff. Linux itself has several features that can be used to isolate processes, and there are use friendly tools like bwrap [0] that make configuration easy. It should be entirely possible to sandbox something like ExifTool itself such that it has no network access and is limited to reading and writing files in a particular directory. https://wiki.archlinux.org/title/Bubblewrap https://wiki.archlinux.org/title/Bubblewrap
    - JoshTriplett 5y ago
      Several reasons: - It's a separate interface with a different attack surface than your system, so compared to a locked-down version of the normal syscall API, it provides better defense-in-depth. - It's designed to be a fully self-contained sandbox, by default. If you're locking down everything but reading and writing previously opened file descriptors, you can build a secure sandbox atop syscalls fairly easily. If you need more nuance than that, WebAssembly seems more likely to remain secure, while syscall sandboxes seem more likely to fail-insecure if you get a detail wrong. - It seems easier to sandbox otherwise-unmodified code that way. If you have code that needs some access to system resources, I think WebAssembly makes it easier to give it just what it needs and nothing else. (Also, note that I'm not talking about running in a browser; I'm talking about standalone WebAssembly runtimes like wasmtime.)
1vuio0pswjnm7 5y ago
Seems like one solution would be to require users uploading images to remove metadata themselves before uploading. The website could reject images that contain matadata. Was Gitlab was extracting the metadata and using it for some purpose. If not, what is the reason to accept images with metadata. Perhaps they assume their customers prefer less "security" and more "convenience", instead of vice versa (less "convenience", more "security").
- Scaevolus 5y ago
  Theoretically the pure-perl implementation will be immune to most categories of exploits that sloppy binary parsing runs into-- buffer overflows, use after frees, and so on. That better security goes out the window once you start using eval(), of course.
  - 1vuio0pswjnm7 5y ago
    Both exiv2 and jhead, as examples, have had numerous such bugs. However, there is a difference IMO between running these programs on one's own images (created by oneself), e.g., before uploading them to the internet, versus running them on untrusted images from others.
- Gigachad 5y ago
  How do you reject an image that has metadata without doing basically exactly the same process as removing the data?
  - 1vuio0pswjnm7 5y ago
    exiftool does not remove djvu metadata.
BuffaloBagel 5y ago
My VOIP vendor, voip.ms, has been under attack for weeks. Wonder if this is the source?
- menscher 5y ago
  Totally different. The attacks on VOIP vendors mostly used UDP amplification, which relies on having a server that can fake its source IP due to an incompetent (or complicit!) network provider, while this is a botnet (that is only about a week old).
  - donkarma 5y ago
    You say incompetent when most haven't implemented BCP38 iirc
Jach 5y ago
It's somewhat refreshing that the underlying bug isn't from some C or C++ utility, but instead a Perl program using eval: https://github.com/exiftool/exiftool/blob/11.70/lib/Image/ExifTool/DjVu.pm#L233 https://github.com/exiftool/exiftool/blob/11.70/lib/Image/Ex... Another instance of "avoid eval as much as possible" for languages that have it.
- xxpor 5y ago
  With dlopen() and libclang/libgccjit, you could argue C (on an OS that supports dynamic loading) has eval too ;)
  - cozzyd 5y ago
    You have system("gcc uploaded.c && ./a.out") too!
    - Spivak 5y ago
      But that code doesn’t run in the memory space of the process.
      - willbudd 5y ago
        Then compile it as a library, and dlopen() it after all. (I'm just guessing the gp's point was that JIT isn't really an essential modality.)
  - superjan 5y ago
    Only in programs that invoke it on hacker controlled data.
    - xxpor 5y ago
      But if you only evoke eval() on non-hacker controlled data in a scripting language, it's probably fine too.
- ed25519FUUU 5y ago
sytse 5y ago
TLDR; A vulnerability in GitLab that was patched on April 14, 2021 is now being exploited to hack into self-managed servers that are accessible through the internet. For more context see https://about.gitlab.com/blog/2021/11/04/action-needed-in-response-to-cve2021-22205/ https://about.gitlab.com/blog/2021/11/04/action-needed-in-re... If you are using GitLab.com you are not affected.
colek42 5y ago
At this point the threat model for you CI system should be to assume it it comprimised if you havent patched.
KronisLV 5y ago
I feel like this article could have been far more useful with the following points being explicitly mentioned, or at least summarized: - the problem appeared in GitLab 11.9.0 - the problem seems to have been fixed in GitLab 13.8.8 - the vulnerability uses ExifTool, so to exploit it, a user needs to be able to upload images - if an update is not (yet) possible, DjVu format file uploads can be blocked to avert this vulnerability - this vulnerability isn't relevant for GitLab instances that just have 1 user, or are not publicly accessible on the Internet Now, i'm not saying that the above is entirely true, but after reading something like the above, one should be able to figure out how to best act: - if you have a public GitLab instance with open registrations, consider updating it immediately (with backups in place, of course) - if you have a private GitLab instance with many users in your own corporate network (that somehow isn't updated yet) - this is a good reason to put updating it into your agenda today, even if your users aren't necessarily hostile - if you have a private GitLab instance or one with registrations closed (e.g. you're the only user or people that you trust use it), mark this down and update whenever possible, however it's probably not necessary right this moment Of course, i can't say the above with 100% confidence, because the article itself lacks this actionable information to aid in decisionmaking and so i'm left to piece things together on my own, because of which i could be wrong. On an unrelated note, DjVu is a pretty interesting file format, though sadly i've only seen it be used very sparsely, on some Russian forums for tractor manuals or something: https://en.wikipedia.org/wiki/DjVu https://en.wikipedia.org/wiki/DjVu
- morelisp 5y ago
  > the vulnerability uses ExifTool, so to exploit it, a user needs to be able to upload images... this vulnerability isn't relevant for GitLab instances that just have 1 user, or are not publicly accessible on the Internet A lot of private GitLabs contain mirrors of public repositories or vendored copies of public libraries. Our GitLab is private but practically speaking there's probably several hundred people, most of whom we couldn't identify, that could "upload an image" to it.
- mjochim 5y ago
  Caution, CVS-2021-22205 has later been found to be exploitable without authentication. No need to be "able to upload an image," unfortunately. Also no need to take the detour through a mirrored repo as sibling suggests; so long as the GitLab instance is accessible from the internet.
dorianmariefr 5y ago
Maybe time to think about the hosted GitLab version, GitHub enterprise or just GitHub like everyone else
kkjjkgjjgg 5y ago
Are there good reasons for people to not turn on automatic updates for security issues, I wonder?
- rightbyte 5y ago
  You need to hunt breaking updates all the time as a sysadmin with autoupdates. It it much easier to pinpoint any problem if you are aware the update happes and choose the time to do so.
BraverHeart 5y ago
Does anyone know how they actually find all these instances to exploit? For instance say your instance is attached to a subdomain, how would they find it? or say your instance is running in a port other than 80, how would they find it as well?

312 points by intunderflow 5y ago

> GitLab Workhorse could check if the file is a valid TIFF of JPEG before passing it to ExifTool This approach doesn't work in general. An attacker could craft a polyglot file - and in that case it's a matter of which format is tried first. Valid tiff's could potentially be processed as something entirely different.

setpatchaddress 5y ago
Yup. PoC||GTFO article (one of many, IIRC) on crafting a polyglot PDF / JPEG file: https://github.com/angea/pocorgtfo/blob/master/contents/articles/03-03.pdf https://github.com/angea/pocorgtfo/blob/master/contents/arti...
- deleted 5y ago
  [deleted]

buildbuildbuild 5y ago

This happened to all Gitlab instances that I manage around 2 days ago. Good to see publicity, I’m still dealing with not-so-understanding abuse departments at my hosting providers. Sure, my fault for not keeping it up to date. But there is much noise to filter through in the many tools we juggle these days, especially if an organization prefers to self-host.

SV_BubbleTime 5y ago
Article said GitLab patched back in April. Safe to say you didn’t deploy these patches? No judgment. I’m paid to make things, not apply patches. This is however why I don’t use self-hosted, pros and cons, etc.
- charcircuit 5y ago
  Not him, but how are you supposed to know about the update? Do you need to check some page every day if there's an update? Why can't security updates just autoupdate like apps on phones or at least email the admin saying there is an important update.
  - Gigachad 5y ago
    Yes, you are meant to be paying attention to the security news for all tools you self host. That's part of self hosting. If you can't manage that (which is completely fair), then you shouldn't be self hosting.
    - justin_oaks 5y ago
      What's the best way to keep up on security news for a set of tools? I once had a CVE RSS feed, but it was mostly noise even after I filtered it to only tools/libraries we used.
  - nightfly 5y ago
    Depending on how you have it installed you could have your package management system automatically install the updates. They aren't always perfect though, I've had at least one Gitlab update that required manually running migrations commands since the ones in the update script failed for some reason. I wouldn't trust doing it automatically. And I'm not sure about gitlab, but their are often mailing lists for security updates for major software packages.
    - buildbuildbuild 5y ago
      Thanks for the productive response. I was using Watchtower in Docker Compose on all instances, but I had it misconfigured. My fault for assuming auto updates were working without verifying, lesson learned.
    - gnramires 5y ago
      I highly recommend debian's auto update feature (works on derivatives): https://www.linode.com/docs/guides/how-to-configure-automated-security-updates-debian/ https://www.linode.com/docs/guides/how-to-configure-automate... I've never had any problems with it, although I just run a couple of servers :). If everyone enabled this one thing I'm sure the Internet would be significantly safer. Obs: You can select security updates only which I believe are unlikely to break anything!
sofixa 5y ago
Honestly GitLab is so stable and foolproof to update you cn have it autoupdate on a schedule like 1-2 weeks after their monthly eelease.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
Someone1234 5y ago
> But there is much noise to filter through in the many tools we juggle these days, especially if an organization prefers to self-host. If an organization is too overloaded to patch for six months, maybe they should re-evaluate if self-hosting is the best course of action. Seems like this is a foot-gun of your own creation.
- Gigachad 5y ago
  There is very little reason to even self host Gitlab unless you are insanely paranoid or for philosophy reasons like debian/gnome.
  - pm90 5y ago
    Compliance often forces the hand.
    - deleted 5y ago
      [deleted]
  - sslalready 5y ago
    Reasons for self-hosting: GitLab.com's not-so-great availability [1], slow code searches (compared to self-hosted with GitLab Advanced Search) and the fact that you can keep your code and resources off the public Internet. The fact that you do need to upgrade it yourself regularly is indeed a drawback. On the other hand, an Omnibus upgrade has only failed me twice in the last five years or so, so there's little reason to not do automatic upgrades at night and fire off an alert in case something doesn't work as expected afterwards. Their releases are typically solid, so kudos to the team. [1] https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
  - mdoms 5y ago
    Sorry but this comment strikes me as incredibly ignorant. There are lots of good reasons to self host, chief among them reasons which can't be worked around - compliance and data sovereignty.

codegeek 5y ago

"..Bowling said he discovered a way to abuse how ExifTool handles uploads for DjVu file format used for scanned documents to gain control over the entire underlying GitLab web server" Ah, the good old "File upload vulnerability". File uploads remain one of the hardest problems to solve when it comes to security.

tata71 5y ago
Is that because people use hackjob dependencies to handle it more often than not?
- djbusby 5y ago
  ExifTool is hackjob? I think not. But also, file uploads should be handled in a jail or box of some type - and never let their analysis make network calls.
  - jeffbee 5y ago
    It's a perl program that evaluates untrusted strings it finds in user files. What exactly is your standard for "is hackjob"? It appears to be a complete piece of shit.
    - rndgermandude 5y ago
      ExifTool is the tool to extract and write and fix metadata, bar none. I haven't seen anything that even comes remotely close at how good it is at handling just jpeg metadata (exif, XMP, IPTC, MarkerNotes, and all the fine bugs every vendor has when creating these), let alone other formats too. ExifTool is essentially for (image) metadata what ffmpeg is for video. It isn't a hack job, but it started out as one, like so very many other things. Version 1.00 was released end of 2003, while the problematic code was added in 2008. Mind you, the problematic code does not just eval whatever it sees, it tries to make sure the input isn't dangerous first. That check failed spectacularly, defeated by a newline combined with how '$' in perl regex works without any special flags[0]. Using eval was a bad choice to begin with (but I am told something you would commonly see in perl software of the time, yes, even in 2008 still), it was the lazy choice of re-using perl to unescape C-strings instead of rolling your own unescaping code. So what do you suggest? Use libexif[1]? Exiv2[2]? Where would I run it? Can you suggest any operating system that never had a "stupid" RCE? >It appears to be a complete piece of shit. Let's see your code then. All the code you ever wrote that is possibly still in use somewhere. So if you never ever fucked up or got lazy, feel free to cast the first stone, otherwise I would suggest you dial down your rhetoric when it comes to taking massive steaming piles on other people's work. Yes, Phil Harvey had a "WTF?!"-class security bug here[4], shit happens, "goto fail", let me deRail your yaml and the Debian random number of the day is: 6. He patched it promptly compared to other vendors and projects (public release on April 13th, while April 7th was the initial bug report, to Gitlab not ExifTool, which Gitlab then passed along[3]). You can blame him for the bug, you can blame Gitlab for not running exiftool in some sandbox. But that half the gitlab instances remain unpatched some 7 months after patches became available, that you'll have to put on the people running these instances. [0] https://github.com/exiftool/exiftool/commit/cf0f4e7dcd024ca99615bfd1102a841a25dde031#diff-fa0d652d10dbcd246e6b1df16c1e992931d3bb717a7e36157596b76bdadb3800 https://github.com/exiftool/exiftool/commit/cf0f4e7dcd024ca9... [1] http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=libexif http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=libexif [2] http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=exiv2 http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=exiv2 [3] https://hackerone.com/reports/1154542 https://hackerone.com/reports/1154542 [4] I cannot be sure if he wrote it, or if somebody else contributed it, but at the very least he didn't catch it during review. Looks like he wrote it, tho.

deleted 5y ago

[deleted]

Eduard 5y ago

As the exploit requires uploading a file, is it required for the attacker to first have a user account with file upload permissions?

nielsole 5y ago
Anyone who can open issues in a repo I think
- mattbee 5y ago
  I am confused by this right now. I built a self-hosted gitlab install years ago for my own use, turned off sign-up, no public project listings - and still it was compromised. The HackerOne PoC URL throws a sign-in redirect for me, so I'm still trying to work it out.
  - cjbprime 5y ago
    It was originally incorrectly thought to require authentication, but actually doesn't.
  - ceochronos 5y ago
    Same thing happened with my self-hosted gitlab. Besides I got a new user (gitlab) with admin privileges and somehow installed a miner script running under git user (system).
cjbprime 5y ago
No. The original submission described the vulnerability as requiring an authenticated user, but it was later discovered (recently) that it works unauthenticated too, and that's what kicked off this mass exploitation. No user account is required.

magicalhippo 5y ago

From the issue report[1] comments: > Thanks in no small part to your recent findings, [GitLab] are rolling back our policy about paying half-bounties for third party findings. These have great impact on GitLab and we want to continue to incentivize research for high+ severity issues in that area. At least they're taking these things a bit more seriously now. [1]: https://hackerone.com/reports/1154542 https://hackerone.com/reports/1154542

Someone1234 5y ago
Anyone have background on what a "third party finding" is? Are they talking about dependencies? Because, to me, this specific bug is both a first AND third party issue (e.g. GitLab not pre-verifying before calling ExifTool).
- magicalhippo 5y ago
  > Because, to me, this specific bug is both a first AND third party issue Seems GitLab agreed on that one, they registered two CVEs: ExifTool https://nvd.nist.gov/vuln/detail/CVE-2021-22204 https://nvd.nist.gov/vuln/detail/CVE-2021-22204 GitLab https://nvd.nist.gov/vuln/detail/CVE-2021-22205 https://nvd.nist.gov/vuln/detail/CVE-2021-22205
- PragmaticPulp 5y ago
  > Anyone have background on what a "third party finding" is? Are they talking about dependencies? Yes. I've had some experience with bug bounty programs. The situation with vulnerabilities in dependencies is complicated. On one hand you want to incentivize researchers to find vulnerabilities in your dependencies, which likely don't have their own bug bounty programs. On the other hand, you run into situations where researchers discover a bug in a 3rd-party library and, instead of reporting it ASAP, they keep it secret as long as possible while they build up an arsenal of bug bounty exploits to report to multiple companies that use the library. This is a weird disincentive to fix the underlying bug quickly because the researchers know they only have so much time to exploit it in bug bounty programs before one of the companies fixes it upstream. We also had a problem where amateurs would spam our bug bounty program any time a CVE came out for one of our dependencies, even if they couldn't exploit it. It was relatively easy to close these bug reports because they couldn't provide a proof of concept exploit, but it still wasted a lot of time arguing with them, especially when they'd try to blackmail us on social media for not paying them out (for a bug that didn't exist in our product). I do not miss my days of dealing with bug bounty programs. Met a few great researchers, but most of the (attempted) participants were trying harder to exploit the bug bounty program than find exploits in our code.

henning 5y ago

Would putting your Gitlab instance behind a VPN mitigate this issue and similar? At least, it would limit attackers to malicious people with VPN access.

staunch 5y ago
Yes, it would. Most people should put their internal cloud infrastructure behind a VPN, if at all possible. It's easy to do and dramatically reduces the surface area for an attacker. It's not a silver bullet but in many cases it will be the difference between getting hacked and not getting hacked. It is totally possible to design applications that can be safely exposed to the public internet but it requires some real effort.
jodoherty 5y ago
It would dramatically limit your attack surface to those who could gain access to your VPN. I prefer requiring TLS mutual authentication with a corporate PKI and issuing employees client certificates. Doing both wouldn't be a bad idea either.
- relaunched 5y ago
  The number of software products, SaaS and on-prem, that don't support mutual tls is a disgrace.
  - tfigment 5y ago
    I frequently do mTLS with a reverse proxy (httpd, nginx, caddy, ...). Not perfect but you can tighten the connection security a lot without touching the other service. But by outsourcing it you lose some control.
    - PLG88 5y ago
      We put all our DevOps tools behind Open Ziti (ziti.dev) which ensures we do not need any public IPs (unlike a VPN or bastion) while giving granular access control for only trusted users.

DelightOne 5y ago

How much effort is it for a company like Gitlab to add kill switches to certain features and to trigger them incase they are exploited in the wild? Has to be pretty fine-grained though to disable analysis or upload on certain files, right? What are alternatives to automatic immediate updates and kill switches besides not exposing the service to the internet?

bink 5y ago
Kill switches sound great in theory until you disable a service used to host a billion dollar company, a hospital admission network, or your biggest paying customer. The solution here is to offer a security notification service, which GitLab does. It's up to the admins to maintain these systems. It's GitLab's job to give them the information they need to do so, which they have.

mattbee 5y ago

Yikes, it would have been nice to have a heads-up email from my Gitlab install that a new user had been created on September 3rd :-O I do occasionally get an email to say my account had been locked for security, but had assumed that was just noise from random login attempts, and the 2FA didn't make me look any further.

surfer7837 5y ago

How can you protect yourself from file upload threats? It's basically the worst possible threat model -- executing complex user input that conforms to a spec that was written 20 years ago by some proprietary company with no security. Executing everything on an isolated container with no permissions? Audit trial etc/good logging? If someone comes up with an RCE you're basically done for, you can only mitigate it but not completely stop it.

JoshTriplett 5y ago
If you have to process it at all, do it in a WebAssembly sandbox on the server. Or, alternatively, in a seccomp-secured sandbox that isn't allowed to make any system calls whatsoever, just read data from one file descriptor and write processed data to another.
- wheresmycraisin 5y ago
  .... why webassembly?
  - JeremyNT 5y ago
    Yeah, I don't see the value here either. You don't need wasm or chrome or any of that stuff. Linux itself has several features that can be used to isolate processes, and there are use friendly tools like bwrap [0] that make configuration easy. It should be entirely possible to sandbox something like ExifTool itself such that it has no network access and is limited to reading and writing files in a particular directory. https://wiki.archlinux.org/title/Bubblewrap https://wiki.archlinux.org/title/Bubblewrap
  - JoshTriplett 5y ago
    Several reasons: - It's a separate interface with a different attack surface than your system, so compared to a locked-down version of the normal syscall API, it provides better defense-in-depth. - It's designed to be a fully self-contained sandbox, by default. If you're locking down everything but reading and writing previously opened file descriptors, you can build a secure sandbox atop syscalls fairly easily. If you need more nuance than that, WebAssembly seems more likely to remain secure, while syscall sandboxes seem more likely to fail-insecure if you get a detail wrong. - It seems easier to sandbox otherwise-unmodified code that way. If you have code that needs some access to system resources, I think WebAssembly makes it easier to give it just what it needs and nothing else. (Also, note that I'm not talking about running in a browser; I'm talking about standalone WebAssembly runtimes like wasmtime.)

1vuio0pswjnm7 5y ago

Seems like one solution would be to require users uploading images to remove metadata themselves before uploading. The website could reject images that contain matadata. Was Gitlab was extracting the metadata and using it for some purpose. If not, what is the reason to accept images with metadata. Perhaps they assume their customers prefer less "security" and more "convenience", instead of vice versa (less "convenience", more "security").

Scaevolus 5y ago
Theoretically the pure-perl implementation will be immune to most categories of exploits that sloppy binary parsing runs into-- buffer overflows, use after frees, and so on. That better security goes out the window once you start using eval(), of course.
- 1vuio0pswjnm7 5y ago
  Both exiv2 and jhead, as examples, have had numerous such bugs. However, there is a difference IMO between running these programs on one's own images (created by oneself), e.g., before uploading them to the internet, versus running them on untrusted images from others.
Gigachad 5y ago
How do you reject an image that has metadata without doing basically exactly the same process as removing the data?
- 1vuio0pswjnm7 5y ago
  exiftool does not remove djvu metadata.

BuffaloBagel 5y ago

My VOIP vendor, voip.ms, has been under attack for weeks. Wonder if this is the source?

menscher 5y ago
Totally different. The attacks on VOIP vendors mostly used UDP amplification, which relies on having a server that can fake its source IP due to an incompetent (or complicit!) network provider, while this is a botnet (that is only about a week old).
- donkarma 5y ago
  You say incompetent when most haven't implemented BCP38 iirc

Jach 5y ago

It's somewhat refreshing that the underlying bug isn't from some C or C++ utility, but instead a Perl program using eval: https://github.com/exiftool/exiftool/blob/11.70/lib/Image/ExifTool/DjVu.pm#L233 https://github.com/exiftool/exiftool/blob/11.70/lib/Image/Ex... Another instance of "avoid eval as much as possible" for languages that have it.

xxpor 5y ago
With dlopen() and libclang/libgccjit, you could argue C (on an OS that supports dynamic loading) has eval too ;)
- cozzyd 5y ago
  You have system("gcc uploaded.c && ./a.out") too!
  - Spivak 5y ago
    But that code doesn’t run in the memory space of the process.
    - willbudd 5y ago
      Then compile it as a library, and dlopen() it after all. (I'm just guessing the gp's point was that JIT isn't really an essential modality.)
- superjan 5y ago
  Only in programs that invoke it on hacker controlled data.
  - xxpor 5y ago
    But if you only evoke eval() on non-hacker controlled data in a scripting language, it's probably fine too.
ed25519FUUU 5y ago

sytse 5y ago

TLDR; A vulnerability in GitLab that was patched on April 14, 2021 is now being exploited to hack into self-managed servers that are accessible through the internet. For more context see https://about.gitlab.com/blog/2021/11/04/action-needed-in-response-to-cve2021-22205/ https://about.gitlab.com/blog/2021/11/04/action-needed-in-re... If you are using GitLab.com you are not affected.

colek42 5y ago

At this point the threat model for you CI system should be to assume it it comprimised if you havent patched.

KronisLV 5y ago

I feel like this article could have been far more useful with the following points being explicitly mentioned, or at least summarized: - the problem appeared in GitLab 11.9.0 - the problem seems to have been fixed in GitLab 13.8.8 - the vulnerability uses ExifTool, so to exploit it, a user needs to be able to upload images - if an update is not (yet) possible, DjVu format file uploads can be blocked to avert this vulnerability - this vulnerability isn't relevant for GitLab instances that just have 1 user, or are not publicly accessible on the Internet Now, i'm not saying that the above is entirely true, but after reading something like the above, one should be able to figure out how to best act: - if you have a public GitLab instance with open registrations, consider updating it immediately (with backups in place, of course) - if you have a private GitLab instance with many users in your own corporate network (that somehow isn't updated yet) - this is a good reason to put updating it into your agenda today, even if your users aren't necessarily hostile - if you have a private GitLab instance or one with registrations closed (e.g. you're the only user or people that you trust use it), mark this down and update whenever possible, however it's probably not necessary right this moment Of course, i can't say the above with 100% confidence, because the article itself lacks this actionable information to aid in decisionmaking and so i'm left to piece things together on my own, because of which i could be wrong. On an unrelated note, DjVu is a pretty interesting file format, though sadly i've only seen it be used very sparsely, on some Russian forums for tractor manuals or something: https://en.wikipedia.org/wiki/DjVu https://en.wikipedia.org/wiki/DjVu

morelisp 5y ago
> the vulnerability uses ExifTool, so to exploit it, a user needs to be able to upload images... this vulnerability isn't relevant for GitLab instances that just have 1 user, or are not publicly accessible on the Internet A lot of private GitLabs contain mirrors of public repositories or vendored copies of public libraries. Our GitLab is private but practically speaking there's probably several hundred people, most of whom we couldn't identify, that could "upload an image" to it.
mjochim 5y ago
Caution, CVS-2021-22205 has later been found to be exploitable without authentication. No need to be "able to upload an image," unfortunately. Also no need to take the detour through a mirrored repo as sibling suggests; so long as the GitLab instance is accessible from the internet.

dorianmariefr 5y ago

Maybe time to think about the hosted GitLab version, GitHub enterprise or just GitHub like everyone else

kkjjkgjjgg 5y ago

Are there good reasons for people to not turn on automatic updates for security issues, I wonder?

rightbyte 5y ago
You need to hunt breaking updates all the time as a sysadmin with autoupdates. It it much easier to pinpoint any problem if you are aware the update happes and choose the time to do so.

BraverHeart 5y ago

Does anyone know how they actually find all these instances to exploit? For instance say your instance is attached to a subdomain, how would they find it? or say your instance is running in a port other than 80, how would they find it as well?