16 ms·

Kaseya Hack: Incident Overview and Technical Details

37 points by gostsamo 5y ago

ijidak 5y ago
So is this vulnerability basically that they didn't sign and check hashes of their executables and downloads?
- gostsamo 5y ago
  > The attackers were able to exploit zero-day vulnerabilities in the VSA product to bypass authentication and run arbitrary command execution. This is all that we have so far.
  - cartoonworld 5y ago
    This sounds like a 10/10 chained out of a rookie PKI mistake in their (editorializing here) poorly audited critical software infrastructure. Surely nobody steering the ship has been bitten by shitty PKI practices before, right? These people have no care for the work of others they have trashed. The acquisition culture is so reprehensibly transparent and incompetent, it would blow my mind if it weren't just another day at the office. This company doesn't even has a CISO from what I can see. They must rely on their CTO's outstanding track record of industry-leading assurance technologies in a rapidly evolving landscape (of cash) Refer to https://www.kaseya.com/company/ https://www.kaseya.com/company/ A) Their CTO presided over DigiCert, who botched 23,000 orthogonally integrated, outside-of-the-box, innovative security certificates[0] through their bad practices and garbage organization while meeting committed timeframes. Dan Timpson leads all product development, security and cloud/SaaS operations teams across all lines of business for all Kaseya companies including Unitrends, Spanning Cloud Apps, Rapidfire Tools, ID Agent, Graphus, RocketCyber and IT Glue. He joins Kaseya with a proven track record of success having led and grown engineering and R&D teams to build architectures focused on security and integrity, data compliance and testing. Through his leadership, Timpson empowers engineering teams to develop solutions that emphasize quality, simplicity and reliability while meeting committed timeframes. Prior to joining Kaseya, Mr. Timpson served as CTO of DigiCert, a provider of high-assurance digital certificates that deliver trusted SSL, private and managed PKI deployments and device certificates for the emerging IoT market. There, Timpson was responsible for Digicert’s technology strategy and played a vital role in leading the security industry by driving innovation. Prior to joining DigiCert, Mr. Timpson worked for Microsoft Corp leading virtualization technology engineering teams and driving the division’s software security release discipline. Earlier in his career, he managed engineering teams at Novell over the span of nine years building identity and access management systems and Novell’s underlying PKI framework. Mr. Timpson holds a Masters of Business Administration and Technology Management from Westminster College, and holds the patents to five technology innovations. B) Their current CEO was onboarded after presiding over a number of pricey acquisitions in whatever tech fads, driving "unprecedented growth" with his undoubtedly forward-looking vision and commitment to shareholder value. Fred Voccola leads the vision, strategy and growth of Kaseya and its family of brands including Unitrends, Spanning Cloud Apps, Rapidfire Tools, IT Glue, Graphus, RocketCyber and ID Agent. Prior to joining Kaseya, Mr. Voccola served as president and general manager of Yodle’s Brand Networks Division, providing digital and market automation solutions to small and medium businesses. In 2016, Yodle successfully was sold to Web.com (NASDAQ: Web). Prior to that, Mr. Voccola was president of Nolio, Inc., a devops SaaS company, where he drove 100% year-over-year growth and successfully sold the business to CA Technologies. Mr. Voccola also co-founded and served as president and CEO of Trust Technology Corp., where he drove significant growth over three years prior to the company’s sale to FGI Global. Earlier in his career, Mr. Voccola co-founded and served as COO of Identify Software, substantially growing the business during his five-year tenure, culminating in a sale to BMC Software. He then served as vice president of worldwide sales and services at BMC Software. In addition, Mr. Voccola has held various management and executive roles at Intira (acquired by Divine Systems) and Prism Solutions (acquired by Ardent Software/Informix/IBM). Mr. Voccola holds a Bachelor of Science in Finance from the Carroll School of Management at Boston College. This stuff pisses me off to no end, especially after all the years of being called a crackpot and how nobody would do that or how could they know or whatever. I'm not a genius, this is crayon on the wall. At least they've "created" a lot of jobs lately. Probably toasted a few, too. They'll be off to the next clown car soon enough, probably driving unprecedented growth in next-generation managed service applications for dialysis machines and pesticide application PLCs. In the cloud. [0] - https://www.bleepingcomputer.com/news/security/23-000-users-lose-ssl-certificates-in-trustico-digicert-spat/ https://www.bleepingcomputer.com/news/security/23-000-users-...
    - toast0 5y ago
      > Their CTO presided over DigiCert, who botched 23,000 orthogonally integrated, outside-of-the-box, innovative security certificates[0] through their bad practices and garbage organization while meeting committed timeframes. I had forgotten about this incident. Somehow I thought this happened before 2018. Anyway, I'm not sure how any of this is DigiCert's fault. Maybe they should have had mandatory audits of resellers to make sure they didn't do stupid things like have the private keys for customer certificates... But given that the reseller sent the private keys to the CA, that's pretty good evidence the keys are compromised, and worthy of revocation.
      - cartoonworld 5y ago
        Indeed, they had one job...
- usr1106 5y ago
  It's not said that they (the vendors code) downloaded that code. If they have a remote code execution elsewhere it can download more. Does anybody check the signature of all DLLs they are loading (into memory, not downloading) and all executables they are a calling? I have not seen such system in real life. The Linux kernel can check the modules it loads, but that's a different level. And I am not sure how widely it is used in practice. Anyway if you can inject a bad kernel module you are already root and have many more options.
  - fmajid 5y ago
    macOS does with notarization.
    - usr1106 5y ago
      Good point, I am not a MacOS user. Users can authorize unnotarized executables to be run according to a quick Web search. So what would prevent malware from authorizing its unnotarized binaries? In the Kaseya case it seemed to have elevated privileges already, because it was able to install bad binaries.
binkHN 5y ago
Sadly, not a whole lot of detail here.
dt3ft 5y ago
Can anyone share the actual binaries used? (Specifically, the dll cryptor which was sideloaded).
bigmattystyles 5y ago
Does Kaseya survive this?
- gostsamo 5y ago
  Yes, I think. Depends on their management of the crisis. Hacks are part of life nowadays and it takes several examples of negligence before disengaging your users.
- usr1106 5y ago
  In my thinking they should not. Closed source is just such an inferior development model. Not that bad practices and stupid mistake would not happen in open source. It has happened and will certainly happen again. But more eyes are always better. And people who value deep technical understanding over just making money are often found in an open source ecosystem. It is incredibly frustrating to have to work with black boxes like Windows where you have inferior choices to explore and fix things. Obviously there are many decision makers who don't see this. So shops like Kaseya do unfortunately survive.
  - funnybeam 5y ago
    Is there an open source Remote Monitoring and Management solution though? My company used to use Kaseya but luckily changed to a competitor a few years ago so dodged this particular bullet, but I’ve always said that these are extremely high value targets to hackers and I’m surprised there haven’t been more incidents like this - though Kaseya was also hacked to deploy crypto miners a few years ago too so this is not a one off event. Given that the nature of RMM software specifically requires being able to push arbitrary code to large numbers of devices security SHOULD be paramount and having a respected open source solution for this would be ideal
    - usr1106 5y ago
      Ansible, Puppet, Chef, Terraform maybe? I am not an expert in that area. I have used Terraform and seen Ansible. Know basicly nothing about the others.
usr1106 5y ago
Not very interesting information. They just describe how to see that the attack as happened for those running the system. Not how it happened in order to learn from and avoid the same mistake for the rest of us.
- Neil44 5y ago
  I heard on the grapevine it was an SQL injection, but thats more of a rumour at this point.
  - usr1106 5y ago
    The SQL injection theory has certainly been mentioned. But not with any evidence I would trust.
    - Neil44 5y ago
      Seems to be a bit more info here with CVE's etc - https://www.tenable.com/blog/cve-2021-30116-multiple-zero-day-vulnerabilities-in-kaseya-vsa-exploited-to-distribute-ransomware https://www.tenable.com/blog/cve-2021-30116-multiple-zero-da...
learc83 5y ago
A former boss used to be a higher up at Kaseya before I worked with him. From his description of their development process, and his general lack of respect for developers, I’m not surprised. According to him, they outsourced the majority of their development, and he had no clue that there could be any downside to this. Within a few months of taking over, every single developer in the company left.
terom 5y ago
https://www.reddit.com/r/msp/comments/ocggbv/comment/h3u5j2e https://www.reddit.com/r/msp/comments/ocggbv/comment/h3u5j2e the Reddit comments from Huntress Labs contain more concrete interim technical details from their investigations. * `/dl.asp` authentication bypass * `/KUpload.dll` file upload, likely using authenticated session * `/userFilterTableRpt.asp` command execution on previously uploaded file
- bithavoc 5y ago
  From Update 12: > The userFilterTableRpt.asp file contains a significant amount of potential SQL injection vulnerabilities, which would offer an attack vector for code execution and the ability to compromise the VSA server. It's been 19 years[0] since Kaseya added this task to their BugZilla: "Migrate from Active Server Pages to ASP.NET". [0]https://en.wikipedia.org/wiki/Active_Server_Pages https://en.wikipedia.org/wiki/Active_Server_Pages
- terom 5y ago
  It's also interesting to note that their initial on-prem VSA fix/updates will include disabling some functionality: https://helpdesk.kaseya.com/hc/en-gb/articles/4403440684689-Important-Notice-July-7th-2021 https://helpdesk.kaseya.com/hc/en-gb/articles/4403440684689-... > We will be releasing VSA with staged functionality to bring services back online sooner. The first release will prevent access to functionality used by a very small fraction of our user base, including: > * Classic Ticketing > * Classic Remote Control (not LiveConnect). > * User Portal Without knowing anything about Kaseya, it sounds to me like the exploited endpoints were related to some kind of legacy features with a higher level of technical debt, and fixing all of the identified vulnerabilities in those components will take time.
  - terom 5y ago
    And apparently one of the vulnerabilities exploited in this attack had been discovered and reported to Kaseya by independent researchers back in April: https://news.ycombinator.com/item?id=27763202 https://news.ycombinator.com/item?id=27763202
uniqueuid 5y ago
Not a good sign when your official incident description has a typo [1]. Not that perfect language is important in and of itself. But it indicates that stress is leaking through into the official communication. [1] (To date, we are aware of fewer than 60 Kaseya customers, all of which were using the VSA on-premises product, who was directly compromised by this attack.
dang 5y ago
Past related threads: Up to 1,500 businesses affected by ransomware attack, Kaseya CEO says - https://news.ycombinator.com/item?id=27750040 https://news.ycombinator.com/item?id=27750040 - July 2021 (290 comments) Cybercrime REvil gang asks for $70M to decrypt systems locked in Kaseya attack - https://news.ycombinator.com/item?id=27734282 https://news.ycombinator.com/item?id=27734282 - July 2021 (39 comments) Supermarket chain Coop closes 800 stores following Kaseya ransomware attack - https://news.ycombinator.com/item?id=27725576 https://news.ycombinator.com/item?id=27725576 - July 2021 (16 comments) Major Swedish supermarket chain hit by cyberattack - https://news.ycombinator.com/item?id=27720623 https://news.ycombinator.com/item?id=27720623 - July 2021 (174 comments) REvil ransomware executes supply chain attack via malicious Kaseya update - https://news.ycombinator.com/item?id=27716383 https://news.ycombinator.com/item?id=27716383 - July 2021 (13 comments) Others?
wanderingmind 5y ago
I have basic computer knowledge but no clue on cyber security. Can someone explain what was the negligence that lead to this attack?
- SAI_Peregrinus 5y ago
  They installed a centralized backdoor (remote management software) on all their computers, allowing attackers to use that to run their ransomware on all the computers instead of just one.
  - wanderingmind 5y ago
    Im assuming these softwares are customised for every customer. So the attacker was able to get elevated privileges at the software level that made them run randsomware across multiple customers.

37 points by gostsamo 5y ago

ijidak 5y ago

So is this vulnerability basically that they didn't sign and check hashes of their executables and downloads?

gostsamo 5y ago
> The attackers were able to exploit zero-day vulnerabilities in the VSA product to bypass authentication and run arbitrary command execution. This is all that we have so far.
- cartoonworld 5y ago
  This sounds like a 10/10 chained out of a rookie PKI mistake in their (editorializing here) poorly audited critical software infrastructure. Surely nobody steering the ship has been bitten by shitty PKI practices before, right? These people have no care for the work of others they have trashed. The acquisition culture is so reprehensibly transparent and incompetent, it would blow my mind if it weren't just another day at the office. This company doesn't even has a CISO from what I can see. They must rely on their CTO's outstanding track record of industry-leading assurance technologies in a rapidly evolving landscape (of cash) Refer to https://www.kaseya.com/company/ https://www.kaseya.com/company/ A) Their CTO presided over DigiCert, who botched 23,000 orthogonally integrated, outside-of-the-box, innovative security certificates[0] through their bad practices and garbage organization while meeting committed timeframes. Dan Timpson leads all product development, security and cloud/SaaS operations teams across all lines of business for all Kaseya companies including Unitrends, Spanning Cloud Apps, Rapidfire Tools, ID Agent, Graphus, RocketCyber and IT Glue. He joins Kaseya with a proven track record of success having led and grown engineering and R&D teams to build architectures focused on security and integrity, data compliance and testing. Through his leadership, Timpson empowers engineering teams to develop solutions that emphasize quality, simplicity and reliability while meeting committed timeframes. Prior to joining Kaseya, Mr. Timpson served as CTO of DigiCert, a provider of high-assurance digital certificates that deliver trusted SSL, private and managed PKI deployments and device certificates for the emerging IoT market. There, Timpson was responsible for Digicert’s technology strategy and played a vital role in leading the security industry by driving innovation. Prior to joining DigiCert, Mr. Timpson worked for Microsoft Corp leading virtualization technology engineering teams and driving the division’s software security release discipline. Earlier in his career, he managed engineering teams at Novell over the span of nine years building identity and access management systems and Novell’s underlying PKI framework. Mr. Timpson holds a Masters of Business Administration and Technology Management from Westminster College, and holds the patents to five technology innovations. B) Their current CEO was onboarded after presiding over a number of pricey acquisitions in whatever tech fads, driving "unprecedented growth" with his undoubtedly forward-looking vision and commitment to shareholder value. Fred Voccola leads the vision, strategy and growth of Kaseya and its family of brands including Unitrends, Spanning Cloud Apps, Rapidfire Tools, IT Glue, Graphus, RocketCyber and ID Agent. Prior to joining Kaseya, Mr. Voccola served as president and general manager of Yodle’s Brand Networks Division, providing digital and market automation solutions to small and medium businesses. In 2016, Yodle successfully was sold to Web.com (NASDAQ: Web). Prior to that, Mr. Voccola was president of Nolio, Inc., a devops SaaS company, where he drove 100% year-over-year growth and successfully sold the business to CA Technologies. Mr. Voccola also co-founded and served as president and CEO of Trust Technology Corp., where he drove significant growth over three years prior to the company’s sale to FGI Global. Earlier in his career, Mr. Voccola co-founded and served as COO of Identify Software, substantially growing the business during his five-year tenure, culminating in a sale to BMC Software. He then served as vice president of worldwide sales and services at BMC Software. In addition, Mr. Voccola has held various management and executive roles at Intira (acquired by Divine Systems) and Prism Solutions (acquired by Ardent Software/Informix/IBM). Mr. Voccola holds a Bachelor of Science in Finance from the Carroll School of Management at Boston College. This stuff pisses me off to no end, especially after all the years of being called a crackpot and how nobody would do that or how could they know or whatever. I'm not a genius, this is crayon on the wall. At least they've "created" a lot of jobs lately. Probably toasted a few, too. They'll be off to the next clown car soon enough, probably driving unprecedented growth in next-generation managed service applications for dialysis machines and pesticide application PLCs. In the cloud. [0] - https://www.bleepingcomputer.com/news/security/23-000-users-lose-ssl-certificates-in-trustico-digicert-spat/ https://www.bleepingcomputer.com/news/security/23-000-users-...
  - toast0 5y ago
    > Their CTO presided over DigiCert, who botched 23,000 orthogonally integrated, outside-of-the-box, innovative security certificates[0] through their bad practices and garbage organization while meeting committed timeframes. I had forgotten about this incident. Somehow I thought this happened before 2018. Anyway, I'm not sure how any of this is DigiCert's fault. Maybe they should have had mandatory audits of resellers to make sure they didn't do stupid things like have the private keys for customer certificates... But given that the reseller sent the private keys to the CA, that's pretty good evidence the keys are compromised, and worthy of revocation.
    - cartoonworld 5y ago
      Indeed, they had one job...
usr1106 5y ago
It's not said that they (the vendors code) downloaded that code. If they have a remote code execution elsewhere it can download more. Does anybody check the signature of all DLLs they are loading (into memory, not downloading) and all executables they are a calling? I have not seen such system in real life. The Linux kernel can check the modules it loads, but that's a different level. And I am not sure how widely it is used in practice. Anyway if you can inject a bad kernel module you are already root and have many more options.
- fmajid 5y ago
  macOS does with notarization.
  - usr1106 5y ago
    Good point, I am not a MacOS user. Users can authorize unnotarized executables to be run according to a quick Web search. So what would prevent malware from authorizing its unnotarized binaries? In the Kaseya case it seemed to have elevated privileges already, because it was able to install bad binaries.

binkHN 5y ago

Sadly, not a whole lot of detail here.

dt3ft 5y ago

Can anyone share the actual binaries used? (Specifically, the dll cryptor which was sideloaded).

bigmattystyles 5y ago

Does Kaseya survive this?

gostsamo 5y ago
Yes, I think. Depends on their management of the crisis. Hacks are part of life nowadays and it takes several examples of negligence before disengaging your users.
usr1106 5y ago
In my thinking they should not. Closed source is just such an inferior development model. Not that bad practices and stupid mistake would not happen in open source. It has happened and will certainly happen again. But more eyes are always better. And people who value deep technical understanding over just making money are often found in an open source ecosystem. It is incredibly frustrating to have to work with black boxes like Windows where you have inferior choices to explore and fix things. Obviously there are many decision makers who don't see this. So shops like Kaseya do unfortunately survive.
- funnybeam 5y ago
  Is there an open source Remote Monitoring and Management solution though? My company used to use Kaseya but luckily changed to a competitor a few years ago so dodged this particular bullet, but I’ve always said that these are extremely high value targets to hackers and I’m surprised there haven’t been more incidents like this - though Kaseya was also hacked to deploy crypto miners a few years ago too so this is not a one off event. Given that the nature of RMM software specifically requires being able to push arbitrary code to large numbers of devices security SHOULD be paramount and having a respected open source solution for this would be ideal
  - usr1106 5y ago
    Ansible, Puppet, Chef, Terraform maybe? I am not an expert in that area. I have used Terraform and seen Ansible. Know basicly nothing about the others.

usr1106 5y ago

Not very interesting information. They just describe how to see that the attack as happened for those running the system. Not how it happened in order to learn from and avoid the same mistake for the rest of us.

Neil44 5y ago
I heard on the grapevine it was an SQL injection, but thats more of a rumour at this point.
- usr1106 5y ago
  The SQL injection theory has certainly been mentioned. But not with any evidence I would trust.
  - Neil44 5y ago
    Seems to be a bit more info here with CVE's etc - https://www.tenable.com/blog/cve-2021-30116-multiple-zero-day-vulnerabilities-in-kaseya-vsa-exploited-to-distribute-ransomware https://www.tenable.com/blog/cve-2021-30116-multiple-zero-da...

learc83 5y ago

A former boss used to be a higher up at Kaseya before I worked with him. From his description of their development process, and his general lack of respect for developers, I’m not surprised. According to him, they outsourced the majority of their development, and he had no clue that there could be any downside to this. Within a few months of taking over, every single developer in the company left.

terom 5y ago

https://www.reddit.com/r/msp/comments/ocggbv/comment/h3u5j2e https://www.reddit.com/r/msp/comments/ocggbv/comment/h3u5j2e the Reddit comments from Huntress Labs contain more concrete interim technical details from their investigations. * `/dl.asp` authentication bypass * `/KUpload.dll` file upload, likely using authenticated session * `/userFilterTableRpt.asp` command execution on previously uploaded file

bithavoc 5y ago
From Update 12: > The userFilterTableRpt.asp file contains a significant amount of potential SQL injection vulnerabilities, which would offer an attack vector for code execution and the ability to compromise the VSA server. It's been 19 years[0] since Kaseya added this task to their BugZilla: "Migrate from Active Server Pages to ASP.NET". [0]https://en.wikipedia.org/wiki/Active_Server_Pages https://en.wikipedia.org/wiki/Active_Server_Pages
terom 5y ago
It's also interesting to note that their initial on-prem VSA fix/updates will include disabling some functionality: https://helpdesk.kaseya.com/hc/en-gb/articles/4403440684689-Important-Notice-July-7th-2021 https://helpdesk.kaseya.com/hc/en-gb/articles/4403440684689-... > We will be releasing VSA with staged functionality to bring services back online sooner. The first release will prevent access to functionality used by a very small fraction of our user base, including: > * Classic Ticketing > * Classic Remote Control (not LiveConnect). > * User Portal Without knowing anything about Kaseya, it sounds to me like the exploited endpoints were related to some kind of legacy features with a higher level of technical debt, and fixing all of the identified vulnerabilities in those components will take time.
- terom 5y ago
  And apparently one of the vulnerabilities exploited in this attack had been discovered and reported to Kaseya by independent researchers back in April: https://news.ycombinator.com/item?id=27763202 https://news.ycombinator.com/item?id=27763202

uniqueuid 5y ago

Not a good sign when your official incident description has a typo [1]. Not that perfect language is important in and of itself. But it indicates that stress is leaking through into the official communication. [1] (To date, we are aware of fewer than 60 Kaseya customers, all of which were using the VSA on-premises product, who was directly compromised by this attack.

dang 5y ago

Past related threads: Up to 1,500 businesses affected by ransomware attack, Kaseya CEO says - https://news.ycombinator.com/item?id=27750040 https://news.ycombinator.com/item?id=27750040 - July 2021 (290 comments) Cybercrime REvil gang asks for $70M to decrypt systems locked in Kaseya attack - https://news.ycombinator.com/item?id=27734282 https://news.ycombinator.com/item?id=27734282 - July 2021 (39 comments) Supermarket chain Coop closes 800 stores following Kaseya ransomware attack - https://news.ycombinator.com/item?id=27725576 https://news.ycombinator.com/item?id=27725576 - July 2021 (16 comments) Major Swedish supermarket chain hit by cyberattack - https://news.ycombinator.com/item?id=27720623 https://news.ycombinator.com/item?id=27720623 - July 2021 (174 comments) REvil ransomware executes supply chain attack via malicious Kaseya update - https://news.ycombinator.com/item?id=27716383 https://news.ycombinator.com/item?id=27716383 - July 2021 (13 comments) Others?

wanderingmind 5y ago

I have basic computer knowledge but no clue on cyber security. Can someone explain what was the negligence that lead to this attack?

SAI_Peregrinus 5y ago
They installed a centralized backdoor (remote management software) on all their computers, allowing attackers to use that to run their ransomware on all the computers instead of just one.
- wanderingmind 5y ago
  Im assuming these softwares are customised for every customer. So the attacker was able to get elevated privileges at the software level that made them run randsomware across multiple customers.