6 ms·

> At the time optimized jar change broke antivirus scanners, which further sped up Firefox startup :) I'm pretty sure that counts as cheating. AV overhead is

> At the time optimized jar change broke antivirus scanners, which further sped up Firefox startup :) I'm pretty sure that counts as cheating. AV overhead is a real issue though. I was recently copying a directory full of small files from one fast SSD to another (larger) one. I started the transfer and Windows estimated about 8 hours to finish the transfer, and the overall progress seemed to be roughly on track. On a whim I tried temporarily turning off Microsoft's built-in antivirus scanning. Total transfer time dropped to about 40 minutes.

cl0ckt0wer 5y ago
the built in AV scans on close of file handle. moving this to a background thread will really speed things up. https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5T4/m/WAMC_Q-DCAAJ https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5...
- cl0ckt0wer 5y ago
  Found the article that goes into depth on this issue https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ and the bug link: https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c94393dbf45ad417da257fe0 https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c9...
jazzyjackson 5y ago
Wild that windows feels it important to scan every file going from one local drive to another. my laptop has gotten a lot quieter after turning off virus protection, I wonder how much electricity is burned just idley checking for viruses. Edit: thanks for the replies, on second thought it is pretty tricky to know a file you’re writing already exists without trusting the process doing the copying
- passerby1 5y ago
  Interesting question! When mankind will really need to calculate costs and try to get rid of such an entrenched "habits" as av-scanning (same as to crypto-mining nowadays), it could be too late to do something. Or take banks. I was amazed to see how much manual work is done in financial institutions. Skyscrapers with bank name on top of it filled with tens of thousands people. Everyone has computer, among other things burning electricity. Separately, datacenters. How does it compares to crypto? I doubt it's less, if sum up all banks. But anyway, av-scanning is doing much good, but can be socially out-engineered in a single click, making it useless in most critical cases.
  - nightfly 5y ago
    Re: banks/crypto Crypto's current energy consumption is doing a miniscule fraction of the transaction handling that banks are. If you scaled up current crypto to match banks, I don't think that would even be possible.
    - therein 5y ago
      Crypto's power consumption is not truly related to the transaction volume.
      - nwah1 5y ago
        Right, the transaction volume it can support will never change, but the energy consumption rises as a function of the price, with no upper bound.
        Groxx 5y ago
        ... every piece of that sentence is straightforwardly incorrect.
        jazzyjackson 5y ago
        Do you want to explain how? AFAIK The number of transactions per block and the rate at which blocks are confirmed has nothing to do with how many people are burning through hashes. And the price of a token is a kind of bounty for mining a block: the higher the price, the more money can be spent trying to win the bounty while still being profitable. You wouldn't see data centers full of ASICs if the reward was only a few hundred bucks. So how straight forward is it?
- deleted 5y ago
  [deleted]
- dataflow 5y ago
  I think it's probably because the system doesn't see it as "copying", it sees it as "writing arbitrary data that just-so-happens-to-be identical to existing data in another open file". It'd be tricky to figure out (and verify in a secure manner) that data is in fact being copied.
  - saghm 5y ago
    I mean, if you're literally using the Windows utility for copying files (either by clicking copy/paste in Explorer or using whatever the DOS equivalent of "cp" is), it seem like it shouldn't be that tricky for them to make it work at least for the built-in AV, which is what the parent comment was talking about.
    - dataflow 5y ago
      I think that depends on whether you want to trust already-executing programs or not. If you trust them, then sure, it's pretty straightforward: you just tell the AV what to scan vs. not. If you don't, then it's not: you have to either offload the copying to kernel-mode, or delegate it to a secured user-mode process, because otherwise a malicious program could just modify the buffer you're copying from. The former makes sense from a performance standpoint but the latter also makes sense from a security standpoint.
    - makeitdouble 5y ago
      An issue is the original files are not guaranteed to be trusted. If for instance your were copying files from a flash drive or a network permission, it would seem natural to rescan every file that is copied to a local disk. From there the more the "outside"/"inside" line gets blurry (e.g. what if it was synced from Dropbox ? or data got injected by an arbitrary program etc.), the more it becomes justified to just scan every file moving anywhere. If the goal is indeed to scan files in the first place.
  - buraktamturk 5y ago
    If I remember correctly the CreateFileEx win32 api takes a flag and template file handle to copy when creating a new file. Maybe the copy operations can use this api and expect AV software to ignore the operation. (I have no idea about the performance of such approach) Or maybe a syscall (preferably at filesystem level) that can be introduced for bulk copying purposes and AVs can ignore such calls.
- greggman3 5y ago
  I don't know under what circumstances Windows scans files but I do no that copying a file to the same drive does not rescan the file. I found this out because grunt-copy-contrib copies incorrectly by opening the src, reading bytes, writing to dest instead of just asking the OS to copy the file. Switch it to ask the OS to copy the file removed the scanning (windows already knows it scanned the file so no need to scan a copy).
thrower123 5y ago
One hasn't felt pain until one has dealt with large npm-based projects on a corporate machine that has an hdd and two competing antivirus scanners battling to scan all 18,000 files in node_modules every time you run webpack.
roca 5y ago
Anything that reduces the startup time users experience is not cheating.
jijji 5y ago
That's primarily a Windows problem though, right?
- josephcsible 5y ago
  McAfee exists for Linux, it's just as bad as you'd think it is, and some of us have the misfortune of being forced to use it on every workstation and server.
roywiggins 5y ago
Even worse than that, I've had my antivirus freak out and BSOD the system during copying too many files around inside WSL1, where "too many" was "checked out a nontrivial git repo." It was repeatable. It meant WSL1 was just not usable at all.
- dgellow 5y ago
  You can add the top directory you use for dev projects to Windows Defender exclusion list. Also, your build tools or IDE processes can be added. For example excluding devenv.exe has a very important impact on Visual Studio performances.
TomGullen 5y ago
Took me longer than it should of to realise Windows Defender scanning everything in wwroot everytime we published a new build of the site was significantly increasing downtime!
- KronisLV 5y ago
  A part of me wants to suggest that you look into using something other than Windows for hosting sites, since in my experience GNU/Linux servers are far less painful to deal with, especially when it comes to services and file permissions. However, it's likely that you have your own reasons for using it (e.g. many projects out there are stuck with legacy versions of .NET and Mono wasn't really stable for that use case either). On the other hand, i can't help but to wonder about why we don't see more anti virus/malware scanning solutions for GNU/Linux, especially with how coupled our development workflows are to the Internet - i've had Ruby gems with malware in them be stopped by Kaspersky on Windows, and honestly the thought of pip, npm and many other package managers being able to compile god knows what and execute random scripts in a non-sandboxed environment is horrifying. Market share aside, i bet we still see malware targeting *nix be developed, especially in the age of pre-compiled binary blobs and such, especially when the complexity of everything increases to the point where you can't audit everything that's present with your current resources, which will probably only become a larger problem with time. So, why don't we have an equivalent to Windows Defender that's either preinstalled or recommended during installation, like you can choose between different desktops while installing distros? Why don't we talk more about something like https://www.kaspersky.com/small-to-medium-business-security/endpoint-linux https://www.kaspersky.com/small-to-medium-business-security/... ?
hyperman1 5y ago
Let me present you McAfee copying a folder full of zip files, at least this is what it did five years ago. First of all, it is horribly slow scanning any kind of zipped content. Second, it scans a whole folder if you touch one file. Third, it forgets you scanned a file if it gets modified, even before closing the file. So you copy a buffer full (64k?) of bytes, and mcafee scans the full folder. Then you copy another buffer full, and it scans the full folder again. And so on. I saw the windows estimated copy time go down from multiple days to a few minutes by adding an exclusion on a folder. Another fun McAfee tip is to keep your recycle bin as empty as possible, especially for zip files, because windows will look at the content regularly, so mcafee keeps on scanning it again and again.
- formerly_proven 5y ago
  McAfee is still utter garbage (like, well... most "security" software) that will make an NVMe SSD look like a late 90s HDD.
- franga2000 5y ago
  > Another fun McAfee tip Here's my favourite one, which was even supported by the man himself: uninstall it as soon as possible! It's not only a pretty bad antivirus, but seems to be straight up dangerous, having had some of the most destructive bugs I've seen in any software (like the time it bootlooped millions of PCs around the world- including hospitals and police stations!). There are so many better options available, I'm amazed anyone actually uses it (unless 100% of their install base is people who got it preinstalled).
  - hyperman1 5y ago
    You're my new best friend! I'd give you the contact for corporate IT, but even they ran away to MS Defender. McAfee's users are not the people who suffer it on their PC, but the manager who love a big name and hate a migration, and the windows domain admins who love the centralized console. The damage caused by user frustration, lost time waiting, troubleshooting, and actual hardware obsolete before its time is not their concern. I can tell from experience the windows domain admins disable it on their own laptop, as they 'know what they are doing', unlike the rest of us.
BiteCode_dev 5y ago
Once, I was working on an intranet web tool generating .odt documents for the user to print. It worked perfectly, except on one machine, where the document could be downloaded but LibreOffice raised an error when opening it. At first I though I had a bug. Checked my code, nope. Then I though maybe there is a problem with LO, but no, same version, nothing special. It worked with a document I brough from an USB disk. After an afternoon of frustration I noted Kaspersky running in the task bar, and did one last desperate attempt at finding the solution by disabling it. It worked. Inspecting the file revealed some parts were missing, Kaspersky assumed it was a malware, and not only intercepted the download, but removed silently part of the data on the fly. No alarm, no pop up, just a broken file. Disabling the antivirus was not acceptable for the client, so I created a self signed certificate, and turned https on. The virtual vigilante couldn't spy on the wire anymore, and the user could download her files, unharmed.
- thrower123 5y ago
  I had a development machine once that had the weirdest problem building one particular C# project. Always complaining that one particular Microsoft DLL was missing. Double check references and Nuget, everything looks right, do a full package restore, no joy. Antivirus was eating this particular DLL every time it was downloaded. I copied a bit-for-bit identical version of the file from a different machine into the correct directory, and bizarrely that worked fine.
  - mort96 5y ago
    I had a very similar issue with BitDefender recently. When downloading the WebRTC C++ library, one of the submodules is harfbuzz, which contains some maliciously crafted font files used by the test suite. BitDefender automatically deletes those test files. That means git sees the repository as dirty, since files are deleted. That means WebRTC's fetch script can't run the 'git checkout' commands it expects to be able to run, and the whole download fails. While my summary here is brief, debugging this whole mess took a while.
resonious 5y ago
This kind of thing really makes me wonder how safe these AV systems really are. Code that runs on every single file dropped into the user's hard drive seems like a great vehicle for zero click exploits.
alibert 5y ago
Windows Defender is the worst AV in term of performance impact on the system. It's so significant in IO application that I don't understand why it is not number one top priority for the Defender team. It has been like this for several years. I have a paid AV just for this issue (avoid the free one). https://www.av-comparatives.org/tests/performance-test-october-2021/ https://www.av-comparatives.org/tests/performance-test-octob...
arghwhat 5y ago
> I'm pretty sure that counts as cheating. It also shows how useless most AV is.
Thiez 5y ago
At a previous job we discovered that the software was strangely slow when running locally. It turns out that the antivirus (ESET, I believe?) was delaying every WCF request by 0.5 seconds. After adding encryption to the connections the delay magically went away.

cl0ckt0wer 5y ago

the built in AV scans on close of file handle. moving this to a background thread will really speed things up. https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5T4/m/WAMC_Q-DCAAJ https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5...

cl0ckt0wer 5y ago
Found the article that goes into depth on this issue https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ and the bug link: https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c94393dbf45ad417da257fe0 https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c9...

jazzyjackson 5y ago

Wild that windows feels it important to scan every file going from one local drive to another. my laptop has gotten a lot quieter after turning off virus protection, I wonder how much electricity is burned just idley checking for viruses. Edit: thanks for the replies, on second thought it is pretty tricky to know a file you’re writing already exists without trusting the process doing the copying

passerby1 5y ago
Interesting question! When mankind will really need to calculate costs and try to get rid of such an entrenched "habits" as av-scanning (same as to crypto-mining nowadays), it could be too late to do something. Or take banks. I was amazed to see how much manual work is done in financial institutions. Skyscrapers with bank name on top of it filled with tens of thousands people. Everyone has computer, among other things burning electricity. Separately, datacenters. How does it compares to crypto? I doubt it's less, if sum up all banks. But anyway, av-scanning is doing much good, but can be socially out-engineered in a single click, making it useless in most critical cases.
- nightfly 5y ago
  Re: banks/crypto Crypto's current energy consumption is doing a miniscule fraction of the transaction handling that banks are. If you scaled up current crypto to match banks, I don't think that would even be possible.
  - therein 5y ago
    Crypto's power consumption is not truly related to the transaction volume.
    - nwah1 5y ago
      Right, the transaction volume it can support will never change, but the energy consumption rises as a function of the price, with no upper bound.
      - Groxx 5y ago
        ... every piece of that sentence is straightforwardly incorrect.
        jazzyjackson 5y ago
        Do you want to explain how? AFAIK The number of transactions per block and the rate at which blocks are confirmed has nothing to do with how many people are burning through hashes. And the price of a token is a kind of bounty for mining a block: the higher the price, the more money can be spent trying to win the bounty while still being profitable. You wouldn't see data centers full of ASICs if the reward was only a few hundred bucks. So how straight forward is it?
deleted 5y ago
[deleted]
dataflow 5y ago
I think it's probably because the system doesn't see it as "copying", it sees it as "writing arbitrary data that just-so-happens-to-be identical to existing data in another open file". It'd be tricky to figure out (and verify in a secure manner) that data is in fact being copied.
- saghm 5y ago
  I mean, if you're literally using the Windows utility for copying files (either by clicking copy/paste in Explorer or using whatever the DOS equivalent of "cp" is), it seem like it shouldn't be that tricky for them to make it work at least for the built-in AV, which is what the parent comment was talking about.
  - dataflow 5y ago
    I think that depends on whether you want to trust already-executing programs or not. If you trust them, then sure, it's pretty straightforward: you just tell the AV what to scan vs. not. If you don't, then it's not: you have to either offload the copying to kernel-mode, or delegate it to a secured user-mode process, because otherwise a malicious program could just modify the buffer you're copying from. The former makes sense from a performance standpoint but the latter also makes sense from a security standpoint.
  - makeitdouble 5y ago
    An issue is the original files are not guaranteed to be trusted. If for instance your were copying files from a flash drive or a network permission, it would seem natural to rescan every file that is copied to a local disk. From there the more the "outside"/"inside" line gets blurry (e.g. what if it was synced from Dropbox ? or data got injected by an arbitrary program etc.), the more it becomes justified to just scan every file moving anywhere. If the goal is indeed to scan files in the first place.
- buraktamturk 5y ago
  If I remember correctly the CreateFileEx win32 api takes a flag and template file handle to copy when creating a new file. Maybe the copy operations can use this api and expect AV software to ignore the operation. (I have no idea about the performance of such approach) Or maybe a syscall (preferably at filesystem level) that can be introduced for bulk copying purposes and AVs can ignore such calls.
greggman3 5y ago
I don't know under what circumstances Windows scans files but I do no that copying a file to the same drive does not rescan the file. I found this out because grunt-copy-contrib copies incorrectly by opening the src, reading bytes, writing to dest instead of just asking the OS to copy the file. Switch it to ask the OS to copy the file removed the scanning (windows already knows it scanned the file so no need to scan a copy).

thrower123 5y ago

One hasn't felt pain until one has dealt with large npm-based projects on a corporate machine that has an hdd and two competing antivirus scanners battling to scan all 18,000 files in node_modules every time you run webpack.

roca 5y ago

Anything that reduces the startup time users experience is not cheating.

jijji 5y ago

That's primarily a Windows problem though, right?

josephcsible 5y ago
McAfee exists for Linux, it's just as bad as you'd think it is, and some of us have the misfortune of being forced to use it on every workstation and server.

roywiggins 5y ago

Even worse than that, I've had my antivirus freak out and BSOD the system during copying too many files around inside WSL1, where "too many" was "checked out a nontrivial git repo." It was repeatable. It meant WSL1 was just not usable at all.

dgellow 5y ago
You can add the top directory you use for dev projects to Windows Defender exclusion list. Also, your build tools or IDE processes can be added. For example excluding devenv.exe has a very important impact on Visual Studio performances.

TomGullen 5y ago

Took me longer than it should of to realise Windows Defender scanning everything in wwroot everytime we published a new build of the site was significantly increasing downtime!

KronisLV 5y ago
A part of me wants to suggest that you look into using something other than Windows for hosting sites, since in my experience GNU/Linux servers are far less painful to deal with, especially when it comes to services and file permissions. However, it's likely that you have your own reasons for using it (e.g. many projects out there are stuck with legacy versions of .NET and Mono wasn't really stable for that use case either). On the other hand, i can't help but to wonder about why we don't see more anti virus/malware scanning solutions for GNU/Linux, especially with how coupled our development workflows are to the Internet - i've had Ruby gems with malware in them be stopped by Kaspersky on Windows, and honestly the thought of pip, npm and many other package managers being able to compile god knows what and execute random scripts in a non-sandboxed environment is horrifying. Market share aside, i bet we still see malware targeting *nix be developed, especially in the age of pre-compiled binary blobs and such, especially when the complexity of everything increases to the point where you can't audit everything that's present with your current resources, which will probably only become a larger problem with time. So, why don't we have an equivalent to Windows Defender that's either preinstalled or recommended during installation, like you can choose between different desktops while installing distros? Why don't we talk more about something like https://www.kaspersky.com/small-to-medium-business-security/endpoint-linux https://www.kaspersky.com/small-to-medium-business-security/... ?

hyperman1 5y ago

Let me present you McAfee copying a folder full of zip files, at least this is what it did five years ago. First of all, it is horribly slow scanning any kind of zipped content. Second, it scans a whole folder if you touch one file. Third, it forgets you scanned a file if it gets modified, even before closing the file. So you copy a buffer full (64k?) of bytes, and mcafee scans the full folder. Then you copy another buffer full, and it scans the full folder again. And so on. I saw the windows estimated copy time go down from multiple days to a few minutes by adding an exclusion on a folder. Another fun McAfee tip is to keep your recycle bin as empty as possible, especially for zip files, because windows will look at the content regularly, so mcafee keeps on scanning it again and again.

formerly_proven 5y ago
McAfee is still utter garbage (like, well... most "security" software) that will make an NVMe SSD look like a late 90s HDD.
franga2000 5y ago
> Another fun McAfee tip Here's my favourite one, which was even supported by the man himself: uninstall it as soon as possible! It's not only a pretty bad antivirus, but seems to be straight up dangerous, having had some of the most destructive bugs I've seen in any software (like the time it bootlooped millions of PCs around the world- including hospitals and police stations!). There are so many better options available, I'm amazed anyone actually uses it (unless 100% of their install base is people who got it preinstalled).
- hyperman1 5y ago
  You're my new best friend! I'd give you the contact for corporate IT, but even they ran away to MS Defender. McAfee's users are not the people who suffer it on their PC, but the manager who love a big name and hate a migration, and the windows domain admins who love the centralized console. The damage caused by user frustration, lost time waiting, troubleshooting, and actual hardware obsolete before its time is not their concern. I can tell from experience the windows domain admins disable it on their own laptop, as they 'know what they are doing', unlike the rest of us.

BiteCode_dev 5y ago

Once, I was working on an intranet web tool generating .odt documents for the user to print. It worked perfectly, except on one machine, where the document could be downloaded but LibreOffice raised an error when opening it. At first I though I had a bug. Checked my code, nope. Then I though maybe there is a problem with LO, but no, same version, nothing special. It worked with a document I brough from an USB disk. After an afternoon of frustration I noted Kaspersky running in the task bar, and did one last desperate attempt at finding the solution by disabling it. It worked. Inspecting the file revealed some parts were missing, Kaspersky assumed it was a malware, and not only intercepted the download, but removed silently part of the data on the fly. No alarm, no pop up, just a broken file. Disabling the antivirus was not acceptable for the client, so I created a self signed certificate, and turned https on. The virtual vigilante couldn't spy on the wire anymore, and the user could download her files, unharmed.

thrower123 5y ago
I had a development machine once that had the weirdest problem building one particular C# project. Always complaining that one particular Microsoft DLL was missing. Double check references and Nuget, everything looks right, do a full package restore, no joy. Antivirus was eating this particular DLL every time it was downloaded. I copied a bit-for-bit identical version of the file from a different machine into the correct directory, and bizarrely that worked fine.
- mort96 5y ago
  I had a very similar issue with BitDefender recently. When downloading the WebRTC C++ library, one of the submodules is harfbuzz, which contains some maliciously crafted font files used by the test suite. BitDefender automatically deletes those test files. That means git sees the repository as dirty, since files are deleted. That means WebRTC's fetch script can't run the 'git checkout' commands it expects to be able to run, and the whole download fails. While my summary here is brief, debugging this whole mess took a while.

resonious 5y ago

This kind of thing really makes me wonder how safe these AV systems really are. Code that runs on every single file dropped into the user's hard drive seems like a great vehicle for zero click exploits.

alibert 5y ago

Windows Defender is the worst AV in term of performance impact on the system. It's so significant in IO application that I don't understand why it is not number one top priority for the Defender team. It has been like this for several years. I have a paid AV just for this issue (avoid the free one). https://www.av-comparatives.org/tests/performance-test-october-2021/ https://www.av-comparatives.org/tests/performance-test-octob...

arghwhat 5y ago

> I'm pretty sure that counts as cheating. It also shows how useless most AV is.

Thiez 5y ago

At a previous job we discovered that the software was strangely slow when running locally. It turns out that the antivirus (ESET, I believe?) was delaying every WCF request by 0.5 seconds. After adding encryption to the connections the delay magically went away.