18 ms·

Container Isolation Gone Wrong

270 points by knoxa2511 9y ago

steven777400 9y ago
I enjoy reading descriptions such as these that start with easily detectable problems and go into a level of debugging far beyond my skill level. Helps to illuminate some of those "unknown unknowns" that I didn't even ever consider before.
- daddykotex 9y ago
  I had the exact same feeling. Well written OP.
- oneplane 9y ago
  I had the same feeling, but this also reinforced my view that Docker and containerization in general (often used as a scapegoat to not have to do proper configuration management) 'for the masses' is more problematic than helpful. In most cases it doesn't solve anything but does add problems that can be hard to debug. The actual 'lack' of isolation wouldn't have happened with true virtualisation, and the method of debugging here is something most people that think they need containers won't have. To me, debugging like this is something that should be far more important to people than slinging words like Docker and NodeJS around all day. (and then mostly on Discord, or to them, the older Slack, but not IRC because that is too hard for that crowd -- totally unfounded opinion/rant)
  - dasil003 9y ago
    Docker didn't cause this problem, the point of the article is that Docker doesn't prevent all such problems. On the other hand it does solve a lot of packaging, dependency and environment parity problems that traditional virtualization is too heavy to accomplish. I'm old enough to also be frustrated with buzzword driven development, and it's pretty annoying that so many believe Docker invented containerization, but don't throw the baby out with the bathwater. Containerization is an awesome tool and orthogonal to config management.
    - eropple 9y ago
      Traditional virtualization is "too heavy", now, for solved problems like packaging? How and why?
      - omginternets 9y ago
        For the reasons mentioned in the article: - slow (re)start times - greater resource consumption Granted, "too heavy" is relative, but starting a few hundred VMs on a single host (assuming commodity hardware) is not going to work very well.
        user5994461 9y ago
        That works just fine. It's typical in a VM farm to have hosts with a hundred VMs.
        omginternets 9y ago
        >It's typical in a VM farm to have hosts with a hundred VMs. Several things: 1. We may have a different definition of "commodity hardware", but you're missing the broader point. 2. The broader point is that VMs are significantly less resource-efficient. 3. 1 & 2 notwithstanding, you're conveniently ignoring the issue of (re)start time 4. It's fine to use VMs, but it's frankly bizarre to fight tooth-and-nail over the ridiculous notion that they should always be preferred over containers.
        user5994461 9y ago
        I am simply addressing the fact that it's perfectly fine and common to have hosts with a hundred VMs and it works flawlessly. VMs are memory intensive because they duplicate the operating system. The starting point is around 500 MB per VM. That's the only meaningful difference in resources compared to containers. I am not discussing that they have different starting and stopping time.
        9y ago
  - bogomipz 9y ago
    >"I had the same feeling, but this also reinforced my view that Docker and containerization in general (often used as a scapegoat to not have to do proper configuration management) 'for the masses' is more problematic than helpful. In most cases it doesn't solve anything but does add problems that can be hard to debug." The issue described in this post has nothing to do with "config management vs containers." Its' odd that this article would have "reinforced" that view. How would configuration management have prevented a noisy neighbor? From the summary: "The core lesson of this story: just because you are using containers and you get the impression that your applications are perfectly virtualized and isolated, don’t assume the kernel is fully isolating every underlying resource at a container granularity." and "Luckily, the solution is there and rather simple: make sure to deeply monitor all your applications." That's nothing to do with any "configuration management vs containers" argument and everything to do with proper metrics collection and monitoring, which should be part of every "operational readiness" checklist whether Docker is used or not. Lastly saying that Docker "In most cases it doesn't solve anything" is an absurd statement. Do you believe that virtualization does't solve any problems? If so why do you imagine the Linux kernel supports it?
    - gaius 9y ago
      just because you are using containers and you get the impression that your applications are perfectly virtualized and isolated Anyone who believes in the first place this shouldn't be running production systems...
      - bogomipz 9y ago
        Your comment contains two sentence fragments, neither of which is coherent.
        gaius 9y ago
        Seems perfectly straightforward. If you believe containers give you the level of isolation that VMs would, then you have fundamental misunderstandings of the technology which in a sane organisation would preclude you from operating important systems.
  - user5994461 9y ago
    sysdig is a monitoring system for docker that sells for $25 per month per host. Part of the debugging method has to do with "let's show our product".
digi_owl 9y ago
Linux have all manner of oddities when it comes to IO it seems.
doomrobo 9y ago
What's the solution here? Can you limit d_entry table size per-process? Do you have to limit it globally? Is the answer to just not use containers?
- gighi 9y ago
  OP here The solution is very simple: as mentioned in the article, just use a newer kernel and always set memory limits for containers, the blog post is based on an older kernel (2.6.32) that quite a few people irresponsibly still use in containerized environments, mostly because EL6 is so popular among enterprises. In newer kernels, allocations from object pools are now tied to the limits of the memory cgroups that requested them in userspace, if any, so you wouldn't incur in this specific issue and you would just effectively have a container not being able to use more than X MB of dcache entries (although there are probably other minor ones, for example related to sharing global kernel mutexes and such).
  - kevinmgranger 9y ago
    > ES6 is so popular among enterprises I had to re-read this a few times-- I think you meant EL6, right?
    - gighi 9y ago
      Updated, thanks! I am working with ElasticSearch (ES) more than EL these days and my muscle memory tricked me ;)
  - fckdemodss 9y ago
    I ran into a similar issue with kernel memory caching behavior. While it's nice to just say LOL upgrade you fool, most of us are stuck with the environment were given. You can adjust kernel level memory behavior, in particular vfs_cache_pressure can be set very high to force dentry to empty more aggressively. https://www.kernel.org/doc/Documentation/sysctl/vm.txt https://www.kernel.org/doc/Documentation/sysctl/vm.txt
  - justtryiothelp 9y ago
everdayimhustln 9y ago
Containers != type 1 hypervisor with all-encompassing resource quotas, reservations and prioritization like VMware ESXi. The problem can be solved either using a hypervisor that deploys only one container per VM (with suitable paravirt/dedupe), or fix the OS to operate with much finer-grained resource contention and allocation knobs for each and every limited resource. The latter is superior when only a single OS is needed, because it reduces the need for virtualization as a crutch for inadequacies of the OS.
- AlphaSite 9y ago
  https://www.vmware.com/products/vsphere/integrated-containers.html https://www.vmware.com/products/vsphere/integrated-container... and I’ve developed a small docker backend for Xen previously (which is vaguely similar to vic).
gtirloni 9y ago
Ah, large directories and d_entries... the bane of any NAS operator. Having seen hundreds of OpenSolaris appliances being abused in similar ways, I can relate. It doesn't seem like Kubernetes supports I/O resource limiting at this point [0][1]. In any case, after a problem like this is identified, a cluster admin can use pod affinity/anti-affinity to avoid both apps co-existing on the same node [2]. EDIT: For hypervisor-based container runtime, check Frakti (https://github.com/kubernetes/frakti https://github.com/kubernetes/frakti) 0 - https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-types https://kubernetes.io/docs/concepts/configuration/manage-com... 1 - https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-con... 2 - http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kub...
- smarterclayton 9y ago
  Regarding why block io limiting isn't implemented yet in Kube - its really hard to make block io sharing work well without killing performance (it's easy for one workload to screw up another if it seeks at the wrong time, and ssds are fast enough that just having to check the io limits may severely limit your max throughout). If you read some of the proposals for io, the end goal is to make it easy to use multiple volumes per workload where possible, and have high level limits in place for other things like inodes, total writes, etc.
  - skyde 9y ago
    Isn't that what BFQ scheduler was designed for ?
    - smarterclayton 9y ago
      Yes, although even the 4.12 code can have substantial overhead vs NOOP (phoronix benchmarks, as an example). It's not a complete slam dunk - turning it on for io indifferent workloads might make sense, but not necessarily on all boxes. http://www.phoronix.com/scan.php?page=article&item=linux-412-io&num=2 http://www.phoronix.com/scan.php?page=article&item=linux-412...
jmull 9y ago
Nice problem solving. I'd classify the primary root cause as a kernel bug. It's good to make use of otherwise unused memory for caches, but not to the extent that the caches grow so large they slow things down. Secondarily, there's probably something wrong in a system where you have to constantly poll and attempt to access large numbers of files that don't exist. (But probably 100% of systems that do anything useful have at least some weird cruft like this somewhere in them at any given time, so I'm not judging.)
- tyingq 9y ago
  Web servers do that, especially when spiders ask for urls that don't exist.
  - jmull 9y ago
    That's a good point, although I don't know why a spider that isn't specifically malicious (or I guess just badly malfunctioning) would request millions of different files that don't exist, which is what's needed to trigger this problem.
    - tyingq 9y ago
      Millions would be unusual, but I could see it happening. An old domain that used to be an image host, for example.
- solatic 9y ago
  The author notes that future kernels did decide to introduce limits that would've prevented the slowdowns from happening, but the customer was running an outdated kernel. That's what made the article disappointing for me. Do all this impressive in-kernel debugging just to find out that you should've upgraded your systems first. Sigh...
  - jdmichal 9y ago
    But the fix was to limit the cache based on process memory constraints. jmull's point is that if a cache is permitted to blow out so big that lookups are impacting performance, then that cache is not really serving its purpose in the first place.
mathattack 9y ago
Lately I've learned to panic whenever a job candidate starts dropping Docker and Kubernetes in the interview.
- adamcw 9y ago
  That's odd criteria. Mind passing those resumes this way?
- striking 9y ago
  Maybe if they think it's a one-stop solution to hosting problems, yeah. But penalizing candidates just for familiarity with new technology?
  - mathattack 9y ago
    The issue is more on people dropping buzzwords rather than what they're actually doing it for.
    - striking 9y ago
      In my opinion, that's entirely the wrong mentality. If someone gave me a pile of raw buzzwords, or told me they used entirely the wrong tool because of buzzwords, then I'd penalize them, sure. But say they made something wonderful, and it was cleaner and more efficient because of their use of Docker/Kubernetes, and they had taken the time to figure out the tradeoffs inherent to that approach. Is that worth penalizing, from your point of view?
      - mathattack 9y ago
        Of course not. My issue is with leading with the tool, rather than the problem it's solving.
cratermoon 9y ago
I knew before I got into the meat of the article that it was going to be i/o contention. The first two sections talked about memory and cpu limits on the containers, but nothing about i/o rates. This was a Known Problem back in the 90s when a variety of filesystems (DEC's AdvFS being one) that were efforts to address the issues around dentries and inode. See also http://www.starcomsoftware.com/proj/usenet/doc/c-news.pdf http://www.starcomsoftware.com/proj/usenet/doc/c-news.pdf
- pvg 9y ago
  But it's not, strictly speaking, about i/o contention.
  - gighi 9y ago
    OP here That's correct, I should have included a chart explicitly measuring the I/O activity done by the two containers, but I can assure you there was literally no I/O activity, a dozen open files per second is a very negligible throughput. The bottleneck was solely in the cache.
    - pvg 9y ago
      Yep, the old version of this was 'why is tar/find/rsync/etc hosing my server even when I've [io]niced it to hell'. Except now (as everything else) with containers!
- RijilV 9y ago
  I think they're actually talking about blowing out the dentry cache, but sure IO contention is anoyher shared resource in containers. Depending on what you're doing you might run into issues in various networking limits (somaxconns comes to mind, as does what's left of the route cache), blowing out the page cache, or something eating up your memory bandwidth (maybe via some unlucky NUMA). All for containers but they don't solve the hard problems folks often ascribe to them, really just shows in most cases you don't need to solve the hard problems. Most of the time what containers are buying you is an easy deployment method that leverages some nice features in the OS to make believe you're on separate machines.
itaysk 9y ago
Really enjoyed this, thanks! One lesson I learned from this, and correct me if I'm misinterpreting the root cause, is that more memory is not always better. This is, to me a far more powerful lesson I gained from this article.
rfraile 9y ago
Close related to this other history: https://blog.booking.com/troubleshooting-a-journey-into-the-unknown.html https://blog.booking.com/troubleshooting-a-journey-into-the-...
Walf 9y ago
What I want to know is why the 'trasher' was looking directly for so many different files. Could it not parse some output of finding/listing existing files to look for its targets?
luord 9y ago
This was a great read, the conclusion of always monitoring, no matter what technologies one is using, should be obvious, but I've noticed that it really isn't, unfortunately. Even I fall into the trap and sometimes I wish I knew about all this stuff but, alas, I prefer development.

270 points by knoxa2511 9y ago

I enjoy reading descriptions such as these that start with easily detectable problems and go into a level of debugging far beyond my skill level. Helps to illuminate some of those "unknown unknowns" that I didn't even ever consider before.

daddykotex 9y ago
I had the exact same feeling. Well written OP.
oneplane 9y ago
I had the same feeling, but this also reinforced my view that Docker and containerization in general (often used as a scapegoat to not have to do proper configuration management) 'for the masses' is more problematic than helpful. In most cases it doesn't solve anything but does add problems that can be hard to debug. The actual 'lack' of isolation wouldn't have happened with true virtualisation, and the method of debugging here is something most people that think they need containers won't have. To me, debugging like this is something that should be far more important to people than slinging words like Docker and NodeJS around all day. (and then mostly on Discord, or to them, the older Slack, but not IRC because that is too hard for that crowd -- totally unfounded opinion/rant)
- dasil003 9y ago
  Docker didn't cause this problem, the point of the article is that Docker doesn't prevent all such problems. On the other hand it does solve a lot of packaging, dependency and environment parity problems that traditional virtualization is too heavy to accomplish. I'm old enough to also be frustrated with buzzword driven development, and it's pretty annoying that so many believe Docker invented containerization, but don't throw the baby out with the bathwater. Containerization is an awesome tool and orthogonal to config management.
  - eropple 9y ago
    Traditional virtualization is "too heavy", now, for solved problems like packaging? How and why?
    - omginternets 9y ago
      For the reasons mentioned in the article: - slow (re)start times - greater resource consumption Granted, "too heavy" is relative, but starting a few hundred VMs on a single host (assuming commodity hardware) is not going to work very well.
      - user5994461 9y ago
        That works just fine. It's typical in a VM farm to have hosts with a hundred VMs.
        omginternets 9y ago
        >It's typical in a VM farm to have hosts with a hundred VMs. Several things: 1. We may have a different definition of "commodity hardware", but you're missing the broader point. 2. The broader point is that VMs are significantly less resource-efficient. 3. 1 & 2 notwithstanding, you're conveniently ignoring the issue of (re)start time 4. It's fine to use VMs, but it's frankly bizarre to fight tooth-and-nail over the ridiculous notion that they should always be preferred over containers.
        user5994461 9y ago
        I am simply addressing the fact that it's perfectly fine and common to have hosts with a hundred VMs and it works flawlessly. VMs are memory intensive because they duplicate the operating system. The starting point is around 500 MB per VM. That's the only meaningful difference in resources compared to containers. I am not discussing that they have different starting and stopping time.
        9y ago
- bogomipz 9y ago
  >"I had the same feeling, but this also reinforced my view that Docker and containerization in general (often used as a scapegoat to not have to do proper configuration management) 'for the masses' is more problematic than helpful. In most cases it doesn't solve anything but does add problems that can be hard to debug." The issue described in this post has nothing to do with "config management vs containers." Its' odd that this article would have "reinforced" that view. How would configuration management have prevented a noisy neighbor? From the summary: "The core lesson of this story: just because you are using containers and you get the impression that your applications are perfectly virtualized and isolated, don’t assume the kernel is fully isolating every underlying resource at a container granularity." and "Luckily, the solution is there and rather simple: make sure to deeply monitor all your applications." That's nothing to do with any "configuration management vs containers" argument and everything to do with proper metrics collection and monitoring, which should be part of every "operational readiness" checklist whether Docker is used or not. Lastly saying that Docker "In most cases it doesn't solve anything" is an absurd statement. Do you believe that virtualization does't solve any problems? If so why do you imagine the Linux kernel supports it?
  - gaius 9y ago
    just because you are using containers and you get the impression that your applications are perfectly virtualized and isolated Anyone who believes in the first place this shouldn't be running production systems...
    - bogomipz 9y ago
      Your comment contains two sentence fragments, neither of which is coherent.
      - gaius 9y ago
        Seems perfectly straightforward. If you believe containers give you the level of isolation that VMs would, then you have fundamental misunderstandings of the technology which in a sane organisation would preclude you from operating important systems.
- user5994461 9y ago
  sysdig is a monitoring system for docker that sells for $25 per month per host. Part of the debugging method has to do with "let's show our product".

digi_owl 9y ago

Linux have all manner of oddities when it comes to IO it seems.

doomrobo 9y ago

What's the solution here? Can you limit d_entry table size per-process? Do you have to limit it globally? Is the answer to just not use containers?

gighi 9y ago
OP here The solution is very simple: as mentioned in the article, just use a newer kernel and always set memory limits for containers, the blog post is based on an older kernel (2.6.32) that quite a few people irresponsibly still use in containerized environments, mostly because EL6 is so popular among enterprises. In newer kernels, allocations from object pools are now tied to the limits of the memory cgroups that requested them in userspace, if any, so you wouldn't incur in this specific issue and you would just effectively have a container not being able to use more than X MB of dcache entries (although there are probably other minor ones, for example related to sharing global kernel mutexes and such).
- kevinmgranger 9y ago
  > ES6 is so popular among enterprises I had to re-read this a few times-- I think you meant EL6, right?
  - gighi 9y ago
    Updated, thanks! I am working with ElasticSearch (ES) more than EL these days and my muscle memory tricked me ;)
- fckdemodss 9y ago
  I ran into a similar issue with kernel memory caching behavior. While it's nice to just say LOL upgrade you fool, most of us are stuck with the environment were given. You can adjust kernel level memory behavior, in particular vfs_cache_pressure can be set very high to force dentry to empty more aggressively. https://www.kernel.org/doc/Documentation/sysctl/vm.txt https://www.kernel.org/doc/Documentation/sysctl/vm.txt
- justtryiothelp 9y ago

everdayimhustln 9y ago

Containers != type 1 hypervisor with all-encompassing resource quotas, reservations and prioritization like VMware ESXi. The problem can be solved either using a hypervisor that deploys only one container per VM (with suitable paravirt/dedupe), or fix the OS to operate with much finer-grained resource contention and allocation knobs for each and every limited resource. The latter is superior when only a single OS is needed, because it reduces the need for virtualization as a crutch for inadequacies of the OS.

AlphaSite 9y ago
https://www.vmware.com/products/vsphere/integrated-containers.html https://www.vmware.com/products/vsphere/integrated-container... and I’ve developed a small docker backend for Xen previously (which is vaguely similar to vic).

gtirloni 9y ago

Ah, large directories and d_entries... the bane of any NAS operator. Having seen hundreds of OpenSolaris appliances being abused in similar ways, I can relate. It doesn't seem like Kubernetes supports I/O resource limiting at this point [0][1]. In any case, after a problem like this is identified, a cluster admin can use pod affinity/anti-affinity to avoid both apps co-existing on the same node [2]. EDIT: For hypervisor-based container runtime, check Frakti (https://github.com/kubernetes/frakti https://github.com/kubernetes/frakti) 0 - https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-types https://kubernetes.io/docs/concepts/configuration/manage-com... 1 - https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-con... 2 - http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kub...

smarterclayton 9y ago
Regarding why block io limiting isn't implemented yet in Kube - its really hard to make block io sharing work well without killing performance (it's easy for one workload to screw up another if it seeks at the wrong time, and ssds are fast enough that just having to check the io limits may severely limit your max throughout). If you read some of the proposals for io, the end goal is to make it easy to use multiple volumes per workload where possible, and have high level limits in place for other things like inodes, total writes, etc.
- skyde 9y ago
  Isn't that what BFQ scheduler was designed for ?
  - smarterclayton 9y ago
    Yes, although even the 4.12 code can have substantial overhead vs NOOP (phoronix benchmarks, as an example). It's not a complete slam dunk - turning it on for io indifferent workloads might make sense, but not necessarily on all boxes. http://www.phoronix.com/scan.php?page=article&item=linux-412-io&num=2 http://www.phoronix.com/scan.php?page=article&item=linux-412...

jmull 9y ago

Nice problem solving. I'd classify the primary root cause as a kernel bug. It's good to make use of otherwise unused memory for caches, but not to the extent that the caches grow so large they slow things down. Secondarily, there's probably something wrong in a system where you have to constantly poll and attempt to access large numbers of files that don't exist. (But probably 100% of systems that do anything useful have at least some weird cruft like this somewhere in them at any given time, so I'm not judging.)

tyingq 9y ago
Web servers do that, especially when spiders ask for urls that don't exist.
- jmull 9y ago
  That's a good point, although I don't know why a spider that isn't specifically malicious (or I guess just badly malfunctioning) would request millions of different files that don't exist, which is what's needed to trigger this problem.
  - tyingq 9y ago
    Millions would be unusual, but I could see it happening. An old domain that used to be an image host, for example.
solatic 9y ago
The author notes that future kernels did decide to introduce limits that would've prevented the slowdowns from happening, but the customer was running an outdated kernel. That's what made the article disappointing for me. Do all this impressive in-kernel debugging just to find out that you should've upgraded your systems first. Sigh...
- jdmichal 9y ago
  But the fix was to limit the cache based on process memory constraints. jmull's point is that if a cache is permitted to blow out so big that lookups are impacting performance, then that cache is not really serving its purpose in the first place.

mathattack 9y ago

Lately I've learned to panic whenever a job candidate starts dropping Docker and Kubernetes in the interview.

adamcw 9y ago
That's odd criteria. Mind passing those resumes this way?
striking 9y ago
Maybe if they think it's a one-stop solution to hosting problems, yeah. But penalizing candidates just for familiarity with new technology?
- mathattack 9y ago
  The issue is more on people dropping buzzwords rather than what they're actually doing it for.
  - striking 9y ago
    In my opinion, that's entirely the wrong mentality. If someone gave me a pile of raw buzzwords, or told me they used entirely the wrong tool because of buzzwords, then I'd penalize them, sure. But say they made something wonderful, and it was cleaner and more efficient because of their use of Docker/Kubernetes, and they had taken the time to figure out the tradeoffs inherent to that approach. Is that worth penalizing, from your point of view?
    - mathattack 9y ago
      Of course not. My issue is with leading with the tool, rather than the problem it's solving.

cratermoon 9y ago

I knew before I got into the meat of the article that it was going to be i/o contention. The first two sections talked about memory and cpu limits on the containers, but nothing about i/o rates. This was a Known Problem back in the 90s when a variety of filesystems (DEC's AdvFS being one) that were efforts to address the issues around dentries and inode. See also http://www.starcomsoftware.com/proj/usenet/doc/c-news.pdf http://www.starcomsoftware.com/proj/usenet/doc/c-news.pdf

pvg 9y ago
But it's not, strictly speaking, about i/o contention.
- gighi 9y ago
  OP here That's correct, I should have included a chart explicitly measuring the I/O activity done by the two containers, but I can assure you there was literally no I/O activity, a dozen open files per second is a very negligible throughput. The bottleneck was solely in the cache.
  - pvg 9y ago
    Yep, the old version of this was 'why is tar/find/rsync/etc hosing my server even when I've [io]niced it to hell'. Except now (as everything else) with containers!
RijilV 9y ago
I think they're actually talking about blowing out the dentry cache, but sure IO contention is anoyher shared resource in containers. Depending on what you're doing you might run into issues in various networking limits (somaxconns comes to mind, as does what's left of the route cache), blowing out the page cache, or something eating up your memory bandwidth (maybe via some unlucky NUMA). All for containers but they don't solve the hard problems folks often ascribe to them, really just shows in most cases you don't need to solve the hard problems. Most of the time what containers are buying you is an easy deployment method that leverages some nice features in the OS to make believe you're on separate machines.

itaysk 9y ago

Really enjoyed this, thanks! One lesson I learned from this, and correct me if I'm misinterpreting the root cause, is that more memory is not always better. This is, to me a far more powerful lesson I gained from this article.

rfraile 9y ago

Close related to this other history: https://blog.booking.com/troubleshooting-a-journey-into-the-unknown.html https://blog.booking.com/troubleshooting-a-journey-into-the-...

Walf 9y ago

What I want to know is why the 'trasher' was looking directly for so many different files. Could it not parse some output of finding/listing existing files to look for its targets?

luord 9y ago

This was a great read, the conclusion of always monitoring, no matter what technologies one is using, should be obvious, but I've noticed that it really isn't, unfortunately. Even I fall into the trap and sometimes I wish I knew about all this stuff but, alas, I prefer development.