chronid

9 ms·

31.
by chronid 2y ago
You arent' forced to use service mesh and complex secrets management schemes. If you add them to the cluster is because you value what they offer you. It's the same thing as kubernetes itself - I'm not sure what people are co
32.
by chronid 2y ago
The k/v store offers primitives to make that happen, but for non-critical controllers you don't want to deal with things like that they can go down and will be restarted (locally by kubelet/containerd) or rescheduled. Whateve
33.
by chronid 2y ago
I am fairly simplifying, but you are expected to know your direct dependencies (and normally wil), pagers have embedded escalation rules with prinaries and secondaries, etc. The tooling once you know what to do is better than anything outsi
34.
by chronid 2y ago
Some FAANGs at least (though they may not cover everything) have a "help something is broken but I don't know what to do" team and/or rotation for incident response, staffed on multiple continents to "follow the sun
35.
by chronid 2y ago
This has been done forever. Ops team had cronjobs to restart misbehaving applications out of business hours since before I started working. In a previous job, the solution for disks being full on a VM on-prem (no, not databases) was an auto
36.
by chronid 2y ago
You should look where the economy is growing and where the salaries are growing. It's not uniform at all. The entire situation (as an EU country citizen who moved to another EU country) and the narratives around it are funny to me beca
37.
by chronid 2y ago
Even software (at least outside academia) eventually has to fight physics and the thing with most gravity of it all, money.
38.
by chronid 2y ago
Doesn't a lot of optimization target making your code and data more cache friendly because memory latency (not bandwidth?) kills performance absolutely (between other things like port usage I guess)? If something is in L3 it is better
39.
by chronid 2y ago
The badging system will notice someone has not left. This should have made security panic a fair bit on Friday evening...
40.
by chronid 2y ago
The "hate" was not about the philosophy of linux. It was about the udev/systemd maintainers writing "fuck you Gentoo developers" in a mailing list when they wanted udev to support only systemd, actually that udev sh
41.
by chronid 2y ago
I'm not trying to defend CrowdStrike, but pointing to the fact Delta is the one maintaining and owning critical infrastructure and the executives trying to shift this responsibility onto someone else is the reason this happened in the
42.
by chronid 2y ago
Did crowdstrike force delta to accept running what essentially is a permanent RCE in their production fleet? You do not buy a software that is capable to do that and you put the fact it's not capable of doing that in the contract. Th
43.
by chronid 2y ago
AFAIK crowdstrike can push updates at any time at any host. There are staging areas they may use, but don't have to (particularly for definitions updates). Crowdstrike should have done a better job , but Delta chose them (to offload t
44.
by chronid 2y ago
Crowdstrike is not handling critical infrastructure. Delta is. The reality is the industry wants its cake and eat it too. No one forced Delta to buy a software which could force upgrades in their production fleet. They're a billion dol
45.
by chronid 2y ago
This is not really correct, and assumes the state of a cloud service (let's say load balancing) is binary. In my experience it's not. The cloud will glitch. The load balancing algo will break subtly for your workload. Your traff
46.
by chronid 2y ago
Sorry, this is untrue. Enterprises have SOCs and oncalls, if there is a high risk they can do at least minimal testing (which would have found this issue as it has a 100% bsod rate) and then fleet rollout. It would have been rolled out by F
47.
by chronid 2y ago
> I am sympathetic to that, but its only possible if both policy and staffing allow. We are not talking about small companies here. We're talking about massive billion revenue enterprises with enormous IT teams and in some cases mul
48.
by chronid 2y ago
Windows updates are also cybersecurity, but the customer has (had?) a choice to how to roll those out (with Intune nowadays?). The customer should decide when to update, they own the fleet not the vendor! You do not know if a content update
49.
by chronid 2y ago
I wrote this in another thread already, but the fuck up was both at crowdstrike (they borked a release) but also and more importantly their customers. Shit happens even with the best testing in the world. You do not deploy anything , ev
50.
by chronid 2y ago
I also blame the customers here to be completely honest. The fact the software does not allow for progressive rollout of a version in your own fleet should be an instantaneous "pass". It's unacceptable for a vendor to decide
51.
by chronid 2y ago
Centralized management is very useful, just a random delay is not enough. One of the (big) companies I worked with had jury rigged something with chef I believe to show different machines different "repositories" and roll things o
52.
by chronid 2y ago
What I find definitely depressing is the fact we used to roll out progressively even OS upgrades (I guess now that is done through intune?) and was one point in favor of windows (on Linux you had to do things yourself at the time AFAIK, I d
53.
by chronid 2y ago
This is not my experience, working in small shops/enterprise companies (some regulated). What I've seen is a constant, hard resistance from security "departments" to do anything that is not making policies (one company I
54.
by chronid 2y ago
Increased efficiency makes systems more fragile though, so minmaxing isn't actually a good outcome. Long term at least. Next quarter we may all get a bonus, unless a rival AI exploits our fragility...
55.
by chronid 2y ago
If they manage it in my company at least of course, the real question is if they want to manage it (and know what that means). Usually they don't, in my experience.
56.
by chronid 2y ago
Those people will actually be grateful they don't have to deal and spend weeks to debug yet another cloud provider hidden gotcha or bug and convince support they are right. They will now very happy as they can now deliver value (at rel
57.
by chronid 3y ago
Salt is unfortunately (in my two year old experience) full of features and strange bugs related to them - I do remember encountering a bunch of memory leaks master side for example. Deployment had the classic issues of python applications.
58.
by chronid 3y ago
I have worked on and around systems with an order of magnitude more data and a single node failing did not matter. We weren't using btrfs anyway (for data drives) and it definitely was not cheap. But storage never is. But again, most s
59.
by chronid 3y ago
Machines die. Hardware has bugs, or is broken. Things just bork. It's a fact of life. Would I build a file storage system around btrfs? No - without proper redundancy at least. But I'm told at least Synology does. I'm pretty
60.
by chronid 3y ago
Yes, that would be great, but unfortunately there are application teams (particularly in the enterprise) lacking such tact when blaming infrastructure for issues. Good old silos are alive and well, and ownership is not always part of the cu

More ›