17 ms·
My manager spent $1M on a backup server that I never used
- binarymax 4y agoNot sure if it’s just me but I get an invalid SSL cert warning and can’t load the page (DDG mobile browser)
- Aachen 4y agoWorks for me on a random old android browser (that wraps around standard webview afaik), using Android 10 or 11 not sure
- zem 4y agotheir manager paid for a valid SSL cert that they never used!
- jborean93 4y agoI get the same on Firefox on Linux (Fedora 37). It looks like the server is responding with an ECDHE key exchange using a SHA1 signature algorithm which is disabled on modern versions of OpenSSL. I'm not sure why it works on Chrome, maybe their TLS client is a bit more lax. You can see openssl fail to verify the signature locally with `openssl s_client -connect blog.dijit.sh:443` due to the wrong signature type. At least for me on Fedora, I believe Firefox uses the system policies and Fedora has set policies to block SHA1 signature algorithms https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2#Detailed_Description https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2....
- 28mm 4y agoI am not sure about that. On my system that openssl invocation fails because the site doesn't staple the intermediate Letsencrypt R3 certificate, and openssl doesn't retrieve it while browsers do. If their system is old enough perhaps they have a version of the ISRG X1 certificate that is cross-signed by the expired DST X3 certificate. Some ssl imlementations did poorly with this. https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ https://letsencrypt.org/docs/dst-root-ca-x3-expiration-septe...
- jborean93 4y agoThat could be it for yourself but I have a newer OpenSSL version (3.0.8) and the error does state the following > 006E552F8D7F0000:error:0A000172:SSL routines:tls12_check_peer_sigalg:wrong signature type:ssl/t1_lib.c:1592: The error from Firefox is SSL_ERROR_UNSUPPORTED_SIGNATURE_ALGORITHM which seems to align to what I'm seeing on the signature algorithm returned. Lastly by setting my system wide policy to allow the LEGACY (Fedora's term) algorithms the website starts to work on Firefox. It could very well be that the server is at fault here (based on what I've read this seems to be the case) and that's due to the Let's Encrypt cross signed certificate. But the reason it's failing to load on the client side is because some clients block SHA1 based signing algorithms and that's what the server is offering here.
- sbierwagen 4y agoWhat version of Android? Qualys shows a bunch of errors and an incorrect cert chain but it does manage to get a working connection on almost all test browsers: https://www.ssllabs.com/ssltest/analyze.html?d=blog.dijit.sh&s=2a02%3a2770%3a0%3a0%3a21a%3a4aff%3afebf%3a3acc https://www.ssllabs.com/ssltest/analyze.html?d=blog.dijit.sh...
- crote 4y agoSame issue here. My Firefox on Fedora gives "SSL_ERROR_UNSUPPORTED_SIGNATURE_ALGORITHM". I found a bug report[0] which suggest it is (mostly) a server issue. Fedora refuses to use SHA1, and the server ignores the client's attempt to negotiate a more secure algorithm. [0]: https://bugzilla.mozilla.org/show_bug.cgi?id=1749670 https://bugzilla.mozilla.org/show_bug.cgi?id=1749670
- techsurfer 4y agoWhere I work we have a 10K a year server budget but used to just use free Heroku instance
- labster 4y agoYeah, but this article is about keeping backups, and Heroku deletes data from paying customers without notifying them first.
- nickpeterson 4y agoHeroku is freeing customers of the burden of compliance with gdpr.
- worthless-trash 4y agoBless them.
- neves 4y ago[flagged]
- jldugger 4y agoWell, then you missed the story. It's more like a life insurance policy that kills you when you turn 67, and pays out when you turn ninety.
- Etheryte 4y agoHave you considered that there may be more to a four-page essay than a short title can convey?
- version_five 4y agoWhile theoretically true, it's so often not the case that I'm not surprised people just use the headline to infer what the article explains.
- vorpalhex 4y agoIt may be wiser to stop reading, plug your ears with wax, and hide in your room until death takes you. If you can't read for pleasure on the site you are reading for pleasure, something is wrong.
- version_five 4y agoAre you having a bad day?
- zeroimpl 4y agoYou likely made the same incorrect assumption from the title that I did. Replace “backup server” with “backup appliance” and it will be more clear.
- kubb 4y agoMy manager doesn't do almost anything. I wonder why they keep him around. He has like two reports and we're both fully self-directed. He can't understand technical issues, and whenever he proposes something, it's completely untenable due to his lack of understanding what's useful or possible.
- roncesvalles 4y agoHe's around so that your skip has to manage 1 person instead of 2.
- telotortium 4y agoAre they planning on hiring more reports?
- bitlad 4y agoAtleast your manager doesn't do anything. I have seen managers who do stuff and ruin the work that people have done.
- lordnacho 4y agoChesterton's Manager?
- yoyohello13 4y agoUnderrated comment. Many people believe their manager does nothing... until they become a manager and realize all the BS their manager protected them from on the daily.
- znpy 4y agoThe author makes a good progression in their reasoning but then somehow arrives to the wrong conclusion: > So when someone says that Amazon has invested a lot of money into security I think about the fact that Ubisoft spent $1M on a backup solution that didn’t work for the game that would have had the best use of it. The missing point here is thst ubisoft is/was spending money for something outside of its domain of expertise, and managed to make a suboptimal decision. Amazon however is spending money on two of its core domains of expertise: building and managing datacenters and building and operating web services. Not to mention, aws is very clear and explicit about what is their responsibility and what’s your responsibility (security of the cloud vs security in the cloud). Sad to see such a nice post ending like that.
- treis 4y agoThe whole post is clickbait. The million dollar solution was to store source code that they never ever wanted to lose. It (presumably) worked for that. What it didn't work for was backing up user profiles and being able to quickly restore them.
- dijit 4y agoThe environment that the datadomain was hosted in was not allowed to host source code or raw assets, but the bank vault thing was because of lost source code. The issue is that most games just fire off into the void if they need to backup and restores are never tested.
- dijit 4y agoI wrote the article in a fit of anger because someone (on hackernews) said that Amazon spent a lot of money on security therefore its better than whatever [you] could possibly need. The main thesis of the post is that sometimes incentives are not aligned and you still need to think critically about what you bought and what its intended purpose is, the cost of a thing does not mean it fits your need. You can pull a train with a Bugatti Veyron, but its not the right tool just because it costs as much or more than a train does.
- 4y ago
- unixhero 4y agoI would have done the same
- deleted 4y ago[deleted]
- JoeAltmaier 4y agoOur remote manager bought us an expensive HP floor-mount server like a dishwasher. It was over spec'd and underpowered. And we didn't need a server, not for work. My colleague loaded up his entire ripped CD collection and we got Bluetooth headphones and had it serve up tunes all day. Back when that was a thing. Only thing it ever did.
- whydoineedthis 4y agoin a way, he's genius. that probably boosted your overall productivity by 5% for a fraction of the cost.
- daneel_w 4y agoThe implied notion that the manager really had no idea what they actually wanted, or at the very least were unable to sensibly describe the requirements, is entirely believable, but from a systems development perspective nothing in this article makes any sense.
- anyfoo 4y agoCare to elaborate?
- jeremyjh 4y agoWell, spending 3 months proving that MySQL can reliably store data raised an eyebrow. I mean, we are talking about gamer profiles here. Worrying about whether or not the log is on the same volume as the table really seems like a bit much in that context. Even worrying about fsync is honestly questionable.
- dijit 4y agoIf you buy something and we grant it to you and your profile gets corrupted. We are sued. Better not risk it. Also Gamers are the walking definition of a hostile customer. Give them no quarter to call you incompetent. (also, we are talking enormous binary blobs, not easy to recover from if partially written- most databases don't store large rows in-line for performance reasons)
- anyfoo 4y agoSo "nothing makes sense from a system developers perspective" because gamer profiles are supposedly not valuable, and it's fine to have a system where they may get lost or corrupted, instead of trying to solve the problem with the premise that they may not? Putting aside that you seem to criticize that premise instead of the system development approach that fulfills the premise, I suspect that the resulting mess of support requests, manual intervention, and community unrest Ubisoft (or any other MMORPG developer) would find themselves in, if, say, "the shiny sword of atrocious doom" that players spent 240 hours sinking their life into obtaining were to disappear regularly, is probably not worth it. I understand this despite not doing any online gaming myself.
- jtchang 4y agoHow does one force exponential backoff of clients without taking in any load on themselves? Is it some sort of TCP window congestion control futzing? But even that wouldn't work since the client can do whatever the heck it wants.
- toast0 4y agoDrop incoming SYNs when the CPU hits thresholds. More specifically, every second the cpu is at or above the 'bad times' threshold, increase the probability that you'll drop a SYN; every second the cpu is at or below the 'good times' threshold, decrease the probability that you'll drop a SYN.
- dijit 4y agoHi, I’m the author of the article and I’m horrified by how many spelling mistakes I made. I must have written this in a rage. To answer your question, since we had control of the clients (since its a game) we used a proof of work challenge on TLS handshake which increased in complexity the more failed attempts you gave us. Very cheap on the server, very expensive on the client, which effectively rate limited connection attempts.
- jtchang 4y agoI see. But still after the initial TLS handshake. So clients can still connect over and over and force your server to do TLS handshakes all day long.
- worthless-trash 4y agoI dont think its possible to do it without "any load" The hardware grabbing the packet, putting it into the socket buffers, analysing if its valid, requires some work.
- Mave83 4y ago[flagged]
- m00x 4y agoLeast obvious ad
- anyfoo 4y ago> and begin real investigation you will quickly find that many databases that are popular are totally fine losing data. MongoDB being the most famous example that I can think off of the top of my head. Always pisses me off. University teaches the principles of ACID and how hard databases work to adhere to these principles, and then so called "NoSQL" comes along and says "lol we have eventual consistency".
- bombcar 4y agoThe heat death of the universe is eventually consistent.
- nickpeterson 4y agoLet’s hope our Kafka topics are caught up in time!
- bombcar 4y agoHeat death of the universe has been postponed until Jira can be updated.
- candiddevmike 4y agoI don't think that is what the vendors mean when they discuss enforcing anti-entropy.
- e40 4y agoIt is very frustrating that many of the NoSQL databases aren't ACID. Some even say they are, but they aren't. Frustrating to me because we have one that is, and it's a lot of work and harder to win benchmarks against the ones that fake it or aren't.
- whydoineedthis 4y agowhich one is that?
- walrus01 4y ago> Ubisoft had built an organisation optimised for treating developers like fools I think this whole write up really says more about how video game developers treat entry and mid-level career developers/programmers/graphic artists. Because there seems to be an inexhaustible supply of young idealistic naive persons who will take any salary offered and work for 80 hours a week, because they've been offered a job at a big name video game company. Film/VFX industry not much different.
- gambiting 4y ago>>Because there seems to be an inexhaustible supply of young idealistic naive persons who will take any salary offered and work for 80 hours a week As someone who also works at ubisoft(and I have actually worked with the author), I want to point out that for all the failings of Ubisoft, our work life culture is top notch and there is an incredibly strong focus on avoiding overtime**. It's drilled into our heads constantly that it's NOT normal to work more than the contracted 7.5h a day, if you as much as send someone an email at 8pm someone will talk to you to make sure you aren't working late(and will tell you to avoid doing that in the future, because it makes it look like reading and sending work emails at 8pm is a normal thing). In my 9 years here I have worked overtime only a very small handful of times, usually around launches of our project - for a week or two. I haven't logged more than my standard 37.5h/week on any project for literally years now. I manage juniors now and I wouldn't let them work more than that even if they wanted to. ** at least in the studio where I work and the studios I have interacted with - Ubisoft has 40(?) studios across the world and I cannot possibly comment on every single one of them. But it does certainly seem to be the company policy worldwide.
- wongarsu 4y agoWait, so Ubisoft spent $1M on a company wide backup system that apparently worked great, it's just that this sysadmin wanted incremental backups and the $1M system wasn't built to support that (blazingly fast ingest, slow online reads). So the sysadmin had to fight a lot to get dedicated hardware to do incremental backups, and eventually got it. I'm not sure Ubisoft is in the wrong at any point here? Everything else mentioned wants me to not work at Ubisoft, and roughly matches what I would have expected, just worse. The disregard for developers, the absurd NIH syndrome, etc. But I don't get the headline of the article
- PragmaticPulp 4y agoAgreed. This reads like the $1 million system did what the company needed it to do (safely archive code to prevent more loss of old games) but it didn’t do exactly what this developer wanted. There are various good points scattered in the article for the author’s specific use case, but it’s written as if the entire company was mistaken to not make this decision revolve around this one developer.
- dijit 4y agoHi, Author here. thanks for reading my poorly written rant! The issue was that nobody ever said what the system was designed for and equally nobody was open to the idea of doing things differently until things went pop. The main thesis of the article is supposed to be that it doesn't matter what something costs or how much money is invested if it doesn't solve your needs. I wrote it a long time ago in a fit of aggravation over someone on hackernews waxing poetic about how much amazon invested in their security- without much regard as to what it was actually spent on.
- ilyt 4y agoSorry but it just looks like you picked wrong solution from start and stuck with it. Vanilla pgsql backup + WAL shipping would just. work. with those constraints. Backup solutions that take time to read data and only "ramp up" once you start restoring full backup are nothing new, literally in industry for decades, before in form of tape libraries, now in form of amazon glacier and similar. Hell, backup solution where you can mount whole backup in directory is on the fancier side, because generally priorities are elsewhere
- cushychicken 4y agoI wish I had some clever, non-fraudulent ways to funnel wasteful corporate spending into things like my own salary or perks.
- madrox 4y agoIn my observation, the game industry and tech industry are like humans and apes: they have the same ancestor, but at some point in our ancient past we diverged. You rarely see crossover between the two these days, and I doubt we'll see any in another decade. What I think that means is there's probably a huge opportunity selling five year old tech ideas to the gaming industry
- deleted 4y ago[deleted]
- nitwit005 4y ago> Data Consistency as a Requirement Probably only sort of consistent, I imagine? Game servers tend not to support seamless failover. If the server crashes at the wrong moment, data is going to be lost, regardless of how politely behaved the data backend is. That is, if you kill a boss and successfully pick up an item, you know it'll be saved to DB. If the game crashes before you can pick it up, it's probably just gone.
- dijit 4y agothe way we had it working was local storage (sqlite) and remote storage (postgresql with a middleware). If you pick something up it flags as an “interesting event” and syncs your profile to local storage- on game server crash, there would be a crash collector which syncs the local database with the remote one and removes the lock - meaning you can join a working server. The crash collector also did things like… well, collect the crash dump and send it for debugging. But thats roughly how it worked.
- nitwit005 4y agoInteresting. Hadn't heard of a scheme like that before. Appreciate the explanation.
- ilyt 4y ago> Probably only sort of consistent, I imagine? Game servers tend not to support seamless failover. If the server crashes at the wrong moment, data is going to be lost, regardless of how politely behaved the data backend is. You could just have a let's say "user profile service" where all of the transactions about user profile (items, XP, etc.) go, on top of internal game server data. That way the important stuff could be send immediately, like "epic or above item drops", and everything else either in batched update (XP, achievements etc.) or periodically (stuff like player rearranging inventory) or at end of the session.
- ilyt 4y ago> The games industry is weird: It simultaneously lags behind the rest of the tech industry by half-a-decade in some areas and yet it can be years ahead in others. I'd love to see examples of "ahead" coz I literally never saw it. Game dev studios act like automated testing is a new thing... > PostgresSQL performed much better and had the additional benefit of being able to cleanly split write-ahead logs (which are largely sequential) and data to separate RAID devices. Something that MySQL doesn’t really support and would have to be hacked in using Symlinks on every table create. It's funny that you can guess that it this happened good few years ago purely because the solution to above wasn't just "slap a NVMe on it" >I tested this and ordered the storage I would need to have a rolling 90 day window of backups (with older backups being taken off-site).. >The hardware request was rejected. > When I inquired as to why, I was told that Ubisoft has a standard backup solution which is replicated globally and sent to cold storage in a bank vault somewhere in Paris. I was told this is because we had lost some source code once upon a time and we could no longer build certain games because of that. My first thought was "why not just ship pg backups + wal logs there" and just a full backup from time to time. No read-back needed. The whole problem seems to be "the admin overengineered solution without knowing the constraint, then tried to headbutt the constraints instead of changing approach > Our EMC DataDomain system was optimised primarily for ingesting huge volumes of traffic, but if we want incremental backups then perhaps we needed a something a little more dynamic. Nope, your incremental approach sucked. Majority of software that I've seen used to do incremental backups could generate incrementals on its own, without any readback from server. The minority *was software that wholly managed backups*, so it was prepared for that, and often it was just "read metadata from database" instead of actual data. > I don’t know what else to take away from this. That people use tapes for backups so ? To get your requirements before implementing solution ? To find better solution when requirements are known ? To not get stuck in initial idea. Literally the basic, recommended way of doing PostgreSQL backups with no extra software would work.
- dijit 4y agoThe “years ahead” was described in the article. We had what amounts to a very well oiled kubernetes installation with mTLS, but on Windows, in C++ and 10 years ago (before Kubernetes was a thing). Everything else you say is true, I could have chosen another solution for backups but the tradeoff between backup speed (replaying WAL can be time consuming) vs database load (full backups delay replication to replicas) was made before I knew the read/write characteristics of the datadomain. (I didn't even know it was a datadomain- nobody told me anything except giving me an NFS mount point until I started having problems, at which point I spent weeks debugging with a storage engineer from Montreal- this article is a 6-8month span of back and forth distilled). Regardless: restore times are important, as the wise sages of our industry once said; you don't have a backup until it is tested in a restore - which is what this system was effectively doing. How would you test without reading?
- Severian 4y agoI can comment directly on this as I work in the backup sector. First hand knowledge, yada yada yada. Dell EMC DataDomains do have good ingest performance, you can typically throw hundreds of streams at them and they'll greedily gulp it down. And it is true that they are dog slow at restoring data. The reason? They are deduplicating appliances, you have to rehydrate the data, and this can take a very long time depending on the number of blocks needed that constitute the backup. Some blocks may be shared between hundreds if not thousands of backups. Dell isn't one to talk about their downsides much. They are best used not as _primary_ storage, but secondary (think 3-2-1 backup rule). You should have some fast nearline storage available for recent backups that require really low RTO (recovery time objective). Depending on the application used for backup, it may not have had native Postgresql WAL processing. This and may have had to be image based which slows down the process and requires some additional scripting.
- dijit 4y agoThats pretty cool, the datadomain appliance is actually really cool and I definitely appreciate its existence. I definitely feel like we were using it wrong and I’m not entirely convinced it was my fault. In my ideal scenario I would have had a weeks worth of point in time backups on a machine in the rack, and replicated the contents to the DD after verification. Sadly I was denied that “in-rack” solution and was sold the datadomain solution (without it being named) as if it was simply a remote disk, not a fancy appliance. the main point I tried (and failed) to convey in the article is that a solution can be brilliant and expensive but that doesn't mean its what you meed for the job at hand.
- Severian 4y agoOh yeah, for sure. I got the gist of your article. I thought it was pretty clear. But overall, planning a backup strategy can take some time. You need to know your goals before you go buy some vendor's hardware. But, in your circumstance, you gotta do what you gotta do.
- rodgerd 4y ago> They are best used not as _primary_ storage, but secondary (think 3-2-1 backup rule). Yeah, these are for archives, not service restoration/disaster recovery scenarios.
- TheRealPomax 4y agoGood old "we spent $1m on it" fallacies. "Exactly, we spent a million on a backup solution that's objectively bad, and we spent 100 times that on the actual code we're trying to secure. So: do we want to have wasted a trivial $1m by redoing the backup part, or do we want to have wasted $100m because of a terrible pretend-backup solution? Because this should be a business no-brainer".
- deleted 4y ago[deleted]
- guidedlight 4y agoI worked for a university that spent about a million dollars on data storage for a research project. They estimated that they required about a petabyte of storage, so a big expensive storage array was purchased. In the end the project used just a few terabytes of storage because they miscalculated their requirements. It was a bit of a running joke that they spent a million dollars, when a $150 SSD would have been fine. They ended up repurposing the storage array (which is when I got involved). Luckily everyone has now moved to cloud, and life is better.
- neilv 4y agoThis could be one of those big-corporate situations: 1. Someone pays a lot of money for something. 2. It turns out to be the wrong thing (either for new needs, or because they didn't consult or listen to the people who could've told them that in time). 3. Someone wants to avoid the political backlash of admitting they bought the wrong thing. 4. The org incurs much more costs than actually fixing the situation would cost, as people have to use the wrong thing and/or can't use the right thing. (Costs from lost productivity, lower quality/uptime, damaged morale, etc.) (But what I really want to know is... Why did "The Division 2" not build upon "The Division", but instead make you start a new character, and then be more of a selling-brightly-colored hats game? Also, TD1's gritty survival mode was the most compelling gameplay of the franchise, IMHO.)
- MikePlacid 4y ago> This could be one of those big-corporate situations Nothing specifically “big” here. Can happen in a corporation of any size. We were not very big, when we bought Siebel software…
- dijit 4y ago> Why did "The Division 2" not build upon "The Division", Four main reasons, you can argue if they are "that big of a deal" but ultimately they were. Please also remember that Division 2 was made in 2 years, and Div 1 was made in 7. 1) Clean slate of the architecture built on google cloud meant that connecting in to the datacenters in Montreal would have been very time consuming to get right. 2) The Division 1 continued, so a migration would have had to have some pretty clearly defined semantics, do we sync both the backends constantly? Do we fork your character? Do we stop your previous account from working. Forking sounds like the right thing until you realise you can't fork again. :\ 3) We would have to carry over every single item from the Division 1, most of which do not make sense in the setting. 4) We'd have to have a clear upgrade path for your binary save game (from a moving target as Div1 was still getting updates!) But it was a topic that was discussed at great length. > Also, TD1's gritty survival mode was the most compelling gameplay of the franchise Yeah, I really liked that, I wish we could have released it as a standalone game, especially as it came out in the height of BR and for me it hit similar themes.
- TuringNYC 4y agoMuch like an Insurance policy, it isnt a bad thing if you dont use it.
- waqasy 4y agoWhy spend $1M instead of getting managed solution. Very painful.
- thumbsup-_- 4y ago>all of this, written by hand in C++, nothing off the shelf, very minimal dependencies (OpenSSL being the only one of note), everything running on Windows and completely bespoke Great engineering but this is the time you could have spent building your core competency. re-inventing the wheel is a good recipe for failure
- tinco 4y agoIn 2014 Kubernetes didn't exist and the alternatives all had downsides. I was in charge of a non trivial deployment then and I also did extensive research and decided to build something bespoke on top of docker. There's a reason Google had Borg and all the other big companies had their internal systems. Those were not reinventing the wheel, they were just building wheels round enough to hit the highway. By the time our product launched Kubernetes had come out and we could migrate to that eventually.
- jiggawatts 4y agoThis rant starts of pretty badly, with "Windows bad, Unix good, Windows especially bad because it doesn't have Unix tools". Reverse that statement. Is Unix bad because it doesn't Windows tools? > but simultaneously there was nothing to lean back on: no shell, no Unix tools like sed/awk, no SSH. I He joined in 2014, when Windows had a superior shell and remoting system that eliminates the requirement for low-level string parsing tools like sed/awk entirely! PowerShell + WinRM + Desired State Configuration (DSC) was all pretty mature at the time, and I had used these technologies to manage huge fleets of servers solo without issues. Then, almost none of this heroic database backup efforts would have been required if they had just a commercial database product, e.g.: one designed for Windows such as SQL Server. In 2014 it supported AlwaysOn Availability groups that allowed multiple synchronous and asynchronous replicas! It also has had true online backups since before 1997. His second mistake is that backups can be incremental, but a restore in a (true) disaster is a full restore by definition! Differential + Log backups are great to capture data regularly during the day, but the business requirement is usually that a full restore must complete in a reasonable time. Typical commercial databases plus the product he mentioned could easily backup and restore tens of terabytes in setups I've seen directly to or from any proper database engine without having to through "NFS" on the way. Typically you'd use a native backup agent. This whole article sounds like a self-important Unix engineer refusing to touch commercial products, lack of understanding of business requirements, and an allergy to Windows.
- sgt 4y agoWhat are you talking about - Windows was always terrible for an admin. Unix was and is vastly superior when it comes to remote admin. Having used both (Powershell is powerful yes, but it's nowhere as fluid as a Unix shell).
- radicalbyte 4y agoIt depends on exactly what and how you're doing it. I have as devops managed both a small fleet of windows boxes (~20 machines) an a similar amount of Unix (Linx-based) machines. Remote management of Windows was initially a pain to set up but once it was running it was fantastic. Being able to manage the entire application layer with properly tested, versioned code written in C# is an absolute killer. Things like debugging/logging etc are also "just there", and SQL Server is a fantastic database. The Linux alternative on the other hand was easier to update and had some benefits to the developers (you can sort-of recreate a production-like environment in containers locally). The management after that was a huge pain because everything seems to have been built McGuyver style. I like Postgres but prefer using SQL Server (probably personal preference at this point tbh). Things are different nowadays: systems like Kubernates apparently simplify the Linux side and make it affectively trivial to deploy and do cool stuff such as dynamically scale systems. It's also more efficient as you can remove the overhead of the Virtual Machines. It doesn't take much imagination to see that that would make a Unix-based system better.
- shoo 4y agoThis kind of thing seems to happen in very large organisations that have accreted many production IT systems, especially once an org realises it has half-a-dozen different services in production that are performing an apparently equivalent job. What an opportunity for architecture simplification! Why run half a dozen different variants of apparently the same thing when we could consolidate on one standard approach, and reap a bunch of cost savings and reductions in operational complexity. Enterprise architecture can commission a project to review options and select the one true enterprise backup solution. After sufficient peer review / diligent testing / bribery from vendors, an enterprise backup solution is chosen, wrangled into production, and then inflicted upon the org. Let's be pragmatic. For existing production systems, it may not make sense to migrate them over to the new enterprise backup solution. Much risk and cost, limited payoff. Perhaps it's simplest to just keep them running their existing legacy backup solution until the entire production system that depends on it is finally decommissioned, which is planned for only 3 years away. We'll definitely decommission it after 3 years. But for any new teams trying to deliver a new service into production. Are you storing data? Well, you must have a backup strategy before you can go to prod, and further, your backup design must align with the enterprise backup strategy -- you must integrate with the enterprise backup solution for your design to be stamped approved by architecture, unless there is an exception granted by appeal and/or ritual sacrifice to your manager's manager's manager's manager's manager.
- hallqv 4y agoPeanuts! Magic Leap spent $3.5 bn on a product no one used
- tagyro 4y agoReading this, one might think that this happens only at X or Y company, but in my experience, this is a lot more common. One example: at a previous job, we spent $1M and almost 1 year to build a (new version of a) process (as in, a website) that served less than 50 customers. The project was launched to great fanfare internally (talks, videos, the whole package) but created no value, even those 50 customers didn't transition to it, as the old one worked fine. The same job where my manager wrote in my review that I'm too negativist and only point out problems. I have so many of these stories, some funny, others more tragic - companies going under and people loosing their job and going into depression because a manager was stubborn and incompetent - that I think I should write a book, but then I realise I'm not that special and my stories are pretty boring.
- deleted 4y ago[deleted]