8 ms·

Oracle engineers caused five days software outage at U.S. hospitals

174 points by jnord 1y ago

buyucu 1y ago
I don't understand why anyone buys Oracle. Every Oracle product I used over the last 15 years has been awful.
- jackvalentine 1y ago
  Nobody bought Oracle here - they bought Cerner which was gobbled up in 2022.
  - chickenzzzzu 1y ago
    Disclaimer: I am not inciting violence. This is polemic humor for entertainment purposes only. Void where prohibited. Theory: A society collectivey buys Oracle when they no longer view armed revolution as acceptable.
  - Scoundreller 1y ago
    Sorta, Cerner uses an Oracle backend
- selivanovp 1y ago
  Because Oracle comes to any country/industry with trucks of money to corrupt officials and lobby their adoption. And after a few years you find yourself in a situation when you already paid for Oracle so much, integrated it so deeply, so switching to any alternative is a massive pain and in most cases it’s safer and easier to keep paying Oracle.
  - goodpoint 1y ago
    They also corrupt management in the companies buying their products. Same with two other very big software vendor.
  - netdevphoenix 1y ago
    I believe this is how they grew and how they remain big. While smaller companies aim for managers, Oracle targets CTOs and CEOs, takes them out for expensive dinners and promises them the world. Then, they legacy handcuffed them forever. And virtually no one chooses to invest crazy amounts of money on migrating or even starting a new codebase when your company lifeblood has been poured into the Faustian Oracle machinery.
    - 1718627440 1y ago
      Are managers really not doing background checks about company behaviour before signing a contract?
      - netdevphoenix 1y ago
        The ones targeted by Oracle salespeople don't. And when they do, it's too late. And most just hope things "will work out". I was third party witness to one of these transformations and I correctly predicted the chaos, uncertainty and sense of powerlessness that it would cause
- gonzo41 1y ago
  It's legacy lock in. You can't just switch stuff. it's incredibly hard to move logic out of store procedures from on DB to another. Its the same reason why the US government runs on Cobol. People don't think about the strategic implication of the tech they choose and how it may come back to limit them in the future.
  - 8fingerlouie 1y ago
    Most Oracle "shops" i know have used it for decades. When they started using it, there weren't many options, so Oracle was what was used. COBOL is in the same category. When invented, it was the absolute easiest programming language to learn and use, so of course it gained popularity. It then turned out to be rather good at what it did, along with running on hardware that "never fails", so most places didn't even think about replacing it until sometime in the 90s. Also keep in mind that the reason companies are migrating away from COBOL is not due to the language as much as due to young people not taking an interest in COBOL and Mainframes, making it hard to recruit new talent. Even then, a migration away from a typical mainframe is a huge task. In most cases you're talking 50-60 years of "legacy" (still running, still working, still updated) code, and replacing a system that has evolved for half a century is not an easy undertaking, at least not if you plan on getting it 100% right the first time.
    - pastage 1y ago
      > due to young people not taking an interest in COBOL and Mainframes Having only fleeting professional experience with COBOL during a summer my view of it is that it is a mix of dataanalyst job and programming. Where the programming is horrible and the report making is ok though archaic. As long as you modify processes already available it is not so bad, but the developer experience was horrible. With all that said I actually liked ideas in COBOL but it is an extremely niche language that does not serve you at all in the real world.
      - decimalenough 1y ago
        The "real world" of airline reservations, the world's financial infrastructure and basically anything involving hardcore real data processing runs on COBOL. But yeah, if you're looking to code up a progressive web app or next blockbuster MMORPG, I wouldn't recommend COBOL.
- vlovich123 1y ago
  I know they get a lot of hate but I personally used Oracle’s cloud and it seemed like a decent piece of infrastructure with really solid engineering team behind it. They had their share of problems (and their first line of support is really really bad and not well suited to handle real issues) but not really any different from any other similar products.
- jeroenhd 1y ago
  Oracle offers everything. Databases, web frameworks, programming languages, CRMs, cloud management tools, auto scaling clusters, identity management, BI, document search, email servers, colocation, mainframes, calendar sync. You name it, they've got it. Oracle has products ready to go to run an entire country if it needs to. Sure, Postgres beats OracleDB, but Postgres doesn't integrate as well with Oracle Fusion and you need to migrate the code yourself. They're like SAP: they're big enough that you can make a career just out of configuring their software packages and make good money while doing so. It's expensive and certainly not the best, but it's reasonably stable and has a huge company backing it. Oracle won't disappear any time soon and they're not as likely as Google or Microsoft to shut down their services within a few years notice. In some countries, Oracle is also very good at doing what Google and Microsoft are doing to students. The Brazilian programmers I've spoken to specifically learned OracleDB when they were taught relational databases. They learned to program in Java, and I'm sure Oracle also sponsored other parts of the professional tooling they got to use for free. Microsoft, on the other hand, didn't seem to generous towards their educational facility (no free MS tooling for their schools like they offer over here). If all you know is Photoshop/Windows/Maya/OracleDB/iOS, you're going to look for jobs where you can use Photoshop/Windows/Maya/OracleDB/iOS, and employers looking for cheap juniors will need to offer Photoshop/Windows/Maya/OracleDB/iOS to make the best use of them.
  - arresin 1y ago
    Damn I’m wondering now if that’s why we studied Java as our first programming language in Australia at uni for CS. Oracle have a relatively big presence here and there’s a comfortable “mates” system that runs the Australia (soft bribery).
    - chickenzzzzu 1y ago
      Java has been a staple of university courses since before Oracle bought Sun. I guess I am getting old now, if the kids all want to program in Python and Javascript for 4 years...
    - jeroenhd 1y ago
      As much as I hate Oracle, there are huge advantages to Java. Java is strongly typed, free, and runs damn near anywhere. It's pretty much the go-to language when people think about OOP and for good reason. It has some excellent open source IDE support and it's widely used in the industry. With the current OpenJDK setup, it's also free of Oracle's licensing issues. It runs fast and can support most algorithms you'll probably ever study, without the manual memory management troubles of something like C++. I was taught C# in uni for very similar reasons except the entire uni ran on Windows and the Microsoft platform, which made doing assignments on Linux rather inconvenient. With the status of dotnet core, I'd say Java finally has a good competitor when it comes to teaching OOP languages.
- ManBeardPc 1y ago
  I made a similar experience. The database just stopped working one day, triggered by a normal restart without any updates or patches. Fixed the configuration, tested it, restarted multiple times, everything works. After a week it was not working again, without me being able to fix it except doing a fresh reinstall on a clean VM. Luckily more and more customers switched to Postgres and I no longer have to deal with it.
wiseowise 1y ago
“Engineers” Must be engineers who write requirements and unrealistic deadlines that lead to such issues.
- devoutsalsa 1y ago
  I just watched a YC video about vibe coding. He says he used Claude Sonnet 3.7 to configure his DNS servers & Heroku hosting. What could possibly go wrong?! "How To Get The Most Out Of Vibe Coding | Startup School " => https://www.youtube.com/watch?v=BJjsfNO5JTo&t=494s https://www.youtube.com/watch?v=BJjsfNO5JTo&t=494s I like his advice on downloading relevant documentation and putting it in the code base. That makes sense to me for targeted use cases.
  - Ygg2 1y ago
    How it actually went: https://youtu.be/_2C2CNmK7dQ?feature=shared https://youtu.be/_2C2CNmK7dQ?feature=shared
    - devoutsalsa 1y ago
      I didn't need a header anyway.
  - BiteCode_dev 1y ago
    I managed to break production before vibe coding was cool. 20 years old me had rm -fr root access to way too many systems. I don't think it's much different today. If anything, I think the youngsters will learn faster from their mistakes because they already have good mentor for the easy stuff and will get grinded on the hard stuff sooner.
    - devoutsalsa 1y ago
      Maybe. One thing I've found is the LLMs give me a lot of short term leverage. I can (get an LLM to) program in a language I know almost nothing about. Then I hit a wall where I don't know enough to get the LLM to fix something I don't know how to ask for, and then I'm stuck. When vibe coding, I'm not learning how to solve problems myself so much, and that means anything the LLM can't do, I also can't do. When I'm coding on my own, I'm picking up lessons along the way the help me build a foundation for incrementally harder projects.
amelius 1y ago
Doesn't matter. They have all the good lawyers.
xupybd 1y ago
I hate working on production systems and I hate any action that can't be easily reversed like deleting stuff.
- cheema33 1y ago
  Yeah, I like working under ideal conditions too. And I also want a toilet made of solid gold.
  - keyle 1y ago
    You sound sarcastic. It's not too much to ask to have all actions reversible in 2021.
  - arandomhuman 1y ago
    Yeah database backups are just too hard for a multi billion dollar corporation to handle.
  - aeonik 1y ago
    Why solid gold? It would be way too ductile, heavy, and conduct far too much heat. Diamond might actually be better: low surface energy means a low coefficient of friction, so it would be much easier to clean. It would still suck the heat right out of your cheeks, though. Realistically, porcelain or other ceramics are probably the ideal material.
    - alwa 1y ago
      Or plastic, backed by electric heating! https://www.totousa.com/washlet https://www.totousa.com/washlet
  - alwa 1y ago
    As I recall, the Guggenheim offered theirs for loan to the White House a few years back… https://www.guggenheim.org/exhibition/maurizio-cattelan-america https://www.guggenheim.org/exhibition/maurizio-cattelan-amer...
surfingdino 1y ago
I'd risk to say it was compounding interest on tech debt, but it doesn't sound as sexy, does it?
- mrweasel 1y ago
  > engineers conducting maintenance work mistakenly deleted critical storage connected to a key database I'd say poor process management. Why is an engineer even deleting critical storage (I take that to mean that they are deleting something of a file-system). Perhaps they where dropping a database, but you wouldn't do it like that in a critical environment. You'd disable the database access first, and then after some time, weeks, you'd drop the database, after doing a final backup. It could also be disconnecting a SAN, deleting a storage pool, something like that, but your process should say: Check of read activity, off-line the storage, anything non-destructive, and the only later, once you've verified that everything runs without this resource, do you delete. At previous jobs I've worked with healthcare system. You have processes, you follow those proesses to the letter and you never delete anything as your first step. Deleting is almost always done by going into read-only mode and ageing out the data. The fact that recovery time is four days tells me that no one followed a single procedure. Because there should be a written step by step plan, including recovery and risk assessment and when the change manager sees: "Recovery time five days" they will ask questions and stop you. This is probably the only answer less sexy than technical debt.
  - pferde 1y ago
    This should be the top comment.
cheema33 1y ago
Oracle has a horrible reputation among devs. But I think they bypass devs for purchase decisions and straight up wine and dine and bamboozle the low information execs. When I worked at a megacorp as a dev, I had near zero say in such purchase decisions. I had to work with what I was given. Thankfully I work for a much smaller shop now. Better pay and much better decision autonomy.
- reactordev 1y ago
  This is exactly right - they say the right things, to the right people, over the right sized bribe rib.
- forinti 1y ago
  Devs get Oracle's best side. They don't know how lucky they are. PL/SQL is a fine language and Oracle throws everything and the kitchen sink into the APIs available inside the database. As a sysadmin/dba I get to handle the nasty side of Oracle: the bugs, the patches that fail, the redundant tools, the wordy documentation that always feels like a never-ending advert.
  - vkazanov 1y ago
    When I was young and beautiful and naive I enjoyed playing with setting things up, even though my main focus was on backend dev. Like, setup postgres, tweak settings to performance, that kind of thing. So I was the first one to get the "figure out on site installation steps" task to share with client admin. Our small company was building enterprises things for large clients. And one of them wanted Oracle on Windows server. They also wanted a failover setup. How hard could that be? Now I hate Oracle. I hate Oracle consultants. I despise the ignorance. My favourite bit was Oracle uninstall procedure. I had like 4 pages printed just for this case. Oracle bad. Postgres good.
  - hahn-kev 1y ago
    The problem is that putting code into your database has all the problems of data migration with no testability. It's really hard to write tests and debug code compared to a proper language.
Pinus 1y ago
Is this the same Oracle/Cerner system ("Millennium", I believe) that, despite protests from the medical staff, was deployed "big-bang style" in the Swedish region of Västra Götaland, with much the same results? (And where during a press conference, where the management was explaining how nearly everything was going to plan, a doctor, who had somehow sneaked in, got up, shouted something to the effect of "you’re lying, it’s a bloody disaster!" and stormed out.) (That was not an outage, though — as far as I understand the system was working, it just didn’t actually work...)
- harvey9 1y ago
  Could be, but oracle only bought cerner a few years ago and millennium has been around much longer. Other EMR systems are also dreadful - my spouse has some stories about EPIC for example.
- shigawire 1y ago
  Swedish privacy laws for EHRs are quite different compared to other countries. So I'm not surprised any software not purpose built for for that market would fail. And we know Oracle is not investing in R&D for Millennium since all efforts are out into their forthcoming AI based EHR - whatever that actually means....
- deleted 1y ago
  [deleted]
phaedrus441 1y ago
Can't wait for the overpriced, late, outdated Oracle to get deployed at the VA! We get to go from one bad EHR to another...
netdevphoenix 1y ago
Title should be "Oracle execs caused five days software outage at US hospitals". If the systems helped save lots of life, you bet the engineers wouldn't be thanked for that and it probably wouldn't even be a news article
- nojito 1y ago
  The engineers are the one's who deleted the db though.
djoldman 1y ago
Is it possible to fix EHR without drastic changes in the health/pharma/hospital/medicine regulation environment?
exabrial 1y ago
As a previous Cerner employee, this is actually pretty good.
DontchaKnowit 1y ago
Used to work on CHS systems for Cerner. If I am not mistaken they were a "communityworks" client, which meant their databases were shared with a number of other clients - a "multitenent" envirnment, we called it. Completely bumblefucked design. Not surprised somthing like this happened. Also - cerner software in general allowed hospitals to freely completely fuck up their own architecture. Sometimes ireperably. If anyone has details about how this happened Id love to hear.
sea_boat 1y ago
[dead]
johnwatson11218 1y ago
I think the Oracle Transaction Manager is one of the best pieces of software that I had to work with in a professional settings. Lots of other stuff in an enterprise setting is very flaky and follows trends but the Oracle internals seem very nice.
torgen-ai 1y ago
My company (torgen.ai) is looking for senior Oracle devs to help customers re-platform.

174 points by jnord 1y ago

buyucu 1y ago

I don't understand why anyone buys Oracle. Every Oracle product I used over the last 15 years has been awful.

jackvalentine 1y ago
Nobody bought Oracle here - they bought Cerner which was gobbled up in 2022.
- chickenzzzzu 1y ago
  Disclaimer: I am not inciting violence. This is polemic humor for entertainment purposes only. Void where prohibited. Theory: A society collectivey buys Oracle when they no longer view armed revolution as acceptable.
- Scoundreller 1y ago
  Sorta, Cerner uses an Oracle backend
selivanovp 1y ago
Because Oracle comes to any country/industry with trucks of money to corrupt officials and lobby their adoption. And after a few years you find yourself in a situation when you already paid for Oracle so much, integrated it so deeply, so switching to any alternative is a massive pain and in most cases it’s safer and easier to keep paying Oracle.
- goodpoint 1y ago
  They also corrupt management in the companies buying their products. Same with two other very big software vendor.
- netdevphoenix 1y ago
  I believe this is how they grew and how they remain big. While smaller companies aim for managers, Oracle targets CTOs and CEOs, takes them out for expensive dinners and promises them the world. Then, they legacy handcuffed them forever. And virtually no one chooses to invest crazy amounts of money on migrating or even starting a new codebase when your company lifeblood has been poured into the Faustian Oracle machinery.
  - 1718627440 1y ago
    Are managers really not doing background checks about company behaviour before signing a contract?
    - netdevphoenix 1y ago
      The ones targeted by Oracle salespeople don't. And when they do, it's too late. And most just hope things "will work out". I was third party witness to one of these transformations and I correctly predicted the chaos, uncertainty and sense of powerlessness that it would cause
gonzo41 1y ago
It's legacy lock in. You can't just switch stuff. it's incredibly hard to move logic out of store procedures from on DB to another. Its the same reason why the US government runs on Cobol. People don't think about the strategic implication of the tech they choose and how it may come back to limit them in the future.
- 8fingerlouie 1y ago
  Most Oracle "shops" i know have used it for decades. When they started using it, there weren't many options, so Oracle was what was used. COBOL is in the same category. When invented, it was the absolute easiest programming language to learn and use, so of course it gained popularity. It then turned out to be rather good at what it did, along with running on hardware that "never fails", so most places didn't even think about replacing it until sometime in the 90s. Also keep in mind that the reason companies are migrating away from COBOL is not due to the language as much as due to young people not taking an interest in COBOL and Mainframes, making it hard to recruit new talent. Even then, a migration away from a typical mainframe is a huge task. In most cases you're talking 50-60 years of "legacy" (still running, still working, still updated) code, and replacing a system that has evolved for half a century is not an easy undertaking, at least not if you plan on getting it 100% right the first time.
  - pastage 1y ago
    > due to young people not taking an interest in COBOL and Mainframes Having only fleeting professional experience with COBOL during a summer my view of it is that it is a mix of dataanalyst job and programming. Where the programming is horrible and the report making is ok though archaic. As long as you modify processes already available it is not so bad, but the developer experience was horrible. With all that said I actually liked ideas in COBOL but it is an extremely niche language that does not serve you at all in the real world.
    - decimalenough 1y ago
      The "real world" of airline reservations, the world's financial infrastructure and basically anything involving hardcore real data processing runs on COBOL. But yeah, if you're looking to code up a progressive web app or next blockbuster MMORPG, I wouldn't recommend COBOL.
vlovich123 1y ago
I know they get a lot of hate but I personally used Oracle’s cloud and it seemed like a decent piece of infrastructure with really solid engineering team behind it. They had their share of problems (and their first line of support is really really bad and not well suited to handle real issues) but not really any different from any other similar products.
jeroenhd 1y ago
Oracle offers everything. Databases, web frameworks, programming languages, CRMs, cloud management tools, auto scaling clusters, identity management, BI, document search, email servers, colocation, mainframes, calendar sync. You name it, they've got it. Oracle has products ready to go to run an entire country if it needs to. Sure, Postgres beats OracleDB, but Postgres doesn't integrate as well with Oracle Fusion and you need to migrate the code yourself. They're like SAP: they're big enough that you can make a career just out of configuring their software packages and make good money while doing so. It's expensive and certainly not the best, but it's reasonably stable and has a huge company backing it. Oracle won't disappear any time soon and they're not as likely as Google or Microsoft to shut down their services within a few years notice. In some countries, Oracle is also very good at doing what Google and Microsoft are doing to students. The Brazilian programmers I've spoken to specifically learned OracleDB when they were taught relational databases. They learned to program in Java, and I'm sure Oracle also sponsored other parts of the professional tooling they got to use for free. Microsoft, on the other hand, didn't seem to generous towards their educational facility (no free MS tooling for their schools like they offer over here). If all you know is Photoshop/Windows/Maya/OracleDB/iOS, you're going to look for jobs where you can use Photoshop/Windows/Maya/OracleDB/iOS, and employers looking for cheap juniors will need to offer Photoshop/Windows/Maya/OracleDB/iOS to make the best use of them.
- arresin 1y ago
  Damn I’m wondering now if that’s why we studied Java as our first programming language in Australia at uni for CS. Oracle have a relatively big presence here and there’s a comfortable “mates” system that runs the Australia (soft bribery).
  - chickenzzzzu 1y ago
    Java has been a staple of university courses since before Oracle bought Sun. I guess I am getting old now, if the kids all want to program in Python and Javascript for 4 years...
  - jeroenhd 1y ago
    As much as I hate Oracle, there are huge advantages to Java. Java is strongly typed, free, and runs damn near anywhere. It's pretty much the go-to language when people think about OOP and for good reason. It has some excellent open source IDE support and it's widely used in the industry. With the current OpenJDK setup, it's also free of Oracle's licensing issues. It runs fast and can support most algorithms you'll probably ever study, without the manual memory management troubles of something like C++. I was taught C# in uni for very similar reasons except the entire uni ran on Windows and the Microsoft platform, which made doing assignments on Linux rather inconvenient. With the status of dotnet core, I'd say Java finally has a good competitor when it comes to teaching OOP languages.
ManBeardPc 1y ago
I made a similar experience. The database just stopped working one day, triggered by a normal restart without any updates or patches. Fixed the configuration, tested it, restarted multiple times, everything works. After a week it was not working again, without me being able to fix it except doing a fresh reinstall on a clean VM. Luckily more and more customers switched to Postgres and I no longer have to deal with it.

wiseowise 1y ago

“Engineers” Must be engineers who write requirements and unrealistic deadlines that lead to such issues.

devoutsalsa 1y ago
I just watched a YC video about vibe coding. He says he used Claude Sonnet 3.7 to configure his DNS servers & Heroku hosting. What could possibly go wrong?! "How To Get The Most Out Of Vibe Coding | Startup School " => https://www.youtube.com/watch?v=BJjsfNO5JTo&t=494s https://www.youtube.com/watch?v=BJjsfNO5JTo&t=494s I like his advice on downloading relevant documentation and putting it in the code base. That makes sense to me for targeted use cases.
- Ygg2 1y ago
  How it actually went: https://youtu.be/_2C2CNmK7dQ?feature=shared https://youtu.be/_2C2CNmK7dQ?feature=shared
  - devoutsalsa 1y ago
    I didn't need a header anyway.
- BiteCode_dev 1y ago
  I managed to break production before vibe coding was cool. 20 years old me had rm -fr root access to way too many systems. I don't think it's much different today. If anything, I think the youngsters will learn faster from their mistakes because they already have good mentor for the easy stuff and will get grinded on the hard stuff sooner.
  - devoutsalsa 1y ago
    Maybe. One thing I've found is the LLMs give me a lot of short term leverage. I can (get an LLM to) program in a language I know almost nothing about. Then I hit a wall where I don't know enough to get the LLM to fix something I don't know how to ask for, and then I'm stuck. When vibe coding, I'm not learning how to solve problems myself so much, and that means anything the LLM can't do, I also can't do. When I'm coding on my own, I'm picking up lessons along the way the help me build a foundation for incrementally harder projects.

amelius 1y ago

Doesn't matter. They have all the good lawyers.

xupybd 1y ago

I hate working on production systems and I hate any action that can't be easily reversed like deleting stuff.

cheema33 1y ago
Yeah, I like working under ideal conditions too. And I also want a toilet made of solid gold.
- keyle 1y ago
  You sound sarcastic. It's not too much to ask to have all actions reversible in 2021.
- arandomhuman 1y ago
  Yeah database backups are just too hard for a multi billion dollar corporation to handle.
- aeonik 1y ago
  Why solid gold? It would be way too ductile, heavy, and conduct far too much heat. Diamond might actually be better: low surface energy means a low coefficient of friction, so it would be much easier to clean. It would still suck the heat right out of your cheeks, though. Realistically, porcelain or other ceramics are probably the ideal material.
  - alwa 1y ago
    Or plastic, backed by electric heating! https://www.totousa.com/washlet https://www.totousa.com/washlet
- alwa 1y ago
  As I recall, the Guggenheim offered theirs for loan to the White House a few years back… https://www.guggenheim.org/exhibition/maurizio-cattelan-america https://www.guggenheim.org/exhibition/maurizio-cattelan-amer...

surfingdino 1y ago

I'd risk to say it was compounding interest on tech debt, but it doesn't sound as sexy, does it?

mrweasel 1y ago
> engineers conducting maintenance work mistakenly deleted critical storage connected to a key database I'd say poor process management. Why is an engineer even deleting critical storage (I take that to mean that they are deleting something of a file-system). Perhaps they where dropping a database, but you wouldn't do it like that in a critical environment. You'd disable the database access first, and then after some time, weeks, you'd drop the database, after doing a final backup. It could also be disconnecting a SAN, deleting a storage pool, something like that, but your process should say: Check of read activity, off-line the storage, anything non-destructive, and the only later, once you've verified that everything runs without this resource, do you delete. At previous jobs I've worked with healthcare system. You have processes, you follow those proesses to the letter and you never delete anything as your first step. Deleting is almost always done by going into read-only mode and ageing out the data. The fact that recovery time is four days tells me that no one followed a single procedure. Because there should be a written step by step plan, including recovery and risk assessment and when the change manager sees: "Recovery time five days" they will ask questions and stop you. This is probably the only answer less sexy than technical debt.
- pferde 1y ago
  This should be the top comment.

cheema33 1y ago

Oracle has a horrible reputation among devs. But I think they bypass devs for purchase decisions and straight up wine and dine and bamboozle the low information execs. When I worked at a megacorp as a dev, I had near zero say in such purchase decisions. I had to work with what I was given. Thankfully I work for a much smaller shop now. Better pay and much better decision autonomy.

reactordev 1y ago
This is exactly right - they say the right things, to the right people, over the right sized bribe rib.
forinti 1y ago
Devs get Oracle's best side. They don't know how lucky they are. PL/SQL is a fine language and Oracle throws everything and the kitchen sink into the APIs available inside the database. As a sysadmin/dba I get to handle the nasty side of Oracle: the bugs, the patches that fail, the redundant tools, the wordy documentation that always feels like a never-ending advert.
- vkazanov 1y ago
  When I was young and beautiful and naive I enjoyed playing with setting things up, even though my main focus was on backend dev. Like, setup postgres, tweak settings to performance, that kind of thing. So I was the first one to get the "figure out on site installation steps" task to share with client admin. Our small company was building enterprises things for large clients. And one of them wanted Oracle on Windows server. They also wanted a failover setup. How hard could that be? Now I hate Oracle. I hate Oracle consultants. I despise the ignorance. My favourite bit was Oracle uninstall procedure. I had like 4 pages printed just for this case. Oracle bad. Postgres good.
- hahn-kev 1y ago
  The problem is that putting code into your database has all the problems of data migration with no testability. It's really hard to write tests and debug code compared to a proper language.

Pinus 1y ago

Is this the same Oracle/Cerner system ("Millennium", I believe) that, despite protests from the medical staff, was deployed "big-bang style" in the Swedish region of Västra Götaland, with much the same results? (And where during a press conference, where the management was explaining how nearly everything was going to plan, a doctor, who had somehow sneaked in, got up, shouted something to the effect of "you’re lying, it’s a bloody disaster!" and stormed out.) (That was not an outage, though — as far as I understand the system was working, it just didn’t actually work...)

harvey9 1y ago
Could be, but oracle only bought cerner a few years ago and millennium has been around much longer. Other EMR systems are also dreadful - my spouse has some stories about EPIC for example.
shigawire 1y ago
Swedish privacy laws for EHRs are quite different compared to other countries. So I'm not surprised any software not purpose built for for that market would fail. And we know Oracle is not investing in R&D for Millennium since all efforts are out into their forthcoming AI based EHR - whatever that actually means....
deleted 1y ago
[deleted]

phaedrus441 1y ago

Can't wait for the overpriced, late, outdated Oracle to get deployed at the VA! We get to go from one bad EHR to another...

netdevphoenix 1y ago

Title should be "Oracle execs caused five days software outage at US hospitals". If the systems helped save lots of life, you bet the engineers wouldn't be thanked for that and it probably wouldn't even be a news article

nojito 1y ago
The engineers are the one's who deleted the db though.

djoldman 1y ago

Is it possible to fix EHR without drastic changes in the health/pharma/hospital/medicine regulation environment?

exabrial 1y ago

As a previous Cerner employee, this is actually pretty good.

DontchaKnowit 1y ago

Used to work on CHS systems for Cerner. If I am not mistaken they were a "communityworks" client, which meant their databases were shared with a number of other clients - a "multitenent" envirnment, we called it. Completely bumblefucked design. Not surprised somthing like this happened. Also - cerner software in general allowed hospitals to freely completely fuck up their own architecture. Sometimes ireperably. If anyone has details about how this happened Id love to hear.

sea_boat 1y ago

[dead]

johnwatson11218 1y ago

I think the Oracle Transaction Manager is one of the best pieces of software that I had to work with in a professional settings. Lots of other stuff in an enterprise setting is very flaky and follows trends but the Oracle internals seem very nice.

torgen-ai 1y ago

My company (torgen.ai) is looking for senior Oracle devs to help customers re-platform.