22 ms·

HSBC moves from 65 relational databases into one global MongoDB database

251 points by truth_seeker 6y ago

pier25 6y ago
After the latest Jepsen analysis of Mongo I'm surprised anyone would use it for anything that serious.
- truth_seeker 6y ago
  I can't speak for Jespen. But I am using it from last several years and it never ditched me even at the transaction data flow rate of 20K per second. Database alone isn't the deciding factor, how you used it in your application architecture matters the most. No database is perfect but some are more flexible than others.
  - deleted 6y ago
    [deleted]
  - pier25 6y ago
    My comment was about failed transactions. Are you using ACID transactions?
    - truth_seeker 6y ago
      Yep. MongoDB 4.2 In fact we replaced Kafka with Change Streams( little over 7k messages per second) and ES with text search (2K queries per second) in MongoDB on 16 CPU core nodes in our cluster. I don't believe in any benchmark unless it matches the business use case and development practices I use at work. I recommend the same to others.
- ra 6y ago
  whenever I hear of mongodb - I think of that "but it's web scale" video.
  - pier25 6y ago
    Haha yeah me too :) This is the video if anyone is wondering: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
- justin66 6y ago
  HSBC Commerical might have just asked everyone to put their data into a mongodb data store that nobody ever uses, which would make the headline technically true but kind of irrelevant. One thing you can be sure of is that they didn't stop using 65 relational databases.
jbverschoor 6y ago
shocking
neximo64 6y ago
If someone was involved could you share details on the rationale behind this? I used to like mongo but has it changed. I remember specific flaws/hacks with bitcoin websites coming down to mongo race conditions/inconsistency issues.
- brabel 6y ago
  The article explains quite in depth the rationale behind this. Basically, they think it's too difficult to manage their current relational databases differently in each country, but they need that due to the different regulations and particulars on each country. With one global DB, they hope they can have one single application for all countries, with country-specific things hidden in only a few sub-documents. _ It's now a one service environment, one database and one execution path for all the countries. This is made possible because of MongoDB's document model and the ability to map all the different table requirements for each country into a single collection, using sub-documents. Everything is simplified into one collection using country specific identifiers._ How they will address race conditions, eventual consistency and so on is not addressed at all though. I guess they trust Mongo's new transactions will take care of that (at which point, it would be nice to know whether that actually worked and what the performance will be when compared to the old relational system).
petepete 6y ago
Everything about this sounds mad. I'd have put referential integrity high on a bank's list of wants. Clearly they have some expertise, I'd love a more in depth look at how they arrived at the decision.
- ponker 6y ago
  These are definitely not source of truth customer transactions. Likely auxiliary databases to power various experiences. Like, a customer walks into the bank and the agent needs to know what services to upsell. Or the customer logs into the website and is shown their current portfolio value.
  - cosmodisk 6y ago
    That kind of stuff could have been implemented with any large CRM system(Salesforce,Dynamics,SAP) without any bigger issues.
    - threeseed 6y ago
      a) Salesforce is usually out of the question due it not being on-premise. b) Dynamics and SAP are both ridiculously expensive especially since you would be need to also buy additional software e.g. Windows to run it on. Plus of course it's much harder to find specialist engineering talent who know these products. c) All three are far too slow both on the ingestion side e.g. millions of mutations a second and on the read side e.g. feeding into a real-time ML model for an upsell prediction. MongoDB is specifically designed to cater to both of these technical requirements since you can mutate sub-documents and retrieve the full document extremely quickly.
      - cosmodisk 6y ago
        a) I'm not suggesting this for banking transactions- that'd be crazy. However customer service, interactions whilst at a branch, handling all sorts of loyalty programs and etc. To my knowledge, Barclays in the UK use Salesforce for some of these. b) Don't know SAP licensing model but I can't see a major bank not being able to afford $100/month per user for Dynamics/Salesforce. c) They do some offering for these kind of things,but I can't comment on how itd work on such a scale.
0xy 6y ago
Very rarely do headlines make me physically cringe but here we are.
- threeseed 6y ago
  So you have no idea about the business and non-functional requirements or the architectural or engineering considerations that went into this decision. But yet you believe it's such a bad decision that you physically cringe. Classic HN.
  - giantDinosaur 6y ago
    I think there's a bit more to it than that, given Mongo DB's history.
  - choeger 6y ago
    Did you ever actually read mongo db code? I had to, and only the drivers, but when I learned that the C++ driver would just nope out of town in some cases and literally exit() the application, I made the decision to get rid of that stuff ASAP. There are other instances, pymongo cannot be forked, for instance. Good luck using it inside celery. The connection string parsing is another matter. All in all their driver's code looks good at the surface, everything documented etc. But the comments are more or less superfluous and the design is a mess.
kchoudhu 6y ago
Yeah, this is about par for HSBC IT.
realusername 6y ago
I'm not in the banking industry but this sounds like the type of industry where you really need strong DB schemas in place and not something mongo-like ...
- jusssi 6y ago
  But the database doesn't need to be the component enforcing those schemas, right? They could validate the schema of a document on storage and retrieval. And keeping the schema version within a document would allow for multiple schema versions to exist within a single collection, which then would allow lazy migrations. I'm not saying I'd choose Mongo specifically for this job (if for anything), but the lack of DB enforced schema might not be the biggest problem here.
  - arcturus17 6y ago
    This is the problem I’m alluding to in another comment about NoSQL in this thread. Why would you want to enforce the schema in code when the DB could do it for you? It becomes such a mess and such a burden for the developer!
    - sjwright 6y ago
      That’s not always true. Often it’s not true. There’s a reason why we don’t have file systems that guarantee all .jpg files are valid JPEG files.
      - WJW 6y ago
        Now I want a filesystem that guarantees all .jpg files are valid JPEGs.
        sjwright 6y ago
        Until you need to do an emergency transfer of some recovered data volume and there’s a few dozen slightly corrupted JPEGs scattered throughout a few million files. Also, I don’t think I’d want my filesystem deciding what qualifies as a valid JPEG. But yes, I can see the theoretical appeal.
asplake 6y ago
Never mind the technology, a single data model would be a feat in itself.
- ilaksh 6y ago
  I think the single data model in this case is probably something like requiring a couple of fields and everything else is whatever you need. Heh.
NicoJuicy 6y ago
When do they go live and how can you short them? Ps. I think they never heard of database per service when looking at their microservice graph. Lol
- shoo 6y ago
  probably safer to buy put options than short the stock (shorting it exposes you to unbounded losses if the stock price rises), but as you say, you'd need to get the timing right. https://finance.yahoo.com/quote/HSBC/options/ https://finance.yahoo.com/quote/HSBC/options/ if they've got a decent QA process and they can't get the new system to actually work reliably there's a fair chance this project would get stuck in QA for a year or two, blocked from going live and cancelled, so it wouldn't cause catastrophic data loss, just reallocate a bunch of money from shareholders to contractors / employees / vendors. > microservice graph banks have graphs of macroservices.
- robjan 6y ago
  HSBC's own website lists dozens of exchange traded warrants with various strike prices and gearing. Need a brokerage with access to the HK market though. https://www.warrants.hsbc.com.hk/en/tools/search/ucode/00005/type/put https://www.warrants.hsbc.com.hk/en/tools/search/ucode/00005... I'm not sure about their choice of Mongo but the initiative seems like it should be helpful for customers like myself who have accounts in a few countries. We can access them all from one app but the experience is inconsistent at the moment. I'm cautiously optimistic.
  - NicoJuicy 6y ago
    Seems like the use case for graphql, not mongodb
_Understated_ 6y ago
I've worked contracts in UK banks for a number of years now. Nothing on the mainframes but I know some of the guys that do work on those and, despite the constant stream of flavour-of-the-month tech that we talk about constantly in places like HN, the mainframes work well and will not be replaced any time soon. The dev cycle for releases and whatnot is incredibly long due to testing and, lets be honest, fear, in case something breaks but when it comes to battle-tested technology, mainframes and DB2 are right up there. I do know of some code that's over 40 years old and still runs at the core of one of the big banks... MongoDB, despite its capabilities, is... an odd choice imo! Not trying to second-guess but if I was in that meeting I would have certainly raised an eyebrow. Good luck to them.
- sk0g 6y ago
  My prediction is something will go horribly wrong, then they will go "see, we tried modern tech and it fucked us over", and they will commit to seeing their code and tech stack turn a century old before considering any updates.
  - AmericanChopper 6y ago
    I read the article and couldn’t figure out what they were actually putting in mongo. If it’s their core processing/ledger then that would be very surprising. But banks have dozens of different systems, some of them would actually almost be suited to Mongo, and some of them have rather loose SLAs. Unless I missed something, the article didn’t really make that clear.
    - refset 6y ago
      There's another thread on the homepage right now about Crux [0], which is a database designed specifically to be able to operate downstream from multiple legacy systems using bitemporal indexes to stitch everything together. This pattern is very common in banks e.g. for calculating and reporting on risk positions. I expect you're right that this story is purely about using Mongo for downstream systems. [0] https://news.ycombinator.com/item?id=23493163 https://news.ycombinator.com/item?id=23493163
AJRF 6y ago
Building in plausible deniability so they can resume their HSBC money laundering services with impunity? Geniuses! Edit: In case you missed the story https://www.bbc.co.uk/news/business-18880269 https://www.bbc.co.uk/news/business-18880269
jauer 6y ago
"is looking to" makes this sound aspirational. The absence of operational lessons learned that would come out of a deployment make me wonder if this actually happened.
- hliyan 6y ago
  Thank you. The headline reads "HSBC moves..." which in normal news parlance means that they've recently completed the move or the move is in progress.
chrisdbanks 6y ago
The whole premise of this seems wrong. I've worked on large financial systems that have happily supported different rules for different countries using a relational database. The database was never the issue, it was how to handle testing and feature enablement. This seems like the wrong cure to the problem, and a cure that will end up causing more problems than it solves. i.e. a hack. I'd love to hear more about the rational. I just don't understand it.
implying 6y ago
A fortune 100 financial institution I once worked for had an initiative to migrate their extremely large, fairly well normalized DB2 database to schema-less dynamoDB. All new tables had to go into dynamo, and if your service begins consuming an old DB2 table, it had the job of migrating that data to dynamo. The end result of this was applications that half pointed to DB2, half dynamo, confusing new data with old, and tons of bugs relating to losing entries into dynamo tables under outdated keys, not being able to clearly match up customer data, and myriad problems that were never a consideration using a relational database. Needless to say, I pulled all of my assets out of this institution as soon as I had the chance.
- QuinnyPig 6y ago
  Oh my god I can only imagine how they provisioned their DynamoDB throughput. Setting a dumpster full of cash on fire would have been more expedient.
  - implying 6y ago
    The dynamoDB bill ran over $25,000 a month, just for their dev environment. Granted, they had thousands of developers, but still.
- threeseed 6y ago
  I've spent decades in enterprise space and unfortunately developers often don't have a clue how actually things work. Companies like IBM specialise in royally screwing companies through their licensing deals and with their legal enforcement teams. They are nothing like dealing with a normal startup vendor or with someone like AWS. They play hard ball. So sure you can complain that they should've just stayed with DB2. But usually that means paying tens of millions which then increase each year since they know you won't migrate. That's why companies take drastic decisions and move to the cloud even if it's technically not the best option.
  - tluyben2 6y ago
    But drastic can be going to Postgres/Mysql instead of db2/oracle; Dynamo (by the sounds of the case here) is more on the insane scale than drastic.
cranekam 6y ago
I have no actual information about this migration but I wouldn’t be surprised to learn that the databases that are being consolidated are for some minor role or are separate instances with duplicate data for analytics or whatever. It sounds wildly unlikely that they are going to smush together all their customer accounts in one MongoDB instance. As well as being a huge technic challenge I’m sure regulators would have something to say about this.
- sjwright 6y ago
  This is very likely. Most banks have stupid-huge quantities of products and services, each with extremely elaborate requirements and data storage needs. “65 relational databases” might well only represent a few percentage points of HSBC’s data footprint.
  - deleted 6y ago
    [deleted]
- bestboy 6y ago
  TBH I think so, too. I guess this is just a glorified global network drive to put all the different, country-specific documents like Excel sheets, Word docs, PDFs and some such. https://youtu.be/GI5kwSap9Ug?t=63 https://youtu.be/GI5kwSap9Ug?t=63
andrewksl 6y ago
Sounds like a win-win for them. They get to market themselves as staying up to date with tech (although the HSBC personal banking app I use indicates otherwise). And when they "lose track of" the money they move for the cartel and terrorists, they can blame MongoDB and spare some execs the jail time.
- deleted 6y ago
  [deleted]
- Charlie_26 6y ago
  I'm glad someone mentioned cartels and terrorists
edf13 6y ago
I’m in the UK and glad I don’t bank with HSBC
nwmcsween 6y ago
Is it common that a sufficiently large doc db is just an un-normalized rdb? I've seen swagger uis of doc dbs that make me wonder wtf/why.
arcturus17 6y ago
There’s been a lot of talk about the Jepsen-Mongo affair on HN lately, but the problem I have with Mongo and NoSQL in general is much more basic - it’s modeling relationships, which always end up appearing in every data model I’ve ever designed. I’m aware you can keep references from one doc to another but it always ends up being a messy affair even at smallish scales, and it feels as if the cognitive burden of managing these relationships ends up falling on the dev, instead of being managed by the DB. For those of you using MongoDB in large scale large production apps, is this not a problem? Is your underlying business domain really non-relational, or do you manage to comfortably run highly relational models on a doc-based DB? How?
- thawkins 6y ago
  Mongodb now supports effectivly "Joins" in its aggregation framework, so it can do some relational style data representation. The latest versions also support transactions.
  - BozeWolf 6y ago
    Yeah but they do not recommend using that. I think the one nosql afficianado in my team spoke with mongo support and they recommended reassessing our document structure for it. (So a migration after all... ) Also in their blogpost [1], about $lookup they say: We’re still concerned that $lookup can be misused to treat MongoDB like a relational database. But instead of limiting its availability, we’re going to help developers know when its use is appropriate, and when it’s an anti-pattern. In the coming months, we will go beyond the existing documentation to provide clear, strong guidance in this area. [1] https://www.mongodb.com/blog/post/revisiting-usdlookup https://www.mongodb.com/blog/post/revisiting-usdlookup
speedgoose 6y ago
The whole thing sounds like a terrible idea. Why would you want to put all the data from at least 65 different applications in the same table? In MongoDB? To save cost? I simply don't get it. It doesn't make sense to me, but perhaps they know something we don't.
- threeseed 6y ago
  360 customer view as an example. If you are feeding a real-time decisioning, support, advertising system then you need all of the customer features accessible as fast as humanly possible. Ideally a consistent O(1) time. Or it could be a real-time fraud system where you need every bit of customer data in one payload to feed into an ML model. You can't do any of that if you are having to do multiple joins across federated databases.
  - speedgoose 6y ago
    I would collect data and push events to a new system, not migrate 65 relational databases to a single nosql database.
sidcool 6y ago
How do they get ACID guarantees and schema enforcement?
- threeseed 6y ago
  MongoDB supports transactions and you enforce schemas in code instead of the database. Pretty common these days to just rely on Git and not DDLs.
  - thawkins 6y ago
    You can also tie json-schema descriptions to collections and have it throw validation fail exceptions on writes that fail validation.
deleted 6y ago
[deleted]
cm2187 6y ago
And of course they will expose it to the web without authentication!
aogaili 6y ago
Lots of negative comments about MongoDB for folks who don't use it, but perhaps give NoSQL a chance and keep an open mind? there is no need to structure the data into tables and schema does not need to be enforced at the DB level. Google has been running its entire infrastructure on a NoSQL database for years. Perhaps folks need to keep some open mind and see if that works instead of dismissing a new trend?
- mrweasel 6y ago
  For MongoDB it’s mostly a case of it had issue years ago when it was the new hotness, but it should be a boring choice by now, and it’s not. Neither is Cassandra. Kafka is boring, even more so if they kill of Zookeeper. NoSQL has proven to be hard to get right than was first assumed and at the same time RDBMS’es have improved and adopted many ideas from the NoSQL world faster than expected, often with better results. What’s no clear from the article is why MongoDB was picked, so maybe there’s something that made it the obvious choice, it’s just hard to see what the might be.
  - aogaili 6y ago
    "For MongoDB it’s mostly a case of it had issue years ago when it was the new hotness, but it should be a boring choice by now, and it’s not" SQL has been around since birth of computing, give NoSQL some time. "NoSQL has proven to be hard to get right than was first assumed" Nothing has been proven, again give it a chance, SQL had years of optimization. Also, no need to downvote if you like SQL or your job depends on it, or that is all you know, just debate with reason and evidence, I've used both and open to change my mind with evidence and case studies which we don't have many yet and folks here predicting doomsday to HBSC without keeping an open mind.
- executive 6y ago
  Google also used to not be evil.
- arcturus17 6y ago
  “Google has been running their entire infrastructure...” I’m skeptical about this - Google’s entire infrastructure is so large that surely they must be using a wide variety of db technologies? I took a databases course that did mention that part of the Google search infrastructure runs on a tailor-made NoSQL technology, but it wasn’t really a doc-based approach, it was more akin to a relational DB but with flexible columns (can’t remember the details sorry). At any rate I don’t feel that this is any vindication for NoSQL, as Google builds much of its underlying tech from scratch and its hardly comparable to the out-of-the-box solutions that the rest of us, even massive corporations like HSBC, have to work with.
deleted 6y ago
[deleted]
ezconnect 6y ago
I think they’re just using it for document storage not customer and financial data. Document storage and retrieval is a big cost for large institution.
- niffydroid 6y ago
  We found mongo expensive for documents. We had simple documents basically saying X sent email Y to Z. The mongo db was cloud hosted and was over 500gb and costing nearly 3k. Moved them into S3 with a simple SQL dB to look up reference IDs. Certainly reduced the cost!
executive 6y ago
Sounds like the opening scene of a horror movie.
BiteCode_dev 6y ago
It they just put commercial details follow up data + some stats in there, it's ok. It is very infrequently updated, integrity is not at too much risk. In case it's lost, business is not impacted too hard as long as you can find contact details eslewhere. But the replication story is good, and if they got tons of different schemas, it will make their life easier. So yes, for transactions it would be bad, but maybe for this particular use case, it's ok.
predictmktegirl 6y ago
And it’s gone. (Sorry)
- egeozcan 6y ago
  While I'm generally against getting to meme-y in HN, this is spot-on, adding to the already-long list of ways banks can make your money disappear and/or make unreachable. Reference: https://www.youtube.com/watch?v=_nVk25ZvTkU https://www.youtube.com/watch?v=_nVk25ZvTkU
markmm 6y ago
The new FinTech banks are eating their lunch by providing mobile apps that innovate rapidly due to being build from the ground up with modern tech. The old banks take years to rollout a simple feature like locking your card from the app due to being stuck on mainframes or some ancient stack. Moving to mongodb doesnt sound like the right solution though there are better choices out there.
sitkack 6y ago
This is like one step above csv files on an NFS share. Could be a great way to wash some crimes off the books, "oh our DB failed and our backups were stored in the database as well, oopsie". In less than 5 years there will be an EU directive on how financial institutions need to have immutable db architectures with full provenance.
- echopom 6y ago
  > In less than 5 years there will be an EU directive on how financial institutions need to have immutable db architectures with full provenance. I'm a former Entreprise Architect from Banking Sector in Europe. To be clear , this will never happen. There has never been any directive that has had a "concrete" impact on IT Architecture of Financial Institutions. Yet there has been dozens of regulations that urged banks to "simplify" their IT Systems. 90% of IT Staff are Baby Boomers with no background in Systems Design or Architecture , thus when new regulation comes in it follow this scenario 100% of the time : - Find a vendor that sell a software that promise compliance with new regulation - Find an integrator that promise integrations within the deadline - Integrate the vendors software in Banks legacy stack of 3000+ monolithic apps - Send report to regulator saying they have "redesigned" their architecture and made "investment" in order to take in account and that new regulation Best examples of this is "PSD2" which has been the biggest fiasco of the industry , has of the last year only 18% of banks complied with the regulation. France and UK said they would not "fine" anyone, because banks "aren't ready" and made "considerable" investment in it. Regardless of what HN thinks , you won't solve Financial Institutions Multi Decade Legacy with a single Directive that would suddenly force them to use "ImMuTABLe Db ARChITectuRES" It Directives have never worked and will never work. The only way to enforce anything would be to remove their IT systems completely and have them use APIs provided by the regulators , and have the regulator become the sole provider of "Financial System". They wont let that happen.
  - boruto 6y ago
    I have worked in a German bank and the bundesbank does regulate on very high-level decisions. f.e., systems which do financial crime monitoring need to be compliant with their rules. Access to pre-production should be limited.
jpswade 6y ago
Oh my. This seems like a really bad idea. I'm calling it now, this will go wrong. That "Micro Service Instance" is not a microservice, it's an Enterprise Service Bus (ESB), aka the Egregious Spaghetti Box. That's aside from the fact that relational data belongs in a relational database. I really can't imagine it being a single instance, that just does not make any sense.
ycombonator 6y ago
HSBC IT is a hollowed out outsourced shop. Just like Avis Budget, Disney, Walmart and Walgreens. They will all move to MongoDB data stores sooner than later.
kristaps 6y ago
Each and every one of these MongoDB migration rationales contains some variation of "schema design is hard, let's go shopping." Granted, there are plenty domains where you can get away with having a blurry schema with the occasional bug creating a new localized data anomaly, but banking, where such things lead to customer trust erosion - I just don't get it.
kgc 6y ago
And I am moving all my money out of HSBC...
tannhaeuser 6y ago
Mongo works well for storing chunks of JSON from transactional browser apps (which I guess is what HSBC is targetting), doesn't require you to have custom endpoints, has a good replication/HA story (on paper at least, if your devs know what they're doing and don't use defaults plus multiple nodes with CORS, etc.), good mindshare among webdevs, and is even the de-facto API for JSON stores with AWS' "DocumentDb" and Azure's Cosmos DB being drop-in replacements. But using it for banking/backoffice apps would be pure madness.
buboard 6y ago
If you 've ever used HSBC web banking u ll know it's one of the worst things mankind ever created. I m not sure if this is good news for MongoDB
deleted 6y ago
[deleted]
dark_light 6y ago
Hiding your complexity behind a (poorly designed) layer doesn't make it go away.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
sohamsankaran 6y ago
Jepsen's latest analysis of MongoDB -- https://jepsen.io/analyses/mongodb-4.2.6 https://jepsen.io/analyses/mongodb-4.2.6 -- finds that it doesn't even preserve snapshot isolation when set at the highest consistency level. This seems like a pretty terrible decision. I've never wanted to short a company more in my life.
- lowwave 6y ago
  Seems like the guys at HSBC tech department are still doing the whole fashion over merit.
  - sohamsankaran 6y ago
    What's wild about this is that Mongo isn't even in fashion anymore! NoSQL has been supplanted by NewSQL, and relational dbs are in season again. Even if they were trying to follow the trend, they'd be a half-decade behind.
    - catmanjan 6y ago
      Interesting, what is considered newsql? Postgres?
      - bob1029 6y ago
        SQLite is the hottest new thing on the block.
        mkl 6y ago
        Huh? SQLite is nearly 20 years old, and is only intended for small scale use.
        bob1029 6y ago
        It was intended as a joke in context, but I believe SQLite is far more capable than most will give it credit for. This is especially true for applications which you only ever intend to run on a single logical machine. But, even for those applications requiring multiple nodes, it can still be a powerful tool if you assume clustering & replication duty in your application logic. For any use case, SQLite is one of the best ways to persist structured data to local disk.
FHermisch 6y ago
Microservices and NoSQL - the wonder-weapons against every IT problem, aren‘t they?! I really have to pivot my business - new name will be „The Microservice-NoSql Consulting Company“ and we will sell tailored versions of the „Microservice-NoSql Strategy“! (Changing names and colors of boxes for 2000$ per day)
jonnypotty 6y ago
Lol. Just lol.
murukesh_s 6y ago
Relax HN guys.. this is the reason for this: "Local requirements for each country will be built into the application, but there's no need to maintain separate data models or separate databases anymore. We could easily design the global data model and database using the MongoDB JSON schema model. That brings data from all operating countries into one database and the application can run on just one database. Which is a lot of reduction in resource and maintenance cost." Is there any Database that can do this? other than MongoDB? I am hearing about data corruption in MogoDB, I believe that is due to bad, out of the box settings, which I believe they would have mitigated when using in cooperation with MongoDB company (which I am sure they are).
- joosters 6y ago
  I just love software where the out-of-the-box settings lose your data. At least HSBC are 'web scale' now!
- BozeWolf 6y ago
  Whats the exact advantage, one can wonder. Each country still needs its own application. With its own models. With its own migrations, or at least ways of dealing with missing or mixed data types. What if... those models map to sql database schema’s? wouldn’t that be magical? Not since 2008. Whats left is maintaining multiple databases. That sucks indeed. Sql databases support bson as well these days. What they probably want (or did) is make a base framework, suitable for all countries. And have each country develop its own stuff on it. No need to use mongo for that though. One use case i can imagine is that forms and input just change and that old information never will be compatible again with the newer forms. Mongodb serves as a giant more or less queryable data bin. Then one can ask the poor dba to lookup something for a client with “client_id”: xyz. Then return the raw bson/json output. Might be sufficient.
- inshadows 6y ago
  POSIX filesystem?
ngcc_hk 6y ago
Old guys ... what dB they use in London and Hong Kong. Are they still use mainframe?
xmorse 6y ago
Hacker News has become just “mongodb bad, sql good”
tluyben2 6y ago
I have seen many rewrites for financial institutions in my life; one of the biggest I had a good insight look at during the entire project was a Tandem Nonstop Cobol rewrite to Java; the famous 10s of millions of lines to nice clean Java code. Many of them never work; this one was, if I remember correctly, upwards of 50m Euros and was thrown away because it did not work in the end. I think the Tandems are still running.
9nGQluzmnq3M 6y ago
The article makes this sound way bigger than it is. HSBC has thousands of applications and probably tens of thousands of databases. Migrating one app to use a different DB model means very little for the big picture.
bob1029 6y ago
I have to say that on the surface, this sounds horrific. - MongoDB: Yes - Micro Services: Yep - Some attempt at a grand unifying model: Of course That's pretty much a 10/10 shitshow as far as I am concerned. Mix in banking regs & regional differences for afterburner on that money furnace. But, in reality, I doubt that one of those 65 relational databases involves the core customer/account/transaction processing facilities. These are probably (hopefully) still running on an IBM system with proper ACID & uptime guarantees. Anything a live transaction flow (especially credit/debit processing) is going to touch could never be trusted on something like MongoDB in this current reality.
- sohamsankaran 6y ago
  My fear is that those IBM systems running Db2 or something were counted among the 65. While it seems obvious to us that they're better off without switching those, I know of several companies in financial services that should know better that trust MongoDB for their core transaction flow.
  - guillaumei 6y ago
    The 65 relational databases are for the same application or similar deployed in 65 countries...At least I read it in that way.
  - jeremyjh 6y ago
    They didn't switch anything of consequence. A bank the size of HSBC has thousands of application databases. They never migrate their core transaction systems, they just build piles of shit around them ever higher.
bluestreak 6y ago
I was there in Aug 2019 and I did hear them talking about mongo, There was a programme at the time to "refactor" their estate, a part of that they wanted to replace bunch of ad-hoc databases with clunky- or no data governance with one (or fewer) central places, which will be under Data IT control. The rationale was to reduce risk of data loss, leak or general business continuity issue in what was 6-12 month project. The cost of architecting relational model that would be cover for 65 mini databases in a bank will be astronomical. It is far easier to setup no-SQL entry point that would auto-conform for whatever requirements upstream applications might have. It is a technical win for Mongo, but I don't believe this is an attempt by HSBC to get up to speed with database trends. HSBC are removing internal audit strikes, nothing more than that.
- simonswords82 6y ago
  We are working with HSBC and have not heard about Mongo being a thing despite working on data with them. What location and department were you in if you don't mind saying?
  - bluestreak 6y ago
    This was in London HSBC HQ, Equities
- sohamsankaran 6y ago
  Fascinating. Are they intending to use this for core transactional workloads? How do they intend to deal with Mongo's (many) known consistency issues?
  - bluestreak 6y ago
    I have not worked on this project, so I don't have details. Mongo is in a nutshell to replace a database trader setup under their desk and that grew quite important. There is nothing pretty about that. I'm quite sure they will have issues and workarounds for years to come :(
coding123 6y ago
This is probably some aspect of HSBCs computing needs, likely less than 2% of the data the store.
prostoalex 6y ago
Any time I have to use an HSBC online product (they throw in an occasional promotion for their credit cards and savings accounts) it feels like a trip back to the 90s - popups and extra windows on different domain names that are slowly "loading..." information for trivial operations like scheduling a payment on a credit card.
svrtknst 6y ago
Uh, how does this work with eg GDPR or data protection laws that require all customer data to remain within the country of operation?
29athrowaway 6y ago
This sounds problematic: 1) mongoDB does not offer the guarantees required by a bank. 2) Application developers using mongoDB rarely protect themselves against NoSQL injections. Most of them are unsuspecting that they even exist. e.g.: What is your user id? {$ne: null}... whoops, now every user record is returned. 3) They better make sure they use the Decimal128 type for their currency fields, and make sure that the value is not being casted to float on the client. e.g.: on a JS client, every number is a float. Unless they've taken the necessary precautions this is a disaster waiting to happen. Hopefully other banks do not do the same.
- mushi 6y ago
  > what is your user id I don’t think you actually tried it. It returns user who’s id is “{$ne: null}” (string)
  - 29athrowaway 6y ago
    It depends. Sometimes you can trick an application to parse that as an object.
will_raw 6y ago
I recently came to this article[1] where Guardian switched to PG from MongoDB [1] https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-h...
surajs 6y ago
Yeah this can't go well...
niffydroid 6y ago
I would suspect they'd keep the core money transactions in the existing DBs, but use mongo for less important things like sending junk mail, customer support sessions.
subins2000 6y ago
isn't this a bad idea
jarym 6y ago
I don’t see how one global mongo is going to help solve any of their issues. Maybe they just want to bury their issues under the carpet a little longer. I also hope they’ve configured their mongo correctly since out of the box Config is not appropriate for many use cases
- malikNF 6y ago
  I want to doubt a billion dollar company like HSBC will have engineers who will install things just out of the box and leave it without properly configuring it. But, they are proposing to use Mongo just almost a month[1] after a report came highlighting several issues with the product, ESPECIALLY for use-cases like a bank that requires a very high level of data consistency, so your point might actually be valid. Would really be interested to know how something like Mongo ends up getting seriously considered for something like this, I mean the whole decision process that goes in to selecting something like this. [1]https://news.ycombinator.com/item?id=23285249 https://news.ycombinator.com/item?id=23285249
  - jarym 6y ago
    I did a contract at HSBC in the past. Some teams are good but there are people who will install things out of the box and leave it.. even in a billion dollar company like HSBC.
Keyframe 6y ago
Modern take on shredding papers
historyremade 6y ago
Behind Bad Indian Coder ...
koheripbal 6y ago
This title is misleading, and the article is missing a key piece of information. This is ONE system at HSBC - out of literally HUNDREDS (maybe thousands) of applications. It's not as if HSBC is moving ALL of its applications to MongoDB. HSBC doesn't have a single tech stack. They have thousands of IT employees all using different tech, in different parts of the world, for different departments. This might be as inconsequential as the system that catalogues security camera feed URLs - or maybe the one that monitors remote employee company mobile data usage - who the F knows, because the article gives us zero information. Account Management, Credit Cards, Mortgages, Security, Asset Management, HR, Legal, Compliance, Regulatory, Risk Management, Trading, Operations, Building management, Payroll, ATM comms, Clearance/settlement, Website, a million different reporting engines, etc, etc, etc... and then each region usually has its own application for each function - literally hundreds of tech stacks in every tech you can imagine from DB2 Mainframe to Oracle to MongoDB. All banks are like this. This article is just vague PR and is just referring to one single group consolidating their regional instances. It does not deserve HN attention.
- flurdy 6y ago
  I initially rebuked thinking that the OP sounds like a disaster, but then I remember from working with other banks that they have an infinite amount of systems from decades of mergers, re-org, etc, all siloed in so many different ways. So in no way was the OP's article as catastrophic as it initially sounded.
  - chrisweekly 6y ago
    Your point is clear enough, but FYI you're misusing the word "rebuked". ("recoiled"? "reacted"?)
    - flurdy 6y ago
      I think I merged two sentences and feelings into one. :)
- tcgv 6y ago
  The article is very shallow on technical details indeed. It seems they are trying to promote MongoDB as the solution to their failed attempt of standardizing their application/data model: > Historically, (...) HSBC did have an application core program environment, which had most of an application's core functionality. But it couldn't have a single programme environment running for all the countries, due to the differences in data models and databases. > Another benefit is to use the same database for global data analytics and reporting. We don't need to translate into another data model or another database to run the analytics and reporting from that particular data. So this "application core program" failed to provide an unified solution to their data warehousing needs, and now they believe they can solve this problem by unifying 65 databases into one MongoDB instance, "taking advantage" of its schema-less model. It looks more like a global storage service than anything else to me.
akho 6y ago
65 databases unified globally, for HSBC’s global presence, is ~2 databases per country. This is, probably, a minor trial effort for one of their services, not the core system (which is un-unifiable due to banking privacy laws in place) or any kind of cross-function unification.
rmoriz 6y ago
I think this isn't primarily about a database migration/consolidation and specifically not about MongoDB. IMHO it's about cutting cost and complexity by building a single world-wide software platform. Of course this also means that local specialities and local teams will be questioned and probably be made redundant.
jsjohnst 6y ago
What’s the over/under odds on betting about how long it’ll be until it’s accidentally exposed to the internet and hacked?
trollied 6y ago
This article is misleading. They have migrated some of their databases to mongo, not all of them
addicted 6y ago
There’s a lot of complaining here but MongoDB has been used by startups for like a decade? I don’t think I’ve seen a single HN article complaining about actual data loss. That would be something that would get upvoted immediately. So what gives? Especially since earlier and older versions of Mongo apparently had far less data stability. I’ve probably read far more complaints about Postgres in HN articles (difficulty setting up, poor defaults, etc). And Postgres may not even be as popular as Mongo. So what gives?
mariopt 6y ago
I can see how they would use in some parts: analytics, internal reports, etc. It is interesting that you can use Mongo and keep the data in a geographical location and still be able to run a global query: https://docs.mongodb.com/manual/tutorial/sharding-segmenting-data-by-location/ https://docs.mongodb.com/manual/tutorial/sharding-segmenting...
ionwake 6y ago
Wasn’t there a problem where mongo would literally lose records under high loads ? There was an article on it about it a few years ago. Did they fix that?
blaufast 6y ago
That was a mistake
anonu 6y ago
I need to LOL a little bit at all this, the headline, the article. First off, how would such an article even come about? Feels like more of a recruiting/marketing piece. The old story at hsbc was that the ATM software hadn't been updated in twenty years, cause stability....
seanhunter 6y ago
To put this into context - I met with a senior tech person there (one layer below the group CIO) a few years back and he said they had something like 40000 distinct IT systems globally. So replacing 65 of them with one MongoDB instance is something like 1/8th of one percent of their systems.

251 points by truth_seeker 6y ago

pier25 6y ago

After the latest Jepsen analysis of Mongo I'm surprised anyone would use it for anything that serious.

truth_seeker 6y ago
I can't speak for Jespen. But I am using it from last several years and it never ditched me even at the transaction data flow rate of 20K per second. Database alone isn't the deciding factor, how you used it in your application architecture matters the most. No database is perfect but some are more flexible than others.
- deleted 6y ago
  [deleted]
- pier25 6y ago
  My comment was about failed transactions. Are you using ACID transactions?
  - truth_seeker 6y ago
    Yep. MongoDB 4.2 In fact we replaced Kafka with Change Streams( little over 7k messages per second) and ES with text search (2K queries per second) in MongoDB on 16 CPU core nodes in our cluster. I don't believe in any benchmark unless it matches the business use case and development practices I use at work. I recommend the same to others.
ra 6y ago
whenever I hear of mongodb - I think of that "but it's web scale" video.
- pier25 6y ago
  Haha yeah me too :) This is the video if anyone is wondering: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
justin66 6y ago
HSBC Commerical might have just asked everyone to put their data into a mongodb data store that nobody ever uses, which would make the headline technically true but kind of irrelevant. One thing you can be sure of is that they didn't stop using 65 relational databases.

jbverschoor 6y ago

shocking

neximo64 6y ago

If someone was involved could you share details on the rationale behind this? I used to like mongo but has it changed. I remember specific flaws/hacks with bitcoin websites coming down to mongo race conditions/inconsistency issues.

brabel 6y ago
The article explains quite in depth the rationale behind this. Basically, they think it's too difficult to manage their current relational databases differently in each country, but they need that due to the different regulations and particulars on each country. With one global DB, they hope they can have one single application for all countries, with country-specific things hidden in only a few sub-documents. _ It's now a one service environment, one database and one execution path for all the countries. This is made possible because of MongoDB's document model and the ability to map all the different table requirements for each country into a single collection, using sub-documents. Everything is simplified into one collection using country specific identifiers._ How they will address race conditions, eventual consistency and so on is not addressed at all though. I guess they trust Mongo's new transactions will take care of that (at which point, it would be nice to know whether that actually worked and what the performance will be when compared to the old relational system).

petepete 6y ago

Everything about this sounds mad. I'd have put referential integrity high on a bank's list of wants. Clearly they have some expertise, I'd love a more in depth look at how they arrived at the decision.

ponker 6y ago
These are definitely not source of truth customer transactions. Likely auxiliary databases to power various experiences. Like, a customer walks into the bank and the agent needs to know what services to upsell. Or the customer logs into the website and is shown their current portfolio value.
- cosmodisk 6y ago
  That kind of stuff could have been implemented with any large CRM system(Salesforce,Dynamics,SAP) without any bigger issues.
  - threeseed 6y ago
    a) Salesforce is usually out of the question due it not being on-premise. b) Dynamics and SAP are both ridiculously expensive especially since you would be need to also buy additional software e.g. Windows to run it on. Plus of course it's much harder to find specialist engineering talent who know these products. c) All three are far too slow both on the ingestion side e.g. millions of mutations a second and on the read side e.g. feeding into a real-time ML model for an upsell prediction. MongoDB is specifically designed to cater to both of these technical requirements since you can mutate sub-documents and retrieve the full document extremely quickly.
    - cosmodisk 6y ago
      a) I'm not suggesting this for banking transactions- that'd be crazy. However customer service, interactions whilst at a branch, handling all sorts of loyalty programs and etc. To my knowledge, Barclays in the UK use Salesforce for some of these. b) Don't know SAP licensing model but I can't see a major bank not being able to afford $100/month per user for Dynamics/Salesforce. c) They do some offering for these kind of things,but I can't comment on how itd work on such a scale.

0xy 6y ago

Very rarely do headlines make me physically cringe but here we are.

threeseed 6y ago
So you have no idea about the business and non-functional requirements or the architectural or engineering considerations that went into this decision. But yet you believe it's such a bad decision that you physically cringe. Classic HN.
- giantDinosaur 6y ago
  I think there's a bit more to it than that, given Mongo DB's history.
- choeger 6y ago
  Did you ever actually read mongo db code? I had to, and only the drivers, but when I learned that the C++ driver would just nope out of town in some cases and literally exit() the application, I made the decision to get rid of that stuff ASAP. There are other instances, pymongo cannot be forked, for instance. Good luck using it inside celery. The connection string parsing is another matter. All in all their driver's code looks good at the surface, everything documented etc. But the comments are more or less superfluous and the design is a mess.

kchoudhu 6y ago

Yeah, this is about par for HSBC IT.

realusername 6y ago

I'm not in the banking industry but this sounds like the type of industry where you really need strong DB schemas in place and not something mongo-like ...

jusssi 6y ago
But the database doesn't need to be the component enforcing those schemas, right? They could validate the schema of a document on storage and retrieval. And keeping the schema version within a document would allow for multiple schema versions to exist within a single collection, which then would allow lazy migrations. I'm not saying I'd choose Mongo specifically for this job (if for anything), but the lack of DB enforced schema might not be the biggest problem here.
- arcturus17 6y ago
  This is the problem I’m alluding to in another comment about NoSQL in this thread. Why would you want to enforce the schema in code when the DB could do it for you? It becomes such a mess and such a burden for the developer!
  - sjwright 6y ago
    That’s not always true. Often it’s not true. There’s a reason why we don’t have file systems that guarantee all .jpg files are valid JPEG files.
    - WJW 6y ago
      Now I want a filesystem that guarantees all .jpg files are valid JPEGs.
      - sjwright 6y ago
        Until you need to do an emergency transfer of some recovered data volume and there’s a few dozen slightly corrupted JPEGs scattered throughout a few million files. Also, I don’t think I’d want my filesystem deciding what qualifies as a valid JPEG. But yes, I can see the theoretical appeal.

asplake 6y ago

Never mind the technology, a single data model would be a feat in itself.

ilaksh 6y ago
I think the single data model in this case is probably something like requiring a couple of fields and everything else is whatever you need. Heh.

NicoJuicy 6y ago

When do they go live and how can you short them? Ps. I think they never heard of database per service when looking at their microservice graph. Lol

shoo 6y ago
probably safer to buy put options than short the stock (shorting it exposes you to unbounded losses if the stock price rises), but as you say, you'd need to get the timing right. https://finance.yahoo.com/quote/HSBC/options/ https://finance.yahoo.com/quote/HSBC/options/ if they've got a decent QA process and they can't get the new system to actually work reliably there's a fair chance this project would get stuck in QA for a year or two, blocked from going live and cancelled, so it wouldn't cause catastrophic data loss, just reallocate a bunch of money from shareholders to contractors / employees / vendors. > microservice graph banks have graphs of macroservices.
robjan 6y ago
HSBC's own website lists dozens of exchange traded warrants with various strike prices and gearing. Need a brokerage with access to the HK market though. https://www.warrants.hsbc.com.hk/en/tools/search/ucode/00005/type/put https://www.warrants.hsbc.com.hk/en/tools/search/ucode/00005... I'm not sure about their choice of Mongo but the initiative seems like it should be helpful for customers like myself who have accounts in a few countries. We can access them all from one app but the experience is inconsistent at the moment. I'm cautiously optimistic.
- NicoJuicy 6y ago
  Seems like the use case for graphql, not mongodb

_Understated_ 6y ago

I've worked contracts in UK banks for a number of years now. Nothing on the mainframes but I know some of the guys that do work on those and, despite the constant stream of flavour-of-the-month tech that we talk about constantly in places like HN, the mainframes work well and will not be replaced any time soon. The dev cycle for releases and whatnot is incredibly long due to testing and, lets be honest, fear, in case something breaks but when it comes to battle-tested technology, mainframes and DB2 are right up there. I do know of some code that's over 40 years old and still runs at the core of one of the big banks... MongoDB, despite its capabilities, is... an odd choice imo! Not trying to second-guess but if I was in that meeting I would have certainly raised an eyebrow. Good luck to them.

sk0g 6y ago
My prediction is something will go horribly wrong, then they will go "see, we tried modern tech and it fucked us over", and they will commit to seeing their code and tech stack turn a century old before considering any updates.
- AmericanChopper 6y ago
  I read the article and couldn’t figure out what they were actually putting in mongo. If it’s their core processing/ledger then that would be very surprising. But banks have dozens of different systems, some of them would actually almost be suited to Mongo, and some of them have rather loose SLAs. Unless I missed something, the article didn’t really make that clear.
  - refset 6y ago
    There's another thread on the homepage right now about Crux [0], which is a database designed specifically to be able to operate downstream from multiple legacy systems using bitemporal indexes to stitch everything together. This pattern is very common in banks e.g. for calculating and reporting on risk positions. I expect you're right that this story is purely about using Mongo for downstream systems. [0] https://news.ycombinator.com/item?id=23493163 https://news.ycombinator.com/item?id=23493163

AJRF 6y ago

Building in plausible deniability so they can resume their HSBC money laundering services with impunity? Geniuses! Edit: In case you missed the story https://www.bbc.co.uk/news/business-18880269 https://www.bbc.co.uk/news/business-18880269

jauer 6y ago

"is looking to" makes this sound aspirational. The absence of operational lessons learned that would come out of a deployment make me wonder if this actually happened.

hliyan 6y ago
Thank you. The headline reads "HSBC moves..." which in normal news parlance means that they've recently completed the move or the move is in progress.

chrisdbanks 6y ago

The whole premise of this seems wrong. I've worked on large financial systems that have happily supported different rules for different countries using a relational database. The database was never the issue, it was how to handle testing and feature enablement. This seems like the wrong cure to the problem, and a cure that will end up causing more problems than it solves. i.e. a hack. I'd love to hear more about the rational. I just don't understand it.

implying 6y ago

A fortune 100 financial institution I once worked for had an initiative to migrate their extremely large, fairly well normalized DB2 database to schema-less dynamoDB. All new tables had to go into dynamo, and if your service begins consuming an old DB2 table, it had the job of migrating that data to dynamo. The end result of this was applications that half pointed to DB2, half dynamo, confusing new data with old, and tons of bugs relating to losing entries into dynamo tables under outdated keys, not being able to clearly match up customer data, and myriad problems that were never a consideration using a relational database. Needless to say, I pulled all of my assets out of this institution as soon as I had the chance.

QuinnyPig 6y ago
Oh my god I can only imagine how they provisioned their DynamoDB throughput. Setting a dumpster full of cash on fire would have been more expedient.
- implying 6y ago
  The dynamoDB bill ran over $25,000 a month, just for their dev environment. Granted, they had thousands of developers, but still.
threeseed 6y ago
I've spent decades in enterprise space and unfortunately developers often don't have a clue how actually things work. Companies like IBM specialise in royally screwing companies through their licensing deals and with their legal enforcement teams. They are nothing like dealing with a normal startup vendor or with someone like AWS. They play hard ball. So sure you can complain that they should've just stayed with DB2. But usually that means paying tens of millions which then increase each year since they know you won't migrate. That's why companies take drastic decisions and move to the cloud even if it's technically not the best option.
- tluyben2 6y ago
  But drastic can be going to Postgres/Mysql instead of db2/oracle; Dynamo (by the sounds of the case here) is more on the insane scale than drastic.

cranekam 6y ago

I have no actual information about this migration but I wouldn’t be surprised to learn that the databases that are being consolidated are for some minor role or are separate instances with duplicate data for analytics or whatever. It sounds wildly unlikely that they are going to smush together all their customer accounts in one MongoDB instance. As well as being a huge technic challenge I’m sure regulators would have something to say about this.

sjwright 6y ago
This is very likely. Most banks have stupid-huge quantities of products and services, each with extremely elaborate requirements and data storage needs. “65 relational databases” might well only represent a few percentage points of HSBC’s data footprint.
- deleted 6y ago
  [deleted]
bestboy 6y ago
TBH I think so, too. I guess this is just a glorified global network drive to put all the different, country-specific documents like Excel sheets, Word docs, PDFs and some such. https://youtu.be/GI5kwSap9Ug?t=63 https://youtu.be/GI5kwSap9Ug?t=63

andrewksl 6y ago

Sounds like a win-win for them. They get to market themselves as staying up to date with tech (although the HSBC personal banking app I use indicates otherwise). And when they "lose track of" the money they move for the cartel and terrorists, they can blame MongoDB and spare some execs the jail time.

deleted 6y ago
[deleted]
Charlie_26 6y ago
I'm glad someone mentioned cartels and terrorists

edf13 6y ago

I’m in the UK and glad I don’t bank with HSBC

nwmcsween 6y ago

Is it common that a sufficiently large doc db is just an un-normalized rdb? I've seen swagger uis of doc dbs that make me wonder wtf/why.

arcturus17 6y ago

There’s been a lot of talk about the Jepsen-Mongo affair on HN lately, but the problem I have with Mongo and NoSQL in general is much more basic - it’s modeling relationships, which always end up appearing in every data model I’ve ever designed. I’m aware you can keep references from one doc to another but it always ends up being a messy affair even at smallish scales, and it feels as if the cognitive burden of managing these relationships ends up falling on the dev, instead of being managed by the DB. For those of you using MongoDB in large scale large production apps, is this not a problem? Is your underlying business domain really non-relational, or do you manage to comfortably run highly relational models on a doc-based DB? How?

thawkins 6y ago
Mongodb now supports effectivly "Joins" in its aggregation framework, so it can do some relational style data representation. The latest versions also support transactions.
- BozeWolf 6y ago
  Yeah but they do not recommend using that. I think the one nosql afficianado in my team spoke with mongo support and they recommended reassessing our document structure for it. (So a migration after all... ) Also in their blogpost [1], about $lookup they say: We’re still concerned that $lookup can be misused to treat MongoDB like a relational database. But instead of limiting its availability, we’re going to help developers know when its use is appropriate, and when it’s an anti-pattern. In the coming months, we will go beyond the existing documentation to provide clear, strong guidance in this area. [1] https://www.mongodb.com/blog/post/revisiting-usdlookup https://www.mongodb.com/blog/post/revisiting-usdlookup

speedgoose 6y ago

The whole thing sounds like a terrible idea. Why would you want to put all the data from at least 65 different applications in the same table? In MongoDB? To save cost? I simply don't get it. It doesn't make sense to me, but perhaps they know something we don't.

threeseed 6y ago
360 customer view as an example. If you are feeding a real-time decisioning, support, advertising system then you need all of the customer features accessible as fast as humanly possible. Ideally a consistent O(1) time. Or it could be a real-time fraud system where you need every bit of customer data in one payload to feed into an ML model. You can't do any of that if you are having to do multiple joins across federated databases.
- speedgoose 6y ago
  I would collect data and push events to a new system, not migrate 65 relational databases to a single nosql database.

sidcool 6y ago

How do they get ACID guarantees and schema enforcement?

threeseed 6y ago
MongoDB supports transactions and you enforce schemas in code instead of the database. Pretty common these days to just rely on Git and not DDLs.
- thawkins 6y ago
  You can also tie json-schema descriptions to collections and have it throw validation fail exceptions on writes that fail validation.

deleted 6y ago

[deleted]

cm2187 6y ago

And of course they will expose it to the web without authentication!

aogaili 6y ago

Lots of negative comments about MongoDB for folks who don't use it, but perhaps give NoSQL a chance and keep an open mind? there is no need to structure the data into tables and schema does not need to be enforced at the DB level. Google has been running its entire infrastructure on a NoSQL database for years. Perhaps folks need to keep some open mind and see if that works instead of dismissing a new trend?

mrweasel 6y ago
For MongoDB it’s mostly a case of it had issue years ago when it was the new hotness, but it should be a boring choice by now, and it’s not. Neither is Cassandra. Kafka is boring, even more so if they kill of Zookeeper. NoSQL has proven to be hard to get right than was first assumed and at the same time RDBMS’es have improved and adopted many ideas from the NoSQL world faster than expected, often with better results. What’s no clear from the article is why MongoDB was picked, so maybe there’s something that made it the obvious choice, it’s just hard to see what the might be.
- aogaili 6y ago
  "For MongoDB it’s mostly a case of it had issue years ago when it was the new hotness, but it should be a boring choice by now, and it’s not" SQL has been around since birth of computing, give NoSQL some time. "NoSQL has proven to be hard to get right than was first assumed" Nothing has been proven, again give it a chance, SQL had years of optimization. Also, no need to downvote if you like SQL or your job depends on it, or that is all you know, just debate with reason and evidence, I've used both and open to change my mind with evidence and case studies which we don't have many yet and folks here predicting doomsday to HBSC without keeping an open mind.
executive 6y ago
Google also used to not be evil.
arcturus17 6y ago
“Google has been running their entire infrastructure...” I’m skeptical about this - Google’s entire infrastructure is so large that surely they must be using a wide variety of db technologies? I took a databases course that did mention that part of the Google search infrastructure runs on a tailor-made NoSQL technology, but it wasn’t really a doc-based approach, it was more akin to a relational DB but with flexible columns (can’t remember the details sorry). At any rate I don’t feel that this is any vindication for NoSQL, as Google builds much of its underlying tech from scratch and its hardly comparable to the out-of-the-box solutions that the rest of us, even massive corporations like HSBC, have to work with.

deleted 6y ago

[deleted]

ezconnect 6y ago

I think they’re just using it for document storage not customer and financial data. Document storage and retrieval is a big cost for large institution.

niffydroid 6y ago
We found mongo expensive for documents. We had simple documents basically saying X sent email Y to Z. The mongo db was cloud hosted and was over 500gb and costing nearly 3k. Moved them into S3 with a simple SQL dB to look up reference IDs. Certainly reduced the cost!

executive 6y ago

Sounds like the opening scene of a horror movie.

BiteCode_dev 6y ago

It they just put commercial details follow up data + some stats in there, it's ok. It is very infrequently updated, integrity is not at too much risk. In case it's lost, business is not impacted too hard as long as you can find contact details eslewhere. But the replication story is good, and if they got tons of different schemas, it will make their life easier. So yes, for transactions it would be bad, but maybe for this particular use case, it's ok.

predictmktegirl 6y ago

And it’s gone. (Sorry)

egeozcan 6y ago
While I'm generally against getting to meme-y in HN, this is spot-on, adding to the already-long list of ways banks can make your money disappear and/or make unreachable. Reference: https://www.youtube.com/watch?v=_nVk25ZvTkU https://www.youtube.com/watch?v=_nVk25ZvTkU

markmm 6y ago

The new FinTech banks are eating their lunch by providing mobile apps that innovate rapidly due to being build from the ground up with modern tech. The old banks take years to rollout a simple feature like locking your card from the app due to being stuck on mainframes or some ancient stack. Moving to mongodb doesnt sound like the right solution though there are better choices out there.

sitkack 6y ago

This is like one step above csv files on an NFS share. Could be a great way to wash some crimes off the books, "oh our DB failed and our backups were stored in the database as well, oopsie". In less than 5 years there will be an EU directive on how financial institutions need to have immutable db architectures with full provenance.

echopom 6y ago
> In less than 5 years there will be an EU directive on how financial institutions need to have immutable db architectures with full provenance. I'm a former Entreprise Architect from Banking Sector in Europe. To be clear , this will never happen. There has never been any directive that has had a "concrete" impact on IT Architecture of Financial Institutions. Yet there has been dozens of regulations that urged banks to "simplify" their IT Systems. 90% of IT Staff are Baby Boomers with no background in Systems Design or Architecture , thus when new regulation comes in it follow this scenario 100% of the time : - Find a vendor that sell a software that promise compliance with new regulation - Find an integrator that promise integrations within the deadline - Integrate the vendors software in Banks legacy stack of 3000+ monolithic apps - Send report to regulator saying they have "redesigned" their architecture and made "investment" in order to take in account and that new regulation Best examples of this is "PSD2" which has been the biggest fiasco of the industry , has of the last year only 18% of banks complied with the regulation. France and UK said they would not "fine" anyone, because banks "aren't ready" and made "considerable" investment in it. Regardless of what HN thinks , you won't solve Financial Institutions Multi Decade Legacy with a single Directive that would suddenly force them to use "ImMuTABLe Db ARChITectuRES" It Directives have never worked and will never work. The only way to enforce anything would be to remove their IT systems completely and have them use APIs provided by the regulators , and have the regulator become the sole provider of "Financial System". They wont let that happen.
- boruto 6y ago
  I have worked in a German bank and the bundesbank does regulate on very high-level decisions. f.e., systems which do financial crime monitoring need to be compliant with their rules. Access to pre-production should be limited.

jpswade 6y ago

Oh my. This seems like a really bad idea. I'm calling it now, this will go wrong. That "Micro Service Instance" is not a microservice, it's an Enterprise Service Bus (ESB), aka the Egregious Spaghetti Box. That's aside from the fact that relational data belongs in a relational database. I really can't imagine it being a single instance, that just does not make any sense.

ycombonator 6y ago

HSBC IT is a hollowed out outsourced shop. Just like Avis Budget, Disney, Walmart and Walgreens. They will all move to MongoDB data stores sooner than later.

kristaps 6y ago

Each and every one of these MongoDB migration rationales contains some variation of "schema design is hard, let's go shopping." Granted, there are plenty domains where you can get away with having a blurry schema with the occasional bug creating a new localized data anomaly, but banking, where such things lead to customer trust erosion - I just don't get it.

kgc 6y ago

And I am moving all my money out of HSBC...

tannhaeuser 6y ago

Mongo works well for storing chunks of JSON from transactional browser apps (which I guess is what HSBC is targetting), doesn't require you to have custom endpoints, has a good replication/HA story (on paper at least, if your devs know what they're doing and don't use defaults plus multiple nodes with CORS, etc.), good mindshare among webdevs, and is even the de-facto API for JSON stores with AWS' "DocumentDb" and Azure's Cosmos DB being drop-in replacements. But using it for banking/backoffice apps would be pure madness.

buboard 6y ago

If you 've ever used HSBC web banking u ll know it's one of the worst things mankind ever created. I m not sure if this is good news for MongoDB

deleted 6y ago

[deleted]

dark_light 6y ago

Hiding your complexity behind a (poorly designed) layer doesn't make it go away.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

sohamsankaran 6y ago

Jepsen's latest analysis of MongoDB -- https://jepsen.io/analyses/mongodb-4.2.6 https://jepsen.io/analyses/mongodb-4.2.6 -- finds that it doesn't even preserve snapshot isolation when set at the highest consistency level. This seems like a pretty terrible decision. I've never wanted to short a company more in my life.

lowwave 6y ago
Seems like the guys at HSBC tech department are still doing the whole fashion over merit.
- sohamsankaran 6y ago
  What's wild about this is that Mongo isn't even in fashion anymore! NoSQL has been supplanted by NewSQL, and relational dbs are in season again. Even if they were trying to follow the trend, they'd be a half-decade behind.
  - catmanjan 6y ago
    Interesting, what is considered newsql? Postgres?
    - bob1029 6y ago
      SQLite is the hottest new thing on the block.
      - mkl 6y ago
        Huh? SQLite is nearly 20 years old, and is only intended for small scale use.
        bob1029 6y ago
        It was intended as a joke in context, but I believe SQLite is far more capable than most will give it credit for. This is especially true for applications which you only ever intend to run on a single logical machine. But, even for those applications requiring multiple nodes, it can still be a powerful tool if you assume clustering & replication duty in your application logic. For any use case, SQLite is one of the best ways to persist structured data to local disk.

FHermisch 6y ago

Microservices and NoSQL - the wonder-weapons against every IT problem, aren‘t they?! I really have to pivot my business - new name will be „The Microservice-NoSql Consulting Company“ and we will sell tailored versions of the „Microservice-NoSql Strategy“! (Changing names and colors of boxes for 2000$ per day)

jonnypotty 6y ago

Lol. Just lol.

murukesh_s 6y ago

Relax HN guys.. this is the reason for this: "Local requirements for each country will be built into the application, but there's no need to maintain separate data models or separate databases anymore. We could easily design the global data model and database using the MongoDB JSON schema model. That brings data from all operating countries into one database and the application can run on just one database. Which is a lot of reduction in resource and maintenance cost." Is there any Database that can do this? other than MongoDB? I am hearing about data corruption in MogoDB, I believe that is due to bad, out of the box settings, which I believe they would have mitigated when using in cooperation with MongoDB company (which I am sure they are).

joosters 6y ago
I just love software where the out-of-the-box settings lose your data. At least HSBC are 'web scale' now!
BozeWolf 6y ago
Whats the exact advantage, one can wonder. Each country still needs its own application. With its own models. With its own migrations, or at least ways of dealing with missing or mixed data types. What if... those models map to sql database schema’s? wouldn’t that be magical? Not since 2008. Whats left is maintaining multiple databases. That sucks indeed. Sql databases support bson as well these days. What they probably want (or did) is make a base framework, suitable for all countries. And have each country develop its own stuff on it. No need to use mongo for that though. One use case i can imagine is that forms and input just change and that old information never will be compatible again with the newer forms. Mongodb serves as a giant more or less queryable data bin. Then one can ask the poor dba to lookup something for a client with “client_id”: xyz. Then return the raw bson/json output. Might be sufficient.
inshadows 6y ago
POSIX filesystem?

ngcc_hk 6y ago

Old guys ... what dB they use in London and Hong Kong. Are they still use mainframe?

xmorse 6y ago

Hacker News has become just “mongodb bad, sql good”

tluyben2 6y ago

I have seen many rewrites for financial institutions in my life; one of the biggest I had a good insight look at during the entire project was a Tandem Nonstop Cobol rewrite to Java; the famous 10s of millions of lines to nice clean Java code. Many of them never work; this one was, if I remember correctly, upwards of 50m Euros and was thrown away because it did not work in the end. I think the Tandems are still running.

9nGQluzmnq3M 6y ago

The article makes this sound way bigger than it is. HSBC has thousands of applications and probably tens of thousands of databases. Migrating one app to use a different DB model means very little for the big picture.

bob1029 6y ago

I have to say that on the surface, this sounds horrific. - MongoDB: Yes - Micro Services: Yep - Some attempt at a grand unifying model: Of course That's pretty much a 10/10 shitshow as far as I am concerned. Mix in banking regs & regional differences for afterburner on that money furnace. But, in reality, I doubt that one of those 65 relational databases involves the core customer/account/transaction processing facilities. These are probably (hopefully) still running on an IBM system with proper ACID & uptime guarantees. Anything a live transaction flow (especially credit/debit processing) is going to touch could never be trusted on something like MongoDB in this current reality.

sohamsankaran 6y ago
My fear is that those IBM systems running Db2 or something were counted among the 65. While it seems obvious to us that they're better off without switching those, I know of several companies in financial services that should know better that trust MongoDB for their core transaction flow.
- guillaumei 6y ago
  The 65 relational databases are for the same application or similar deployed in 65 countries...At least I read it in that way.
- jeremyjh 6y ago
  They didn't switch anything of consequence. A bank the size of HSBC has thousands of application databases. They never migrate their core transaction systems, they just build piles of shit around them ever higher.

bluestreak 6y ago

I was there in Aug 2019 and I did hear them talking about mongo, There was a programme at the time to "refactor" their estate, a part of that they wanted to replace bunch of ad-hoc databases with clunky- or no data governance with one (or fewer) central places, which will be under Data IT control. The rationale was to reduce risk of data loss, leak or general business continuity issue in what was 6-12 month project. The cost of architecting relational model that would be cover for 65 mini databases in a bank will be astronomical. It is far easier to setup no-SQL entry point that would auto-conform for whatever requirements upstream applications might have. It is a technical win for Mongo, but I don't believe this is an attempt by HSBC to get up to speed with database trends. HSBC are removing internal audit strikes, nothing more than that.

simonswords82 6y ago
We are working with HSBC and have not heard about Mongo being a thing despite working on data with them. What location and department were you in if you don't mind saying?
- bluestreak 6y ago
  This was in London HSBC HQ, Equities
sohamsankaran 6y ago
Fascinating. Are they intending to use this for core transactional workloads? How do they intend to deal with Mongo's (many) known consistency issues?
- bluestreak 6y ago
  I have not worked on this project, so I don't have details. Mongo is in a nutshell to replace a database trader setup under their desk and that grew quite important. There is nothing pretty about that. I'm quite sure they will have issues and workarounds for years to come :(

coding123 6y ago

This is probably some aspect of HSBCs computing needs, likely less than 2% of the data the store.

prostoalex 6y ago

Any time I have to use an HSBC online product (they throw in an occasional promotion for their credit cards and savings accounts) it feels like a trip back to the 90s - popups and extra windows on different domain names that are slowly "loading..." information for trivial operations like scheduling a payment on a credit card.

svrtknst 6y ago

Uh, how does this work with eg GDPR or data protection laws that require all customer data to remain within the country of operation?

29athrowaway 6y ago

This sounds problematic: 1) mongoDB does not offer the guarantees required by a bank. 2) Application developers using mongoDB rarely protect themselves against NoSQL injections. Most of them are unsuspecting that they even exist. e.g.: What is your user id? {$ne: null}... whoops, now every user record is returned. 3) They better make sure they use the Decimal128 type for their currency fields, and make sure that the value is not being casted to float on the client. e.g.: on a JS client, every number is a float. Unless they've taken the necessary precautions this is a disaster waiting to happen. Hopefully other banks do not do the same.

mushi 6y ago
> what is your user id I don’t think you actually tried it. It returns user who’s id is “{$ne: null}” (string)
- 29athrowaway 6y ago
  It depends. Sometimes you can trick an application to parse that as an object.

will_raw 6y ago

I recently came to this article[1] where Guardian switched to PG from MongoDB [1] https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-h...

surajs 6y ago

Yeah this can't go well...

niffydroid 6y ago

I would suspect they'd keep the core money transactions in the existing DBs, but use mongo for less important things like sending junk mail, customer support sessions.

subins2000 6y ago

isn't this a bad idea

jarym 6y ago

I don’t see how one global mongo is going to help solve any of their issues. Maybe they just want to bury their issues under the carpet a little longer. I also hope they’ve configured their mongo correctly since out of the box Config is not appropriate for many use cases

malikNF 6y ago
I want to doubt a billion dollar company like HSBC will have engineers who will install things just out of the box and leave it without properly configuring it. But, they are proposing to use Mongo just almost a month[1] after a report came highlighting several issues with the product, ESPECIALLY for use-cases like a bank that requires a very high level of data consistency, so your point might actually be valid. Would really be interested to know how something like Mongo ends up getting seriously considered for something like this, I mean the whole decision process that goes in to selecting something like this. [1]https://news.ycombinator.com/item?id=23285249 https://news.ycombinator.com/item?id=23285249
- jarym 6y ago
  I did a contract at HSBC in the past. Some teams are good but there are people who will install things out of the box and leave it.. even in a billion dollar company like HSBC.

Keyframe 6y ago

Modern take on shredding papers

historyremade 6y ago

Behind Bad Indian Coder ...

koheripbal 6y ago

This title is misleading, and the article is missing a key piece of information. This is ONE system at HSBC - out of literally HUNDREDS (maybe thousands) of applications. It's not as if HSBC is moving ALL of its applications to MongoDB. HSBC doesn't have a single tech stack. They have thousands of IT employees all using different tech, in different parts of the world, for different departments. This might be as inconsequential as the system that catalogues security camera feed URLs - or maybe the one that monitors remote employee company mobile data usage - who the F knows, because the article gives us zero information. Account Management, Credit Cards, Mortgages, Security, Asset Management, HR, Legal, Compliance, Regulatory, Risk Management, Trading, Operations, Building management, Payroll, ATM comms, Clearance/settlement, Website, a million different reporting engines, etc, etc, etc... and then each region usually has its own application for each function - literally hundreds of tech stacks in every tech you can imagine from DB2 Mainframe to Oracle to MongoDB. All banks are like this. This article is just vague PR and is just referring to one single group consolidating their regional instances. It does not deserve HN attention.

flurdy 6y ago
I initially rebuked thinking that the OP sounds like a disaster, but then I remember from working with other banks that they have an infinite amount of systems from decades of mergers, re-org, etc, all siloed in so many different ways. So in no way was the OP's article as catastrophic as it initially sounded.
- chrisweekly 6y ago
  Your point is clear enough, but FYI you're misusing the word "rebuked". ("recoiled"? "reacted"?)
  - flurdy 6y ago
    I think I merged two sentences and feelings into one. :)
tcgv 6y ago
The article is very shallow on technical details indeed. It seems they are trying to promote MongoDB as the solution to their failed attempt of standardizing their application/data model: > Historically, (...) HSBC did have an application core program environment, which had most of an application's core functionality. But it couldn't have a single programme environment running for all the countries, due to the differences in data models and databases. > Another benefit is to use the same database for global data analytics and reporting. We don't need to translate into another data model or another database to run the analytics and reporting from that particular data. So this "application core program" failed to provide an unified solution to their data warehousing needs, and now they believe they can solve this problem by unifying 65 databases into one MongoDB instance, "taking advantage" of its schema-less model. It looks more like a global storage service than anything else to me.

akho 6y ago

65 databases unified globally, for HSBC’s global presence, is ~2 databases per country. This is, probably, a minor trial effort for one of their services, not the core system (which is un-unifiable due to banking privacy laws in place) or any kind of cross-function unification.

rmoriz 6y ago

I think this isn't primarily about a database migration/consolidation and specifically not about MongoDB. IMHO it's about cutting cost and complexity by building a single world-wide software platform. Of course this also means that local specialities and local teams will be questioned and probably be made redundant.

jsjohnst 6y ago

What’s the over/under odds on betting about how long it’ll be until it’s accidentally exposed to the internet and hacked?

trollied 6y ago

This article is misleading. They have migrated some of their databases to mongo, not all of them

addicted 6y ago

There’s a lot of complaining here but MongoDB has been used by startups for like a decade? I don’t think I’ve seen a single HN article complaining about actual data loss. That would be something that would get upvoted immediately. So what gives? Especially since earlier and older versions of Mongo apparently had far less data stability. I’ve probably read far more complaints about Postgres in HN articles (difficulty setting up, poor defaults, etc). And Postgres may not even be as popular as Mongo. So what gives?

mariopt 6y ago

I can see how they would use in some parts: analytics, internal reports, etc. It is interesting that you can use Mongo and keep the data in a geographical location and still be able to run a global query: https://docs.mongodb.com/manual/tutorial/sharding-segmenting-data-by-location/ https://docs.mongodb.com/manual/tutorial/sharding-segmenting...

ionwake 6y ago

Wasn’t there a problem where mongo would literally lose records under high loads ? There was an article on it about it a few years ago. Did they fix that?

blaufast 6y ago

That was a mistake

anonu 6y ago

I need to LOL a little bit at all this, the headline, the article. First off, how would such an article even come about? Feels like more of a recruiting/marketing piece. The old story at hsbc was that the ATM software hadn't been updated in twenty years, cause stability....

seanhunter 6y ago

To put this into context - I met with a senior tech person there (one layer below the group CIO) a few years back and he said they had something like 40000 distinct IT systems globally. So replacing 65 of them with one MongoDB instance is something like 1/8th of one percent of their systems.