Firstly, that does sound fairly insane. Barring any unusual computational requirements, your data is several orders of magnitude smaller than the 'big data' sta

Firstly, that does sound fairly insane. Barring any unusual computational requirements, your data is several orders of magnitude smaller than the 'big data' stack it seems like you're using. If you can quickly set up a simple version to test with any of the rest of the team that are willing, that'd be my first port of call. A simple ETL job to pull in fresh data from your S3 store and push it into Postgres. Then you can hopefully get everyone else on board with how much easier this setup will make their lives, and build a consensus that this migration would be a good idea. A quick idea re. Postgres and 'quick enough' vs. 'fast enough' - what are your requirements around data freshness for the models. Can you take data from a read replica or even your DB backups (extra points for letting you test your backups) for your model training workloads to keep load off the main instance? W.r.t. the 'It takes several days to explain where all the data is' - in the first instance, I'd draw up a shared spreadsheet listing all this stuff. Yes it's best case going to be eventually consistent with the actual data you hold, but it should cut down the time spent regurgitating the same information over and over and gives everyone a central point to store it. This will be much easier and faster to implement than a full blown data catalog, which you could look at doing once you've got the 'duct tape' solution going. If you want to chat, email is in profile.