4 ms·
That brings back an old memory of the data warehouse being extremely overburdened during peak. Capacity constraints and over utilization kept bringing down clus
by saosebastiao 8y ago
That brings back an old memory of the data warehouse being extremely overburdened during peak. Capacity constraints and over utilization kept bringing down clusters. As a last ditch effort, the data warehouse team started randomly disabling queries en masse, under the assumption that if they were actually mission critical, the user would just re-enable them again.
I knew some engineers that worked on a centralized data engineering team that served 100+ software teams, and they managed several thousand scheduled queries. I felt so bad for the guy that had pager duty that first night. He said he got a sev-2 every 12 minutes on average for 24 hours straight.
I certainly hope Amazon has fixed their data warehouse issues since then.
- newguyatamazon 8y agoI've only been here for a few months (BI engineer), but overall uptime for all data platforms has been good and things run smooth pretty much 24/7. Extremely impressive to me especially considering the breadth and depth of some of these datasets. I do have my gripes about how things are laid out here (layers upon layers of abstraction + codewords for everything + some big tooling changes not being documented or still tribal knowledge), but all in all it's a well-oiled machine here