3 ms·
This reminds me of the company I worked at that was paying for dozens of unused Redshift clusters at any one time, peaking over a hundred. There were common dat
by dkarl 3y ago
This reminds me of the company I worked at that was paying for dozens of unused Redshift clusters at any one time, peaking over a hundred. There were common data science (read: ETL configuration) tasks that required spinning up a Redshift cluster, and the "data engineers" (read: fresh grads with a few weeks of training) sometimes forgot to shut them down afterwards, or got diverted to different work temporarily and couldn't find the original cluster when they returned to complete the task.
The first proposed solution was to ask the forgetful data engineers to manually tag all the resources they created, so un-tagged resources could be deleted promptly. When we asked how the tagging scheme would allow us to distinguish between clusters that had been forgotten and clusters that were being actively used, we were told that it was not necessarily a complete solution, but it was "a step in the right direction." Ultimately the problem was declared solved, but I wonder if it really was.
The company was swimming in VC money, but even so, that level of waste (which was consistent across all of our astronomical AWS spend) was impossible to ignore. Checking back now, all of the engineering jobs they're hiring for are overseas, so they must have ultimately nuked the entire department.