I made the mistake of writing a searchable syslog archiver where the parsing was done in stored functions (very basic string-splitting) and the log entries were

I made the mistake of writing a searchable syslog archiver where the parsing was done in stored functions (very basic string-splitting) and the log entries were stored in a heavily indexed Postgres table. At the time it was created, the company was in its infancy, there wasn't much throughput, the service emitting the logs was kind of obscure, and the solution was a quick, dirty stopgap meant to be easily digestible and consumable to the customer, compatible with the rest of their in-house knowledge and the rest of their backend stack. However, as happens too often, the company grew meteorically, the volume got to be insane, many more default-on log entries were added to the standard event loop, and the system was put straight into production without any interest in redesign; after all, it worked fine so far. As you might guess, it got to the point where we were seeing insanely degraded performance and cascade failures after only a few days' log activity, and increasing time and resources were devoted to the endless care and feeding of this beast. It took a surprisingly long time for the rubber band to snap and for the customer to consider ELK, despite the fact that I advocated for this almost from the very beginning.