I worked for a job search company that bought all of their listings from a third party. Every day, they would get an XML feed of active job listings that was a

I worked for a job search company that bought all of their listings from a third party. Every day, they would get an XML feed of active job listings that was a few GB in size, from what I remember. To update their database, they had a program (written in C++ for speed!) that would go through each one of today's postings and scan the old list to see if it was new, then write update instructions to a SQL script that would be executed at the end of the run. (According to a senior developer there, they didn't just interact with the database directly because the database couldn't handle updating large amounts of data (???)) It would take about 8 hours every day for the program to run. So for 8 hours of every day, the company would be returning stale listings. I rewrote it in PHP and brought the time down to ~20 minutes, which could have still been significantly improved. But I didn't stick around for too long after that.