4 ms·

At my company, we operate on very large amounts of data. Let's pretend, though, that we woke up this morning and had a 600mb dataset. We'd still have hundreds

At my company, we operate on very large amounts of data. Let's pretend, though, that we woke up this morning and had a 600mb dataset. We'd still have hundreds of jobs running on that 600mb dataset. Most of them would ideally run as often as possible--we'd only schedule them periodically because of performance. Some of them would explode the 600mb dataset into several gigs. Most of them would copy the dataset several times over. Some of them would be memory intensive. Some of them would be CPU intensive. Some of them would be disk intensive. Many of those jobs emanate counters, logs, and metrics, which we want to track over time. We'd want to track the overall resource utilization of those jobs. We'd want the particular jobs to be killed automatically if they use too many resources. If the jobs are in danger of taking up too many resources in the system, we want them to be automatically queued and scheduled piecemeal. We'd want the jobs to be written in a variety of frameworks targeted to the particular task at hand. Given those hundreds of jobs competing for scarce resources, we'd want to not have to completely rewrite the system if the 600mb dataset became a 1 gig dataset, or a 1 tb dataset. Sure, we could put together a multi server ssh/bash job scheduling system, but in the end that's kind of what hadoop is, except having addressed a lot of problems that you don't even know you're going to have up front.

lmm 11y ago
With a 600mb dataset you probably don't need batch jobs. You can afford to run ad-hoc reports when they're requested. You can process data as it arrives, streaming it into a structured representation in postgres or similar. You can afford to stop the pipeline when you get an item whose format doesn't match, and investigate, rather than having to tolerate a certain error percentage. You don't need a scheduling framework because if you ever have too many jobs running you can kill a few, you're not going to lose hours of work by doing so. You want metrics but IME hadoop isn't actually very good at that. If you're doing enough processing that you can't do it on one beefy database server then sure, the concerns that go with that are similar to when you have enough data that you can't handle it on one beefy database server. But most people with 600mb of data don't want or need to do multiple cpu-hours worth of processing on it.

lmm 11y ago

With a 600mb dataset you probably don't need batch jobs. You can afford to run ad-hoc reports when they're requested. You can process data as it arrives, streaming it into a structured representation in postgres or similar. You can afford to stop the pipeline when you get an item whose format doesn't match, and investigate, rather than having to tolerate a certain error percentage. You don't need a scheduling framework because if you ever have too many jobs running you can kill a few, you're not going to lose hours of work by doing so. You want metrics but IME hadoop isn't actually very good at that. If you're doing enough processing that you can't do it on one beefy database server then sure, the concerns that go with that are similar to when you have enough data that you can't handle it on one beefy database server. But most people with 600mb of data don't want or need to do multiple cpu-hours worth of processing on it.