3 ms·
> We also don't know how big the dataset is or how often it changes. I'm going to take a stab at small and infrequently. Every 2-3 months we had to execute a
by cs02rm0 7y ago
> We also don't know how big the dataset is or how often it changes.
I'm going to take a stab at small and infrequently.
Every 2-3 months we had to execute a python script that takes 1s on all our data (500k rows), to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done.
- thaumaturgy 7y agoYeah, probably. But we shouldn't be calling these guys out for not taking the "obvious and simple" solution when we aren't 100% certain that it would actually work. That happens too often on HN, and then sometimes the people involved pop in to explain why it's not so simple, and everyone goes "...oh." Seems like we should learn something from that. I've gone with "don't assume it's as simple as your ego would lead you to believe."