5 ms·
And if it ends up being un-important, what then? My contention here is that if you can't pursue small ideas that might end up being unimportant, you're limiting
by sritchie 15y ago
And if it ends up being un-important, what then? My contention here is that if you can't pursue small ideas that might end up being unimportant, you're limiting yourself creatively in a big way.
These methods exemplify the sort of complexity I was getting at in the blog post; how many tests do you end up writing if you need a full debug server?
Midje and Cascalog make it possible to do test-driven development at large scale. Is anyone else in MapReduce able to practice TDD? Really curious here.
- moultano 15y agoI find writing a debug server easier than writing tests, but maybe that's just me. :) Especially for speculative things, the notion of whether the code is "correct" could change from moment to moment, so I find it much more useful to look at what it's doing than to test if what it's doing matches a spec that doesn't exist.
- sritchie 15y agoHey, if it works, no problems here. Personally, I've found it really helpful to be able to think of and test queries as pure functions (without side effects related to data storage). Composable, pure queries keep our code base really small, and allow us to code and test features in hours. I end up writing and testing a lot of queries that I never use in production, but it's this play that allows the good stuff to bubble up.
- deleted 15y ago[deleted]
- mitultiwari 15y agoI use Pig to write many of my Hadoop Mapreduce jobs and test my Pig script on a small dataset using Pig Grunt. Pig 0.9 has a lot more debug information as well.