3 ms·

Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have

by kitd 11y ago

Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have very different requirements so a combination of a tool that can do all the data handling easily (ie Python) + a tool that can throw the CPU at a numerical problem (ie C) will work as a great combination. In contrast, if you work in Java, you are trying to use the same tool for both jobs, and you may well fall between 2 stools. And I say that as a typical Java-head. My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node? As for the number crunching, I think Rust would be a better choice here. Good memory management is its USP and that can have significant performance benefits.

iamsohungry 11y ago
> My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node? Absolutely not. Python has a much more mature set of libraries which are much better designed, better tooling, and a much more reasonable type system, and the community has only recently begun to be polluted by Web 2.0 "move fast and break things" mentality. The Node ecosystem is built on that mentality, and while outlier developers at the front line of the JS community are able to be extraordinarily productive in JS/Node, developers building user-facing programs just get bogged down in breaking changes in dependencies, buggy and poorly-designed 0.x libraries, untraceable framework code that prioritizes configuration over convention, and intractable errors caused by a broken type system (this final issue is significantly improved in ES6). Source: Worked in Python 2.x for a few years, worked in Node for a few years, currently work in Python 3.x and Node (general design is to limit Node to compiling and testing the browser-based part of our product). JS/Node is maybe 20% of our code, 20% of our features, 50% of our dev time, and 80% of our bugs.
- kitd 11y ago
  Yes you're right, Node wouldn't be good. Then I remembered Perl and realised that data pre-/post-processing is precisely why it was invented in the first place. Perl + Rust would be an interesting combo IMHO.
spot 11y ago
Agreed. It is often true that different parts of a problem are best solved in different languages. That's one reason we built Beaker and released it as open source: http://BeakerNotebook.com http://BeakerNotebook.com. Its polyglot architecture and autotranslation makes working with multiple languages easy.
mr_overalls 11y ago
Julia could be a contender once a few bugs are worked out.

by kitd 11y ago

iamsohungry 11y ago

> My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node? Absolutely not. Python has a much more mature set of libraries which are much better designed, better tooling, and a much more reasonable type system, and the community has only recently begun to be polluted by Web 2.0 "move fast and break things" mentality. The Node ecosystem is built on that mentality, and while outlier developers at the front line of the JS community are able to be extraordinarily productive in JS/Node, developers building user-facing programs just get bogged down in breaking changes in dependencies, buggy and poorly-designed 0.x libraries, untraceable framework code that prioritizes configuration over convention, and intractable errors caused by a broken type system (this final issue is significantly improved in ES6). Source: Worked in Python 2.x for a few years, worked in Node for a few years, currently work in Python 3.x and Node (general design is to limit Node to compiling and testing the browser-based part of our product). JS/Node is maybe 20% of our code, 20% of our features, 50% of our dev time, and 80% of our bugs.

kitd 11y ago
Yes you're right, Node wouldn't be good. Then I remembered Perl and realised that data pre-/post-processing is precisely why it was invented in the first place. Perl + Rust would be an interesting combo IMHO.

spot 11y ago

Agreed. It is often true that different parts of a problem are best solved in different languages. That's one reason we built Beaker and released it as open source: http://BeakerNotebook.com http://BeakerNotebook.com. Its polyglot architecture and autotranslation makes working with multiple languages easy.

mr_overalls 11y ago

Julia could be a contender once a few bugs are worked out.