4 ms·
Sometimes I wonder whether actual optimization (as a craft, if you like) has been poisoned by scripting languages (Python etc.) and frameworks. For one thing t
by blattimwind 9y ago
Sometimes I wonder whether actual optimization (as a craft, if you like) has been poisoned by scripting languages (Python etc.) and frameworks.
For one thing they completely changed how the speed of computers is perceived, e.g. it is actually considered fast when a web app spends "only" 20 server CPU milliseconds on a request.
And because most things are so inefficient, it is relatively easy to get integer speed-ups. When looking at applications written in C++, written by actual engineers, then often just getting 5 or 10 % on some functionality is appreciated, and achieving more can be seriously difficult. With software in scripting languages improvements below a few hundred percent don't raise an eye brow.
However, actually optimizing code in a scripting language (and not changing the way a native library is used) can be seriously difficult and annoying, because most simply don't have efficient tools — obviously, if a scripting language (like Python) had something substantially faster than, say, attribute access, then that something would simply be used to power attribute access in the general case. So in C++ (or other native languages) you can actually make good use of the processor's and memory system's resources, but you just can't writing in a scripting language.
It is also surprising that most tooling available for e.g. Python is rather immature, insufficient and cumbersome to use compared to the tooling (especially commercial) available for native code (or Java). This likely has something to do with people a) not caring about performance at all and/or b) since it is basically not possible to write fast Python code, as soon as something becomes important for performance, it is written in another language and used from Python, and for analyzing the performance of that code, which is now an external library, you don't need tools for Python.
- minimaxir 9y agoAd-hoc statistical analysis that takes a few seconds to run is a different technical case than production environments where every second is critical. For most datasets/workflows, as long as it can fit into a few GB of memory, it can run on a computer in a reasonable amount of time on any modern data analysis library like pandas or R's dplyr without having to dive into bare metal. The exceptions are if you are coding your algorithm very poorly (like in this case) and have bad time complexity that causes the run time to balloon at large amounts of data.
- stestagg 9y agoThis comment contains some fairly pejorative language I can assure you that there are real-world high performance python systems with per-event budgets of far less than 1ms being used for critical systems in large organisations. Sometimes with the help of compiled components, but made understandable by being coordinated in python. Of course these systems are atypical, but the idea that ‘actual’ engineers always use c/Java, while python is just a scripting language is just ignorant
- blattimwind 9y agoThat's not what I wrote.
- pwang 9y agoWhen Python is being used for these data analytical tasks, random attribute access is not the common case. Instead, SIMD and vectorized mathematical transformations is the order of the day, and those can be specified very concisely and powerfully within the syntax of a "scripting" language. More importantly, that conciseness means that that language is accessible to people who understand the right maths to use. This is frequently not your CS-trained C++/Java software engineer. It's the age-old philosophical quandary: is it more useful to have the fastest bomber that can only be piloted by specialized pilots who must receive targeting instructions via long-distance radio, or is it better to have a moderately fast bomber than can be piloted by people who can actually see the targets? Python is succeeding and taking over the world because businesses are overwhelmingly seeing the value of the latter approach.