6 ms·

I left the following comment at the blog: For this example, idiomatic Pandas would look like this: def gen_stats_pandas2(dataset_pandas): grouped = da

• apl 9y ago
  Thanks for coding this up. The code in the blog post is, at best, incredibly misleading.
• minimaxir 9y ago
  It's especially weird in this case because the value-proposition of pandas is the ability to do things like aggregation in 3 lines of code and not have to iterate through every row.
  • Radim 9y ago
    Well pandas iterates through every row anyway, just in native-land rather than pure Python. That's not the difference. The trouble with this code was the choice of the quadratic algorithm (which the idiomatic pandas version avoids). You could achieve a similar speed-up as OP for the pure Python code too, by dropping the doubly-nested loop. The point stands as usual: better algo > clever tricks :) Personally, I find the code brevity that pandas offers does not outweigh the added complexity and cognitive load of its (IMO hard to remember and reason about) API. EDIT: pure Python version def gen_stats_python2(dataset_python): aggr = {} for product_id, product_order_num, quantity, price in dataset_python: item = aggr.get(product_id, [0, []]) item[0] += quantity item[1].append(price) aggr[product_id] = item return [ (int(product_id), len(prices), int(quantity), round(float(sum(prices)) / len(prices), 2)) for product_id, (quantity, prices) in aggr.items() ] 761x speedup! %timeit gen_stats_python(dataset_python) 1 loops, best of 3: 36.3 s per loop %timeit gen_stats_python2(dataset_python) 10 loops, best of 3: 47.7 ms per loop
    • iClaudiusX 9y ago
      It's a lot faster if you use python's itertools and operator itemgetter: def gen_stats(data): start = time.time() stats = [] data.sort(key=itemgetter(0)) for p_id, p_items in itertools.groupby(data, key=itemgetter(0)): num_orders, _, total_quantity, total_price = map(sum, zip(*p_items)) p_id, num_orders, total_quantity = map(int, [p_id, num_orders/p_id, total_quantity]) stats.append([p_id, num_orders, total_quantity, round(float(total_price/num_orders),2)]) end = time.time() working_time = end-start return stats, working_time Results: Dataset size 55096 records [[1, 8, 340, 22.55], [2, 4, 94, 15.83], [3, 2, 97, 3.89]] [[1, 8, 340, 22.55], [2, 4, 94, 15.83], [3, 2, 97, 3.89]] All results equal: True Runtime (s) gen_stats_slow: 28.489468574523926 gen_stats_fast: 0.06000018119812012
      • Radim 9y ago
        That doesn't look faster -- 60ms? Nor clearer, at least to me :) Additionally, you might have a bug in the code: groupby() only groups consecutive keys, so your code will fail if the dataset is not already sorted by product_id. I guess that's the lesson here -- use pandas to make sure you get a sane, fast, tested implementation of some common use-cases. That is, if you can remember its API, when it makes copies vs views under the hood, and all that gnarly stuff :-)
      • eesmith 9y ago
        Wee! Pointless optimization Saturday! Looks like my gen_stats_eesmith is even faster. Radim's is about 400x faster, iClaudiusX is 450x faster, and eesmith is 580x faster. Under pypy2-v5.8.0 those are 65x, 16x, and 91x, respectively. The original algorithm is 10x faster under pypy. def gen_stats_python(dataset_python): start = time.time() product_stats = [] unique_products = set([x[0] for x in dataset_python]) for product_id in unique_products: product_items = [x for x in dataset_python if x[0]==product_id ] num_orders = len(product_items) total_quantity = 0 total_price = 0 for row in product_items: total_quantity += row[2] total_price += row[3] avg_price = float(total_price/num_orders) product_stats.append([int(product_id),int(num_orders),int(total_quantity),round(avg_price,2)]) end = time.time() working_time = end-start return product_stats,working_time def gen_stats_radim(dataset_python): start = time.time() aggr = {} for product_id, product_order_num, quantity, price in dataset_python: item = aggr.get(product_id, [0, []]) item[0] += quantity item[1].append(price) aggr[product_id] = item return [ [int(product_id), len(prices), int(quantity), round(float(sum(prices)) / len(prices), 2)] for product_id, (quantity, prices) in aggr.items() ], time.time()-start def gen_stats_iclaudiusx(data): start = time.time() stats = [] data.sort(key=itemgetter(0)) for p_id, p_items in itertools.groupby(data, key=itemgetter(0)): num_orders, _, total_quantity, total_price = map(sum, zip(*p_items)) p_id, num_orders, total_quantity = map(int, [p_id, num_orders/p_id, total_quantity]) stats.append([p_id, num_orders, total_quantity, round(float(total_price/num_orders),2)]) end = time.time() working_time = end-start return stats, working_time def gen_stats_eesmith(dataset_python): start = time.time() product_stats = [] total_num_orders = defaultdict(int) total_quantity_by_product_id = defaultdict(int) total_price_by_product_id = defaultdict(float) for product_id, order_id, quantity, price in dataset_python: total_num_orders[product_id] += 1 total_quantity_by_product_id[product_id] += quantity total_price_by_product_id[product_id] += price for product_id in total_price_by_product_id: num_orders = total_num_orders[product_id] total_quantity = total_quantity_by_product_id[product_id] total_price = total_price_by_product_id[product_id] avg_price = float(total_price)/num_orders product_stats.append([int(product_id),int(num_orders),int(total_quantity),round(avg_price,2)]) end = time.time() working_time = end-start return product_stats,working_time reference_result = reference_dt = None gen_stats_functions = (gen_stats_python, gen_stats_radim, gen_stats_iclaudiusx, gen_stats_eesmith) for gen_stats_function in gen_stats_functions: result, dt = gen_stats_function(dataset_python) if reference_result is None: reference_result = result reference_dt = dt assert result == reference_result, (gen_stats_function.__name__, result[:2], reference_result[:2]) print(gen_stats_function.__name__, dt, reference_dt/dt)
• blattimwind 9y ago
  Sometimes I wonder whether actual optimization (as a craft, if you like) has been poisoned by scripting languages (Python etc.) and frameworks. For one thing they completely changed how the speed of computers is perceived, e.g. it is actually considered fast when a web app spends "only" 20 server CPU milliseconds on a request. And because most things are so inefficient, it is relatively easy to get integer speed-ups. When looking at applications written in C++, written by actual engineers, then often just getting 5 or 10 % on some functionality is appreciated, and achieving more can be seriously difficult. With software in scripting languages improvements below a few hundred percent don't raise an eye brow. However, actually optimizing code in a scripting language (and not changing the way a native library is used) can be seriously difficult and annoying, because most simply don't have efficient tools — obviously, if a scripting language (like Python) had something substantially faster than, say, attribute access, then that something would simply be used to power attribute access in the general case. So in C++ (or other native languages) you can actually make good use of the processor's and memory system's resources, but you just can't writing in a scripting language. It is also surprising that most tooling available for e.g. Python is rather immature, insufficient and cumbersome to use compared to the tooling (especially commercial) available for native code (or Java). This likely has something to do with people a) not caring about performance at all and/or b) since it is basically not possible to write fast Python code, as soon as something becomes important for performance, it is written in another language and used from Python, and for analyzing the performance of that code, which is now an external library, you don't need tools for Python.
  • minimaxir 9y ago
    Ad-hoc statistical analysis that takes a few seconds to run is a different technical case than production environments where every second is critical. For most datasets/workflows, as long as it can fit into a few GB of memory, it can run on a computer in a reasonable amount of time on any modern data analysis library like pandas or R's dplyr without having to dive into bare metal. The exceptions are if you are coding your algorithm very poorly (like in this case) and have bad time complexity that causes the run time to balloon at large amounts of data.
  • stestagg 9y ago
• azag0 9y ago
  Was the comment redacted?