5 ms·
It's especially weird in this case because the value-proposition of pandas is the ability to do things like aggregation in 3 lines of code and not have to itera
by minimaxir 9y ago
It's especially weird in this case because the value-proposition of pandas is the ability to do things like aggregation in 3 lines of code and not have to iterate through every row.
- Radim 9y agoWell pandas iterates through every row anyway, just in native-land rather than pure Python. That's not the difference. The trouble with this code was the choice of the quadratic algorithm (which the idiomatic pandas version avoids). You could achieve a similar speed-up as OP for the pure Python code too, by dropping the doubly-nested loop. The point stands as usual: better algo > clever tricks :) Personally, I find the code brevity that pandas offers does not outweigh the added complexity and cognitive load of its (IMO hard to remember and reason about) API. EDIT: pure Python version def gen_stats_python2(dataset_python): aggr = {} for product_id, product_order_num, quantity, price in dataset_python: item = aggr.get(product_id, [0, []]) item[0] += quantity item[1].append(price) aggr[product_id] = item return [ (int(product_id), len(prices), int(quantity), round(float(sum(prices)) / len(prices), 2)) for product_id, (quantity, prices) in aggr.items() ] 761x speedup! %timeit gen_stats_python(dataset_python) 1 loops, best of 3: 36.3 s per loop %timeit gen_stats_python2(dataset_python) 10 loops, best of 3: 47.7 ms per loop
- iClaudiusX 9y agoIt's a lot faster if you use python's itertools and operator itemgetter: def gen_stats(data): start = time.time() stats = [] data.sort(key=itemgetter(0)) for p_id, p_items in itertools.groupby(data, key=itemgetter(0)): num_orders, _, total_quantity, total_price = map(sum, zip(*p_items)) p_id, num_orders, total_quantity = map(int, [p_id, num_orders/p_id, total_quantity]) stats.append([p_id, num_orders, total_quantity, round(float(total_price/num_orders),2)]) end = time.time() working_time = end-start return stats, working_time Results: Dataset size 55096 records [[1, 8, 340, 22.55], [2, 4, 94, 15.83], [3, 2, 97, 3.89]] [[1, 8, 340, 22.55], [2, 4, 94, 15.83], [3, 2, 97, 3.89]] All results equal: True Runtime (s) gen_stats_slow: 28.489468574523926 gen_stats_fast: 0.06000018119812012
- Radim 9y agoThat doesn't look faster -- 60ms? Nor clearer, at least to me :) Additionally, you might have a bug in the code: groupby() only groups consecutive keys, so your code will fail if the dataset is not already sorted by product_id. I guess that's the lesson here -- use pandas to make sure you get a sane, fast, tested implementation of some common use-cases. That is, if you can remember its API, when it makes copies vs views under the hood, and all that gnarly stuff :-)
- eesmith 9y agoWee! Pointless optimization Saturday! Looks like my gen_stats_eesmith is even faster. Radim's is about 400x faster, iClaudiusX is 450x faster, and eesmith is 580x faster. Under pypy2-v5.8.0 those are 65x, 16x, and 91x, respectively. The original algorithm is 10x faster under pypy. def gen_stats_python(dataset_python): start = time.time() product_stats = [] unique_products = set([x[0] for x in dataset_python]) for product_id in unique_products: product_items = [x for x in dataset_python if x[0]==product_id ] num_orders = len(product_items) total_quantity = 0 total_price = 0 for row in product_items: total_quantity += row[2] total_price += row[3] avg_price = float(total_price/num_orders) product_stats.append([int(product_id),int(num_orders),int(total_quantity),round(avg_price,2)]) end = time.time() working_time = end-start return product_stats,working_time def gen_stats_radim(dataset_python): start = time.time() aggr = {} for product_id, product_order_num, quantity, price in dataset_python: item = aggr.get(product_id, [0, []]) item[0] += quantity item[1].append(price) aggr[product_id] = item return [ [int(product_id), len(prices), int(quantity), round(float(sum(prices)) / len(prices), 2)] for product_id, (quantity, prices) in aggr.items() ], time.time()-start def gen_stats_iclaudiusx(data): start = time.time() stats = [] data.sort(key=itemgetter(0)) for p_id, p_items in itertools.groupby(data, key=itemgetter(0)): num_orders, _, total_quantity, total_price = map(sum, zip(*p_items)) p_id, num_orders, total_quantity = map(int, [p_id, num_orders/p_id, total_quantity]) stats.append([p_id, num_orders, total_quantity, round(float(total_price/num_orders),2)]) end = time.time() working_time = end-start return stats, working_time def gen_stats_eesmith(dataset_python): start = time.time() product_stats = [] total_num_orders = defaultdict(int) total_quantity_by_product_id = defaultdict(int) total_price_by_product_id = defaultdict(float) for product_id, order_id, quantity, price in dataset_python: total_num_orders[product_id] += 1 total_quantity_by_product_id[product_id] += quantity total_price_by_product_id[product_id] += price for product_id in total_price_by_product_id: num_orders = total_num_orders[product_id] total_quantity = total_quantity_by_product_id[product_id] total_price = total_price_by_product_id[product_id] avg_price = float(total_price)/num_orders product_stats.append([int(product_id),int(num_orders),int(total_quantity),round(avg_price,2)]) end = time.time() working_time = end-start return product_stats,working_time reference_result = reference_dt = None gen_stats_functions = (gen_stats_python, gen_stats_radim, gen_stats_iclaudiusx, gen_stats_eesmith) for gen_stats_function in gen_stats_functions: result, dt = gen_stats_function(dataset_python) if reference_result is None: reference_result = result reference_dt = dt assert result == reference_result, (gen_stats_function.__name__, result[:2], reference_result[:2]) print(gen_stats_function.__name__, dt, reference_dt/dt)
- NumberCruncher 9y ago>> Well pandas iterates through every row anyway The power of pandas comes - partly - from vectorized operations. This means that instead of iterating through every row it processes the data in data sets transforming the operations into matrix/vector manipulation. For matrix manipulation it uses low lewel high performat packages borrowed from c, numpy, etc. Writing "pandas code" not using the pandas operations is like eating soup with a fork. It works but it is pretty slow.
- Radim 9y agoEven vectorized operations still have to go through every data item. I'm not sure what you imagine happens inside pandas, but I assure you it's not magic.
- deleted 9y ago[deleted]