4 ms·
> Would somebody have some benchmarks against pandas for some standard operations ? pandas creator here. Numba is a complementary technology to pandas, so you
by wesm 9y ago
> Would somebody have some benchmarks against pandas for some standard operations ?
pandas creator here. Numba is a complementary technology to pandas, so you can and should use them together. It is designed for use with NumPy arrays and so does not deal with missing data and other things that pandas does. It does not help as much with non-numeric data types.
- t8ge55geu8ygt 9y agoAre you saying that if you have missing data you can't use numba, or that pandas will handle that part which numba couldn't otherwise do alone?
- j88439h84 9y agoAre you saying if that you have missing data you can't use numba, or if you have missing data, and you use numba together with pandas, that pandas will handle the missing data where numba alone could not?
- grej 9y agoHeavy numba user here. What Wes is saying is that while Pandas handles some of those missing values in an automated way, if you choose to use numba it uses numpy arrays so you may have to handle some of those things yourself. I have at times used a separate numpy array to indicate whether values are missing or not. You could also use a value which is far out of the bounds of what you might ever see in your real data, then test for that while you're looping over those values (eg. fill missing values with -3.4E38 if you have a float32). Depending on what you're doing, you might be able to use numpy.nan as a value. It does work inside of numpy arrays. But some methods that operate on those objects might not work as you expect. For instance, if you run numpy.mean on a numpy array of [nan, 4, 5], it will return nan. If you run the same thing on a pandas dataframe of the same values, you'll get 4.5.