4 ms·

Agreed. One big win here over numpy is that there's the `push!` function on Julia arrays (equivalent to the C++ vector's `push_back`). Numpy arrays on the othe

Agreed. One big win here over numpy is that there's the `push!` function on Julia arrays (equivalent to the C++ vector's `push_back`). Numpy arrays on the other hand are awkward to use in custom algorithms because once the memory is reserved, you can't grow the array at either end.

rjeli 10y ago
Yep, a problem I repeatedly run into with Numpy is that my data is too big to fit in memory as Python data, but can easily fit as Numpy matrices. It seems incrementally building Numpy matrices is very awkward and inefficient edit: if anyone knows a solution to this, please let me know...
- tgb 10y ago
  I think the standard recommendation is to know how much you need to allocate and allocate it in advance. You can call numpy.zeros( size ) to get the appropriate array, then fill it with your actual data, for example. If you don't know how much you need, you can try to over-estimate it and then shrink the array (i.e. allocate one of the right size and copy) after your done.
- tempay 10y ago
  Where are you filling your matrix from?
- inlineint 10y ago
  Under the hood this `push!` function have to do memory reallocation and copying the entire array when the array grows to be larger than some internal buffer, which happens each time the array becomes larger than a^n, where a is usually, but not necessary, selected to be equal to 2. The amortized complexity is θ(1), but individual insertions might take O(n) time. It means that it is not hard to have the same in numpy, with the same performance, the only difference is that the growing have to be done manually and that the push_back function has to be called as something like x, x_len = push_back(x, x_len, 10) with push_back being defined like def push_back(x, x_len, value): if len(x) == x_len: x = np.concatenate((x, np.empty(max(len(x), 1))) x[x_len] = value return x, x_len + 1 so that the x variable is supposed to be updated after each insertion.
zeptomu 10y ago
I think that is some kind of feature. Numpy tries hard to guarantee efficient access to its ndarrays and this is simpler to implement using fixed arrays and mostly you know your bounds at initialization time anyway. Btw. you can grow them using hstack and vstack although I am not sure how efficient that is.
- xaedes 10y ago
  Growing with vstack or hstack is very slow. I have seen code where a big csv with time series data is parsed line by line the corresponding numpy array was grown using vstack for each row. worst case: quadratic runtime
  - xapata 10y ago
    That'd be so much faster using a generator and a NumPy.concatenate at the end. Orders of magnitude faster, if we're talking about a large CSV.
- dnautics 10y ago
  There are fixed arrays in Julia too, for those who absolutely must eke out that last 1% gain in performance
  - xapata 10y ago
    It ain't just a 1% gain. Use a Python list if you want to append/pop.
    - _Wintermute 10y ago
      A deque is probably a better option for performance.
      - xapata 10y ago
        Depends on how much it grows. I'd actually expect very little difference for most append/pop situations. A deque is there for appendleft and popleft.
deleted 10y ago
[deleted]

rjeli 10y ago

Yep, a problem I repeatedly run into with Numpy is that my data is too big to fit in memory as Python data, but can easily fit as Numpy matrices. It seems incrementally building Numpy matrices is very awkward and inefficient edit: if anyone knows a solution to this, please let me know...

tgb 10y ago
I think the standard recommendation is to know how much you need to allocate and allocate it in advance. You can call numpy.zeros( size ) to get the appropriate array, then fill it with your actual data, for example. If you don't know how much you need, you can try to over-estimate it and then shrink the array (i.e. allocate one of the right size and copy) after your done.
tempay 10y ago
Where are you filling your matrix from?
inlineint 10y ago
Under the hood this `push!` function have to do memory reallocation and copying the entire array when the array grows to be larger than some internal buffer, which happens each time the array becomes larger than a^n, where a is usually, but not necessary, selected to be equal to 2. The amortized complexity is θ(1), but individual insertions might take O(n) time. It means that it is not hard to have the same in numpy, with the same performance, the only difference is that the growing have to be done manually and that the push_back function has to be called as something like x, x_len = push_back(x, x_len, 10) with push_back being defined like def push_back(x, x_len, value): if len(x) == x_len: x = np.concatenate((x, np.empty(max(len(x), 1))) x[x_len] = value return x, x_len + 1 so that the x variable is supposed to be updated after each insertion.

zeptomu 10y ago

I think that is some kind of feature. Numpy tries hard to guarantee efficient access to its ndarrays and this is simpler to implement using fixed arrays and mostly you know your bounds at initialization time anyway. Btw. you can grow them using hstack and vstack although I am not sure how efficient that is.

xaedes 10y ago
Growing with vstack or hstack is very slow. I have seen code where a big csv with time series data is parsed line by line the corresponding numpy array was grown using vstack for each row. worst case: quadratic runtime
- xapata 10y ago
  That'd be so much faster using a generator and a NumPy.concatenate at the end. Orders of magnitude faster, if we're talking about a large CSV.
dnautics 10y ago
There are fixed arrays in Julia too, for those who absolutely must eke out that last 1% gain in performance
- xapata 10y ago
  It ain't just a 1% gain. Use a Python list if you want to append/pop.
  - _Wintermute 10y ago
    A deque is probably a better option for performance.
    - xapata 10y ago
      Depends on how much it grows. I'd actually expect very little difference for most append/pop situations. A deque is there for appendleft and popleft.

deleted 10y ago

[deleted]