3 ms·
At what scale has this been tested? As in, are you aware of any data file size limits? I have a csv with ~6M rows and when paging through the docs the "Explorin
by chrsstrm 7y ago
At what scale has this been tested? As in, are you aware of any data file size limits? I have a csv with ~6M rows and when paging through the docs the "Exploring your data source" gave me pause thinking this app might try and open all 6M rows at once. Will I be OK importing such a large source or will my computer turn into a space heater before refusing to respond?
- bhou 7y agoAnanas has been tested on production processing terabyte data on a daily basis (with Google Dataflow, but you can achieve the same thing with your own spark cluster too). In term of exploring large source file, the design principle is to paginate any kind of data that support random access records (for example CSV, logs, etc). So when "exploring the data" of a CSV with 6M rows, Ananas will not load 6M rows at once, but read a few rows at a time for each page. For example, in this early demo video, exploring a 755M CSV file in seconds. https://www.youtube.com/watch?v=GwqZlhmei78&t=01m00s https://www.youtube.com/watch?v=GwqZlhmei78&t=01m00s