8 ms·

Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

60 points by jastr 10y ago

jastr 10y ago
Hi HN! A few months ago, a college friend reached out because his consulting company had just received a 60 Gb spreadsheet, and they didn't know what to do with it! They actually tried opening it in Excel. I'm excited to Show HN CSV Explorer - a simple web tool for opening really big CSVs! Try it out, and let me know how it goes!
- goatlover 10y ago
  Open it with a programming language like R, Julia or Python that supports the data frames. No reason to upload 60 gigs to the web or Excel.
  - jastr 10y ago
    This project came out of some of my contracting work for consultants, journalists, lawyers - people who don't know what a command line is.
- danvoell 10y ago
  Worked like a charm. This is really cool. I can see a lot of uses with large files. Especially filtering down "faceted" search items.
cocktailpeanuts 10y ago
What's "YC F1"?
- jastr 10y ago
  I was in the first batch of YCombinator's Fellowship.
patwalls 10y ago
This is really awesome. What sparked this idea? Not sure if you have this feature, but I think it would be really cool if you could open APIs to your own datasets. This could be really useful for enterprise applications that do a lot of flat file imports/exports to push/pull data.
- jastr 10y ago
  Some consultant friends asked for help when they got a 60Gb CSV file from their client! Since then, I've also found that lots of tech companies share CSV internally. I've also worked with journalists attempting to look at large public government datasets.
  - seanp2k2 10y ago
    Plenty of Big Data things also work with CSV. Like >1TB datasets big. It's pretty insane, but it works.
xaybey 10y ago
I usually use csvkit (https://csvkit.readthedocs.io/en/1.0.1/index.html https://csvkit.readthedocs.io/en/1.0.1/index.html). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decimals. That being said I would never want to (and often legally cannot) etl my data to some third party. That would be terribly slow. But I would happily pay for a nice desktop tool to do it for me; command line or GUI.
- jastr 10y ago
  Csvkit is great, and I use it a lot. The target users for CSV Explorer though are mostly non-engineers ie. people who don't use the command line or code.
  - lomereiter 10y ago
    Kibana has quite a few rough edges but it takes non-technical users very far in terms of data exploration. Local setup also alleviates privacy concerns. The bummer is import/export: 'Upload CSV' feature has been almost introduced in 5.0 release but removed in the end [1], and CSV export has been asked for many times [2] but is still not there. So a user-friendly fork might be a worthy business idea. [1]: https://github.com/elastic/kibana/pull/8497 https://github.com/elastic/kibana/pull/8497 [2]: https://github.com/elastic/kibana/issues/1992 https://github.com/elastic/kibana/issues/1992
- seanp2k2 10y ago
  shuf -n10000 foo.csv > foo-sample.csv; open $_ That should give you a 10k-line random sample to play with that should open quite quickly in Excel.
- a3n 10y ago
  At a former job, our embedded device logs decoded to csv. Some of them were too large for Excel. Pandas handled them without a burp. Pandas in Jupyter (Ipython Notebook) was a godsend. There's a minimal amount of variable setup, but once you've done that once or twice it's easy. Of course, any analysis or manipulation takes a bit of python code, but I see that as a feature, not the least because you can read it right there in the open instead of having to hunt for formulas in cells.
hprotagonist 10y ago
My solution to this problem, so far, has been ipython and pandas. (and maybe jupyter notebook for visualization and sharing). What's the value-add here?
- jastr 10y ago
  It's like a Jupyter notebook for non-engineers, people who don't know what python is, eg. consultants.
chrisweekly 10y ago
Does this do stuff lnav can't?
deadringerr 10y ago
Cool idea - Having to work with sensitive data, it'd be great to have this functionality without importing/uploading to the internet.
- jastr 10y ago
  Thanks - having worked with health data for a few years I can relate. Unfortunately, I don't have plans for a desktop app right now.
  - seanp2k2 10y ago
    Think not of a desktop app, but of doing everything client-side in JS. That way, it's still a web app, but you're not schlepping [sensitive|large] data between front and back-ends. Also, by offloading the work onto clients, it scales much better - you could host the app on a CDN and have no real back-end.
    - jastr 10y ago
      I originally wrote it in entirely client side JS, but it didn't scale nicely past a few hundred thousand rows. For the really big datasets, CSV Explorer loads them into Redshift - queries takes a few seconds!
emodendroket 10y ago
Is there any reason this tool couldn't be 100% client-side?
- jastr 10y ago
  Perhaps - I tried a client-side implementation, but I had issues scaling past a few hundred thousand rows. I understand hesitations of uploading data to the cloud, but I now have users looking at hundreds of millions of rows in seconds thanks to Postgres!
  - Zaheer 10y ago
    You can run PostGres natively as well and do the same thing no?
    - emodendroket 10y ago
      Not really an all-in-browser solution then.
  - chatmasta 10y ago
    I suggest exposing a config variable in the js client to set the hostname of the Postgres server. This way you can provide a self hosted version of the backend, and customers can self host it either locally on their machine, or within an internal network. Same benefits of the browser, just need to change config.
pclark 10y ago
you should integrate with commatocolumn.com! ;)
jastr 10y ago
Update: A one minute demo video - https://youtu.be/RBiDL5neWDc https://youtu.be/RBiDL5neWDc
dromenkoning 10y ago
This tool would be great were it not for me (rightfully) getting fired and possibly sued by any of my clients if I uploaded even a single file. In fact, what kind of business with millions-of-rows kind of files would entrust said datasets to a service that to me seems strangely vague [1] on how the data is secured, or where it is actually going to be physically stored. [1] https://www.csvexplorer.com/legal/privacy/ https://www.csvexplorer.com/legal/privacy/

60 points by jastr 10y ago

jastr 10y ago

Hi HN! A few months ago, a college friend reached out because his consulting company had just received a 60 Gb spreadsheet, and they didn't know what to do with it! They actually tried opening it in Excel. I'm excited to Show HN CSV Explorer - a simple web tool for opening really big CSVs! Try it out, and let me know how it goes!

goatlover 10y ago
Open it with a programming language like R, Julia or Python that supports the data frames. No reason to upload 60 gigs to the web or Excel.
- jastr 10y ago
  This project came out of some of my contracting work for consultants, journalists, lawyers - people who don't know what a command line is.
danvoell 10y ago
Worked like a charm. This is really cool. I can see a lot of uses with large files. Especially filtering down "faceted" search items.

cocktailpeanuts 10y ago

What's "YC F1"?

jastr 10y ago
I was in the first batch of YCombinator's Fellowship.

patwalls 10y ago

This is really awesome. What sparked this idea? Not sure if you have this feature, but I think it would be really cool if you could open APIs to your own datasets. This could be really useful for enterprise applications that do a lot of flat file imports/exports to push/pull data.

jastr 10y ago
Some consultant friends asked for help when they got a 60Gb CSV file from their client! Since then, I've also found that lots of tech companies share CSV internally. I've also worked with journalists attempting to look at large public government datasets.
- seanp2k2 10y ago
  Plenty of Big Data things also work with CSV. Like >1TB datasets big. It's pretty insane, but it works.

xaybey 10y ago

I usually use csvkit (https://csvkit.readthedocs.io/en/1.0.1/index.html https://csvkit.readthedocs.io/en/1.0.1/index.html). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decimals. That being said I would never want to (and often legally cannot) etl my data to some third party. That would be terribly slow. But I would happily pay for a nice desktop tool to do it for me; command line or GUI.

jastr 10y ago
Csvkit is great, and I use it a lot. The target users for CSV Explorer though are mostly non-engineers ie. people who don't use the command line or code.
- lomereiter 10y ago
  Kibana has quite a few rough edges but it takes non-technical users very far in terms of data exploration. Local setup also alleviates privacy concerns. The bummer is import/export: 'Upload CSV' feature has been almost introduced in 5.0 release but removed in the end [1], and CSV export has been asked for many times [2] but is still not there. So a user-friendly fork might be a worthy business idea. [1]: https://github.com/elastic/kibana/pull/8497 https://github.com/elastic/kibana/pull/8497 [2]: https://github.com/elastic/kibana/issues/1992 https://github.com/elastic/kibana/issues/1992
seanp2k2 10y ago
shuf -n10000 foo.csv > foo-sample.csv; open $_ That should give you a 10k-line random sample to play with that should open quite quickly in Excel.
a3n 10y ago
At a former job, our embedded device logs decoded to csv. Some of them were too large for Excel. Pandas handled them without a burp. Pandas in Jupyter (Ipython Notebook) was a godsend. There's a minimal amount of variable setup, but once you've done that once or twice it's easy. Of course, any analysis or manipulation takes a bit of python code, but I see that as a feature, not the least because you can read it right there in the open instead of having to hunt for formulas in cells.

hprotagonist 10y ago

My solution to this problem, so far, has been ipython and pandas. (and maybe jupyter notebook for visualization and sharing). What's the value-add here?

jastr 10y ago
It's like a Jupyter notebook for non-engineers, people who don't know what python is, eg. consultants.

chrisweekly 10y ago

Does this do stuff lnav can't?

deadringerr 10y ago

Cool idea - Having to work with sensitive data, it'd be great to have this functionality without importing/uploading to the internet.

jastr 10y ago
Thanks - having worked with health data for a few years I can relate. Unfortunately, I don't have plans for a desktop app right now.
- seanp2k2 10y ago
  Think not of a desktop app, but of doing everything client-side in JS. That way, it's still a web app, but you're not schlepping [sensitive|large] data between front and back-ends. Also, by offloading the work onto clients, it scales much better - you could host the app on a CDN and have no real back-end.
  - jastr 10y ago
    I originally wrote it in entirely client side JS, but it didn't scale nicely past a few hundred thousand rows. For the really big datasets, CSV Explorer loads them into Redshift - queries takes a few seconds!

emodendroket 10y ago

Is there any reason this tool couldn't be 100% client-side?

jastr 10y ago
Perhaps - I tried a client-side implementation, but I had issues scaling past a few hundred thousand rows. I understand hesitations of uploading data to the cloud, but I now have users looking at hundreds of millions of rows in seconds thanks to Postgres!
- Zaheer 10y ago
  You can run PostGres natively as well and do the same thing no?
  - emodendroket 10y ago
    Not really an all-in-browser solution then.
- chatmasta 10y ago
  I suggest exposing a config variable in the js client to set the hostname of the Postgres server. This way you can provide a self hosted version of the backend, and customers can self host it either locally on their machine, or within an internal network. Same benefits of the browser, just need to change config.

pclark 10y ago

you should integrate with commatocolumn.com! ;)

jastr 10y ago

Update: A one minute demo video - https://youtu.be/RBiDL5neWDc https://youtu.be/RBiDL5neWDc

dromenkoning 10y ago

This tool would be great were it not for me (rightfully) getting fired and possibly sued by any of my clients if I uploaded even a single file. In fact, what kind of business with millions-of-rows kind of files would entrust said datasets to a service that to me seems strangely vague [1] on how the data is secured, or where it is actually going to be physically stored. [1] https://www.csvexplorer.com/legal/privacy/ https://www.csvexplorer.com/legal/privacy/