5 ms·

Show HN: VisiData – vi for data

221 points by rabidrat 9y ago

pcunite 9y ago
I love ncurses style applications. Perhaps for the nostalgia. However, since I'm almost always logged into a window manager, a console window running this type of app is a very nice way of having a tabbed > mouse application.
- _jal 9y ago
  I tend to prefer them, too. They're remarkably 'sticky' for me. I've been using Vim for at least 20 years, and mutt for close to that (can't recall exactly when I started using them). Been wishing for a solid, console-based spreadsheet for a while, but I haven't really gone looking that hard, as I don't use spreadsheets as much as I used to. When I do, though, I find modern GUI spreadsheets want to 'help' way too much and frankly bug the crap out of me. I suppose if I used them more, they'd brainwash me into their 'right' way of doing things, but I generally get fed up and use Postgres. That works surprisingly well for a lot of things, but updating 'cells' via SQL gets tedious, and if I'm trying to get an understanding of the data, a more visual approach is really handy. Definitely will check this tool out.
  - dredmorbius 9y ago
    There are a few ncurses spreadsheets, including one for Linux which retains a large number of Lotus123 bindings. That said, I find I tend to use awk for tasks others use spreadsheets for. This gives me the data in a file, and a program I can run against it. (You can also use a bash script and here documents to feed in data.)
    - _jal 9y ago
      I've tried doing that with awk a few times, and find I just haven't written enough to make it non-painful. I did write a ton of Perl at one point, and it seems to have hijacked the associated neurons. Spending a weekend messing about with awk until I'm comfortable is one line in a very, very long list of things I'll probably never get to.
      - dredmorbius 9y ago
        Any programming language with a line- and field-based orientation should do it. Awk is easy because of the implied input loop over the data, and the ability to slice by columns. If you want additional outputs, you simply create them. For a trivial example, if you've got a list of revenues and expenses (say, by month), and want a report with monthly and annual totals (I'll assume only one year is fed, though the example could be expanded: #!/usr/bin/gawk -f BEGIN { printf( "%-8s %-8s | %-8s\n", "revenue", "costs", "profit" )} { rev = $1; cost = $2; profit = rev - cost; totrev+=rev; totcost+=cost; printf( "%8.2f %8.2f | %8.2f\n", rev, cost, profit ) } END { printf( "----------------------------\n") printf( "%8.2f %8.2f | %8.2f\n", totrev, totcost, totrev - totcost ) printf( "============================\n") } That's the equivalent of a spreadsheet with columns for revenues, costs, and profits, with a summary line. But you can run it over any annual datasource. Now let me check if that's right.... OK, not too far off, though yes, I'd goofed bits (the first time).
n1vz3r 9y ago
Any plans to support SQL-like queries for data tables? For example, using SQLite?
- rabidrat 9y ago
  It can already connect (read-only) to SQLite databases. Use `-f sqlite`. It would not be too hard to implement UPDATE/DELETE queries. Let me know if you're interested in helping and we can make this happen!
roadnottaken 9y ago
I use tabview [1] for this. I install it on all my servers and use it regularly. Simple command-line spreadsheet-style navigation of tabular data. VisiData looks nice, too, with possibly more-advanced features. [1] https://github.com/femtotrader/tabview https://github.com/femtotrader/tabview
hbbio 9y ago
And, 33 years later, Multiplan finally gets open sourced!
yakshaving_jgt 9y ago
Off topic, sorry, but… Saul Pwanson? I can't not read your name as a Spoonerism.
gaetanrickter 9y ago
I used this with a dataset for stocks here https://hackernoon.com/unsupervised-machine-learning-for-fun-profit-with-basket-clusters-17a1161e7aa1 https://hackernoon.com/unsupervised-machine-learning-for-fun... and it's great for getting a handle on your data and reorganizing it even before any exploratory data analysis (EDA).
azinman2 9y ago
This is awesome. I'd love to see this incorporated into psql or mysql's cli tool for results!!!! The current implementations are horrible.
- rabidrat 9y ago
  This should really straightforward (like maybe 1-3 hours) for someone who knows use the Python3 modules for those databases. See the SQLite addon, for example: https://github.com/saulpw/visidata/blob/stable/visidata/addons/sqlite.py https://github.com/saulpw/visidata/blob/stable/visidata/addo...
nafizh 9y ago
Is there a tutorial available? Awesome to get something like this in terminal but seems like it has a steep learning curve.
- rabidrat 9y ago
  There is a bin/vdtutor in the git repository, which only has one sheet so far. Also the animated screenshot links to a page with a tour of the commands used. More of each are planned, but it seemed more urgent to release a beta right now. I'd love some help, if you or anyone is interested in helping put together another tutorial level or screencast or tour.
- polm23 9y ago
  This is the user guide: https://github.com/saulpw/visidata/blob/stable/docs/user-guide.rst https://github.com/saulpw/visidata/blob/stable/docs/user-gui... (The link is broken in the docs/README file) This is the walkthrough on sample data: https://github.com/saulpw/visidata/blob/develop/docs/tours.rst https://github.com/saulpw/visidata/blob/develop/docs/tours.r...
alexvay 9y ago
Really nice! I used https://github.com/dinedal/textql https://github.com/dinedal/textql a bit, but your project seems much more practical.
marklgr 9y ago
In the same vein, you can also check sc-im: https://github.com/andmarti1424/sc-im https://github.com/andmarti1424/sc-im
co_dh 9y ago
I'm glad that I'm using Emacs, and it has csv mode, sql intergration, and anything you want.
SirLJ 9y ago
Looks really cool, the interface kind of reminds me of the long forgotten informix 4gl, infotmix forms, and the like... Thanks for the trip down memory lane :-)
dmitripopov 9y ago
The name is a nod to VisiCalc?
- rabidrat 9y ago
  Yes it is! VisiCalc was a pioneering 'killer app', and I feel like this is the next iteration :)
easytiger 9y ago
Amazing idea, i've been using some command line fudges when i need to work on csvs as i often

221 points by rabidrat 9y ago

I love ncurses style applications. Perhaps for the nostalgia. However, since I'm almost always logged into a window manager, a console window running this type of app is a very nice way of having a tabbed > mouse application.

_jal 9y ago
I tend to prefer them, too. They're remarkably 'sticky' for me. I've been using Vim for at least 20 years, and mutt for close to that (can't recall exactly when I started using them). Been wishing for a solid, console-based spreadsheet for a while, but I haven't really gone looking that hard, as I don't use spreadsheets as much as I used to. When I do, though, I find modern GUI spreadsheets want to 'help' way too much and frankly bug the crap out of me. I suppose if I used them more, they'd brainwash me into their 'right' way of doing things, but I generally get fed up and use Postgres. That works surprisingly well for a lot of things, but updating 'cells' via SQL gets tedious, and if I'm trying to get an understanding of the data, a more visual approach is really handy. Definitely will check this tool out.
- dredmorbius 9y ago
  There are a few ncurses spreadsheets, including one for Linux which retains a large number of Lotus123 bindings. That said, I find I tend to use awk for tasks others use spreadsheets for. This gives me the data in a file, and a program I can run against it. (You can also use a bash script and here documents to feed in data.)
  - _jal 9y ago
    I've tried doing that with awk a few times, and find I just haven't written enough to make it non-painful. I did write a ton of Perl at one point, and it seems to have hijacked the associated neurons. Spending a weekend messing about with awk until I'm comfortable is one line in a very, very long list of things I'll probably never get to.
    - dredmorbius 9y ago
      Any programming language with a line- and field-based orientation should do it. Awk is easy because of the implied input loop over the data, and the ability to slice by columns. If you want additional outputs, you simply create them. For a trivial example, if you've got a list of revenues and expenses (say, by month), and want a report with monthly and annual totals (I'll assume only one year is fed, though the example could be expanded: #!/usr/bin/gawk -f BEGIN { printf( "%-8s %-8s | %-8s\n", "revenue", "costs", "profit" )} { rev = $1; cost = $2; profit = rev - cost; totrev+=rev; totcost+=cost; printf( "%8.2f %8.2f | %8.2f\n", rev, cost, profit ) } END { printf( "----------------------------\n") printf( "%8.2f %8.2f | %8.2f\n", totrev, totcost, totrev - totcost ) printf( "============================\n") } That's the equivalent of a spreadsheet with columns for revenues, costs, and profits, with a summary line. But you can run it over any annual datasource. Now let me check if that's right.... OK, not too far off, though yes, I'd goofed bits (the first time).

n1vz3r 9y ago

Any plans to support SQL-like queries for data tables? For example, using SQLite?

rabidrat 9y ago
It can already connect (read-only) to SQLite databases. Use `-f sqlite`. It would not be too hard to implement UPDATE/DELETE queries. Let me know if you're interested in helping and we can make this happen!

roadnottaken 9y ago

I use tabview [1] for this. I install it on all my servers and use it regularly. Simple command-line spreadsheet-style navigation of tabular data. VisiData looks nice, too, with possibly more-advanced features. [1] https://github.com/femtotrader/tabview https://github.com/femtotrader/tabview

hbbio 9y ago

And, 33 years later, Multiplan finally gets open sourced!

yakshaving_jgt 9y ago

Off topic, sorry, but… Saul Pwanson? I can't not read your name as a Spoonerism.

gaetanrickter 9y ago

I used this with a dataset for stocks here https://hackernoon.com/unsupervised-machine-learning-for-fun-profit-with-basket-clusters-17a1161e7aa1 https://hackernoon.com/unsupervised-machine-learning-for-fun... and it's great for getting a handle on your data and reorganizing it even before any exploratory data analysis (EDA).

azinman2 9y ago

This is awesome. I'd love to see this incorporated into psql or mysql's cli tool for results!!!! The current implementations are horrible.

rabidrat 9y ago
This should really straightforward (like maybe 1-3 hours) for someone who knows use the Python3 modules for those databases. See the SQLite addon, for example: https://github.com/saulpw/visidata/blob/stable/visidata/addons/sqlite.py https://github.com/saulpw/visidata/blob/stable/visidata/addo...

nafizh 9y ago

Is there a tutorial available? Awesome to get something like this in terminal but seems like it has a steep learning curve.

rabidrat 9y ago
There is a bin/vdtutor in the git repository, which only has one sheet so far. Also the animated screenshot links to a page with a tour of the commands used. More of each are planned, but it seemed more urgent to release a beta right now. I'd love some help, if you or anyone is interested in helping put together another tutorial level or screencast or tour.
polm23 9y ago
This is the user guide: https://github.com/saulpw/visidata/blob/stable/docs/user-guide.rst https://github.com/saulpw/visidata/blob/stable/docs/user-gui... (The link is broken in the docs/README file) This is the walkthrough on sample data: https://github.com/saulpw/visidata/blob/develop/docs/tours.rst https://github.com/saulpw/visidata/blob/develop/docs/tours.r...

alexvay 9y ago

Really nice! I used https://github.com/dinedal/textql https://github.com/dinedal/textql a bit, but your project seems much more practical.

marklgr 9y ago

In the same vein, you can also check sc-im: https://github.com/andmarti1424/sc-im https://github.com/andmarti1424/sc-im

co_dh 9y ago

I'm glad that I'm using Emacs, and it has csv mode, sql intergration, and anything you want.

SirLJ 9y ago

Looks really cool, the interface kind of reminds me of the long forgotten informix 4gl, infotmix forms, and the like... Thanks for the trip down memory lane :-)

dmitripopov 9y ago

The name is a nod to VisiCalc?

rabidrat 9y ago
Yes it is! VisiCalc was a pioneering 'killer app', and I feel like this is the next iteration :)

easytiger 9y ago

Amazing idea, i've been using some command line fudges when i need to work on csvs as i often