7 ms·
Csvlens: Command line CSV file viewer. Like less but made for CSV
- jwiz 3y agoSee also: https://www.visidata.org/ https://www.visidata.org/
- mistaken 3y agovisidata is awesome. It also supports a bunch of other file formats such as excel sheets.
- standing_user 3y agoDidn't know about it, thank you
- rrr_oh_man 3y ago> Scatterplots in the terminal Wow!
- jellyfishbeaver 3y agoStrongly considering donating to the creator of Visidata, it's one of the terminal tools I can't live without. I use it every day.
- bbkane 3y agoIf it's that important and you can afford a few bucks a month, I strongly urge you stop considering your donation, and (right now) follow the links on the GitHub to actually start supporting the project. It's really easy to wallow in the "considering" phase of a decision indefinitely (or maybe that's just me :D)
- letmeinhere 3y agohuh, something (maybe similarity to VisiCalc name) had me dismissing it as commercial software
- jwiz 3y agoYep, I am doing the $5/mo Patreon level because I love it so much.
- zoogeny 3y agoI've been thinking about a terminal spreadsheet editor for about a year now. I haven't deeply investigated this tool while I write this comment but I'd love for something like this for Terminal that supported `=A1+B2` kind of formulas.
- runjake 3y agoBack in the day, we called this Lotus 123 and VisiCalc. https://en.m.wikipedia.org/wiki/Lotus_1-2-3 https://en.m.wikipedia.org/wiki/Lotus_1-2-3
- zoogeny 3y agoYes, exactly - imagine a modern version written in Rust, with vi key-bindings, etc. Please someone be inspired by this idea and build this. TUIs are making a comeback and the spreadsheet is still one of the most useful and effective tools in all information science.
- PaulHoule 3y agoThe spreadsheet data model lives a double life. (1) On one hand you can use it for data that is really tabular, (2) But it also has an engine that can compute the dependency relationships between cells and recalculate the cells affected by a change. This is quite different from conventional programming languages where you are required to specify an order to put operations in. Of course this a problem for exploiting parallelism but I'd charge it is one more bit of cognitive load that makes it harder for beginners and non-professional programmers. (Professional programmers are just used to it and only run into problems in unusual cases where circularity is involved, but I think it's one more thing that beginners struggle with.) The worst problem is a lack of separation between code and data. If you are doing an analysis you might put something like =SUM(A1:A28) in A30. Excel will change that to =SUM(A1:A29) if you insert a row in there, which helps, but they lock you into the mindset of "I'm making the December sales report" as opposed to "I'm making the monthly sales report". That is, the data and the analysis should be two separate things: just as you can put the December data or the January data into a Python script. (Note some of the same problems still exist with "notebooks" and "workspaces" where you wind up with a file that has both code and data in it which can be problematic to check into git, particularly when your are working for people who would like to have a beautiful notebook with an analysis in it to view in GitHub but will then struggle to version control it. Many "data scientists" fail to rise above the December sales report even though there's a clear path to turn a Jupyter notebook into a Python script.) --- As much as that is a rant I think there's a huge untapped market for things that are like spreadsheets but different. That is, something that looks like Excel but is specialized for editing tabular data (no formulas, CSV import 'just works' all the time, ...) or something that has formulas like Excel but not on a grid or not just on a grid. For the latter there was this product https://en.wikipedia.org/wiki/TK_Solver https://en.wikipedia.org/wiki/TK_Solver which is still around. People were amazed with TK Solver when it came out and I'm surprised to this day that it hasn't had a lot of competition.
- deleted 3y ago[deleted]
- somat 3y agoCan confirm, I consider visidata to be the gold standard for data exploration. I have largely given up on spreadsheets, replacing them with relational databases. And on top of being a world class tool for understanding the data for import, visidata makes for a pretty great query pager.
- retrochameleon 3y agoI've been using Visidata and love it for viewing, extracting rows or columns to a different file, but I know it is capable of much more that I haven't figured out yet
- musha68k 3y agoNot on “computer” but some such in your pipeline should be good enough more often than not? awk -v OFS='\t' '{$1=$1; print}' | column -t
- cowsandmilk 3y agomany csv data sets cannot be handled by awk due to csv supporting things like commas inside fields, newlines inside fields, etc. Your solution would also fall over for any tabs within fields. Plenty of tools out there support RFC 4180 CSV files as well as common csv variants. And these tools have been around for a decade and have suites of cli commands that can be piped together, just speaking "CSV" as a format rather than pure text streams.
- musha68k 3y agoGood points on using proper CSV tooling 100%; so that’s why the disclaimer “more often than not” BTW I just realized that I was omitting the field separator ‘-F,’ actually. For those pesky nested commas from top off simplistic text hat still I’d go back to sed for fun and potential profit: sed 's/","/"\t"/g; s/^"/"\t/; s/"$//; s/,,/, ,/g' | column -t Not elegant nor hands-off and might be missing something else; def needs the right data / volume and fiddling as we know ymmv; can get unwieldy quickly etc etc TLDR; use column -t when applicable
- prudentpomelo 3y agoAwk now supports a `--csv` flag for processing csv's. https://github.com/onetrueawk/awk/blob/master/README.md https://github.com/onetrueawk/awk/blob/master/README.md
- kkoncevicius 3y agoDoes it allow specifying columns when filtering for rows? For instance, the shown example filters for "Bug" but seems to filter for rows that contain "Bug" in any of the columns. Can I specify to filter for "Bug" only looking at a certain column? At the same time - is filtering by numbers implemented, like filtering for rows that have a numeric value above X in a certain column? If not these should be on TODO list - those are very common operations for .csv type data.
- dmd 3y agoYou want https://www.visidata.org/ https://www.visidata.org/
- account-5 3y agoI love these sorts of programs. Is there a list so where of data programs like this for the command line? Im sure there will be.
- drog 3y agoOne of the things that greatly improved my csv workflow is duckdb. It’s a small binary that allows querying csv with sql.
- lyjackal 3y agoYou can do the same with SQLite, which is usually already installed in most environments sqlite> .import test.csv foo --csv
- llimllib 3y agoA poor man's version of csvlens is something like: sqlite -column :memory: '.import --csv file.csv tmp' 'select * from tmp;' | bat which imports the csv into sqlite and outputs it to bat, my favorite pager - use `less` or whatever else you desire.
- 1vuio0pswjnm7 3y agoIn addition to using a pager with sqlite3's fantastic text-only output .modes, if the CSV contains hyperlinks I use a custom UNIX filter I wrote that outputs simple, minimalist HTML. Then I view with text-only browser. For example, this is how I use YouTube. I never use the YouTube website, with its gigantic pages and its "Javascript player", not to mention all of the telemetry. All the search results and information about videos is stored in SQL or CSV, viewed with a text-only sqlite3 output .mode, and optionally converted to simple HTML. For me, this is better than a "modern" web browser that's too large for me to compile.
- dbreunig 3y agoduckdb is a single file with no dependencies and it's fast. Still blows my mind how quickly it can query GB sized gzipped CSVs.
- xcdzvyn 3y agoWow, this is awesome. Thanks a lot.
- 3y ago
- deleted 3y ago[deleted]
- RhysU 3y agongrid is quite usable, stable, and sensible for streaming work: https://github.com/twosigma/ngrid https://github.com/twosigma/ngrid
- ranger_danger 3y agoa TUI program without a screenshot might as well not exist... nobody is going to use it if they can't see what it looks like.
- pradeepchhetri 3y agoI enjoy using clickhouse-local for parsing csv files. I generally hit situations where I need custom delimiter and custom parsing rules, I find it handles all of these edge cases very well. Recently I found that if my csv files are compressed, i don't even need to uncompress them, it auto-magically figures out the compression format and process it for me.
- tambourine_man 3y agoDidn’t know about them, seems interesting.
- magundu 3y agoWhich one is better for querying CSV from both command line and programmatically? 1. SQLite 2. DucksDB 3. clickhouse-local
- pradeepchhetri 3y agoDefinitely clickhouse ecosystem. - For querying csv data from command line, I use clickhouse-local. - For querying csv data programmatically using a library, I use chdb (embedded version of clickhouse) - For querying large amount of csv data programmatically, I offload it to clickhouse cluster which can do processing in distributed fashion. If you are looking from query performance perspective, this blog is useful: https://www.vantage.sh/blog/clickhouse-local-vs-duckdb https://www.vantage.sh/blog/clickhouse-local-vs-duckdb
- bishfish 3y agoThis would be nice to have with lesspipe.sh viewer.
- leonim 3y agoI don't think this would work as a lesspipe.sh viewer, since it is interactive. I have wanted this for VisiData for data files. You would need a wrapper around less that could start other programs instead of less. You might be interested in Tidy Viewer with lesspipe.sh [1] https://news.ycombinator.com/item?id=28670252 https://news.ycombinator.com/item?id=28670252 [2] https://github.com/alexhallam/tv https://github.com/alexhallam/tv
- tinkertamper 3y agoLove this! Call me lame, but are there any open-source projects that accomplish this type of gui in typescript? I have an idea but it uses something written in JavaScript.
- chuckadams 3y agoYou can use any javascript library in typescript, and TS is all JS at runtime anyway. You can even add types without touching the JS library, just write a .d.ts file and stick it anywhere that TS looks for sources.
- zeckalpha 3y agoI'd love something like this in csvkit, xsv, or qsv. The refragmentation of CSV CLI tools is counter to the long term trend.
- wodenokoto 3y agoIs there an open source thing similar to this that has a windows gui?
- trackofalljades 3y agoThis is glorious, someone put it in homebrew!
- edu_guitar 3y agoNice! Once or twice I've used tad as a GUI to view csv files, but I usually use vi with nowrap or read the file in R. Now csvlens will be my default for csv files. [1]:https://www.tadviewer.com/ https://www.tadviewer.com/
- bee_rider 3y agoI like viewing most CSVs in vim. It can be a little annoying when the columns don’t line up well. Often I’ll replace all the commas with a tab, and then set the tab width to some very high value to make it all line up. Of course, there exist plenty of CSV files that don’t follow any particular standard, so the trick doesn’t always work (if there are tabs in the data, if there’s some very wide field, or if there’s a comma in the data). Works good enough for the files I have, though.
- quesera 3y agoThere are a few vim plugins that make CSVs more convenient.
- RicoElectrico 3y agoI tried some TUI tool that was pretty advanced at viewing tables (sorting, filtering etc.), but I don't remember the name. It was on a WSL instance on my work laptop which I nuked when they laid me off. I remember that it was tab-based. Any idea what that could have been?
- nickjj 3y agoOne thing I often do with CSVs is sum up all or specific rows in a column. For example maybe you're doing end of year taxes and now you have this large CSV export from your bank or payment provider with multiple categories and you want to get the totals for certain things. In a GUI tool it's really easy to sort by a column and drag your mouse to select what you want and see it summed in real time. Oftentimes things aren't clean enough to have 100% confidence that you can solve this with an automated script because maybe something is spelled slightly different but it's really the same thing. This feels like one of those things where spending a legit 10-15 minutes once a year to do it manually is better than trying to account for every known and unknown edge case you could think of. The stakes are too high if you get it wrong since it's related to taxes. Has anyone found a really good standalone basic spreadsheet app that "just works" which isn't Microsoft Excel that works on Windows or Linux? I don't know why but Libre and Open Office both struggle to parse columns out in certain types of CSVs and the sorting behavior is typically a lot worse than Google's spreadsheet app but I'd like to remove some dependence on using Google.
- AB1908 3y agoI've never tried this myself but you could try importing the CSV in sqlite and then run aggregate queries over it. Does that sound useful?
- mongol 3y agoWas about to suggest the same
- nickjj 3y agoYou can but it's back to depending on code to get the totals. This is one spot where IMO being able to visualize the data by seeing the rows and have immediate feedback on the sum is useful. You can continue dragging or use CTRL + SHIFT clicking to select more stuff as needed while letting your brain decide what should be grouped together. With a SQL / code approach you have to account for these things without being able to see them and then adjust the code afterwards to include your custom groupings. It ends up taking more time. If the categories didn't change every year it would for sure be worth it to code up a solution since you'll know the edge cases by looking at the existing CSV but it's a moving target because it could change next year.
- deleted 3y ago[deleted]
- fao_ 3y agoSee we wouldn't need this if everyone had standardized on TSV instead
- JohnKemeny 3y agoI don't understand if this is meant as a joke or not. Do you mean that we then would have Tsvlens? Tabs don't automatically make columns aligned, nor do they eliminate the problem of quoting. I don't get it, but then again, I'm exceptionally dim-witted.
- bee_rider 3y agoBecause tabs are flexible space, you can render them really wide in most text editors, wide enough that all the columns line up. But it is a hacky solution!
- m2f2 3y agoTSV are far superior to CSV as the separator is a TAB and not commas, semicolons or other weird stuff depending on your locale. Same for single and double quotes. Only thing to be aware are tsv created by Excel with newlines embedded in text fields, which are a huge PITA for almost all parsers.
- fao_ 3y agoOther people have mentioned a lot of advantages but I'll go for another one: TSV is eminently more readable and parsable just off the bat. Empty space is always more readable than `,`, and `,` appears a lot in strings, tab appears a lot less.
- leeoniya 3y agodiffeence from https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv ?
- adius 3y agoI found the perfect solution for me with https://www.moderncsv.com https://www.moderncsv.com. Starts fast, focused GUI, fully featured, no bullshit!
- minroot 3y agoReally cool, hope it was open source
- hermitcrab 3y agoSome people are trying to make a living from the software they write. Open source is not very conducive to that.
- BasilPH 3y agoThat looks like a great CSV viewer. I've enjoyed using csvkit[^0] in the past. The viewer isn't as good as csvlens seem to be, but it comes with the ability to grep, cut and pipe CSV data which has come in handy. csvlens + csvkit might be a great combination. [^0]: https://csvkit.readthedocs.io/en/latest/ https://csvkit.readthedocs.io/en/latest/
- yzzyx 3y agoI have been looking for a viewer exactly like this for so long! Visidata is nice, but way more complex than what I was after. This will fit perfectly into my workflow, thank you for sharing!
- flusteredBias 3y agoSee also: (tv) Tidy Viewer. A cross-platform CLI csv pretty printer. https://github.com/alexhallam/tv https://github.com/alexhallam/tv