12 ms·
Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment
- iso1210 5y agoDemo looks great, alas the prebuilt binaries don't work $ wget https://github.com/alexhallam/tv/releases/download/0.0.10/tidy-viewer --2021-09-27 16:47:43-- https://github.com/alexhallam/tv/releases/download/0.0.10/tidy-viewer Resolving github.com (github.com)... 140.82.121.3 Connecting to github.com (github.com)|140.82.121.3|:443... connected. HTTP request sent, awaiting response... 404 Not Found 2021-09-27 16:47:44 ERROR 404: Not Found. The deb in https://github.com/alexhallam/tv/target https://github.com/alexhallam/tv/target doesn't exist either
- flusteredBias 5y ago1. Looks like you are using 0.0.10 the current version is 0.0.13 2. I need to update the README the binaries are here https://github.com/alexhallam/tv/releases/tag/0.0.13 https://github.com/alexhallam/tv/releases/tag/0.0.13 I recently moved away from manually building binaries to automated building for many architectures. I am still learning how to use github actions to build for a matrix of architectures. I am still learning.
- flusteredBias 5y agoI spend a lot of time in the terminal and want to quickly glance at a csv files without making a new script, opening excel, or using a tui. I made tidy-viewer (tv) because current tools like cat and column were not pretty enough. tv modifies raw files in the following ways: 1. NA detection and highlighting 2. Printing only significant digits 3. Header and footer meta data I have been using this a lot at work. There is a lot more work to do, but it is in a usable state. Give it a try! If you like it then star on Github!
- IgorPartola 5y agoThis looks great! I wonder how long it’ll be until someone posts a long ask snippet that will do something similar and claim this isn’t progress, but rest assured that they are wrong. I’m adding tv to my toolbox.
- flusteredBias 5y agoThanks! I appreciate the compliment!
- mileza 5y agoI think it's a great first effort, but there are a number of possible improvements to do. The most obvious one would be to support passing the file as an argument instead of using cat or the redirection operator every time. It's great that it works with stdin to allow piping into it, but it's cumbersome if you just want to take a file and print it, which will no doubt be a common use case.
- MobiusHorizons 5y agoDo you think ‘tv <file.csv’ does what you want well enough? What is the behavior when you run ‘tv file.csv’ does it just block waiting for input? I think it’s great for a visualizer like this to encourage people to get used to the power of shell pipelines if possible.
- Someone 5y agoI think it’s a bad idea to go against half a century of conventions without good reason. It’s very surprising to see a tool that only works as a filter, and doesn’t take file paths as arguments.
- GuB-42 5y agoIt works, but almost all UNIX commands that work on pipelines can take a list of files as arguments. Out of the commands I use regularly, "patch" is the only one that works exclusively from stdin, probably because file arguments have a different, somewhat obscure, and probably historical meaning. If appropriate, using files as arguments instead of using shell pipelines is a best practice. Commands can optimize for that use case, print better error messages, etc... And it is not a good thing to encourage useless use of cat. If you goal is to show how your tool is to be used with pipelines, show an actually useful pipeline for example "sed '1b;/abc/!d' file.csv | tv". The "sed" command prints the first line (header), and all lines containing "abc".
- MobiusHorizons 5y agoFair enough.
- moritonal 5y agoMan.. TV is such a good name for a visualiser tool that it'd be excellent if it could pretty-print any content given to it. Does your framework support the idea of pre-parsing the file's content and selecting an appropriate renderer, or is it fairly tied to CSVs?
- flusteredBias 5y agoI would need more details on what you mean by "pre-parsing". It works with any deliminator, it could be comma-separated, pipe-separated, etc.
- mixmastamyk 5y agoDoesn't need to be pre-parsed. Perhaps give the filename to the utility instead of content via stdin. Then filename gives a hint. If there is none, run "file filename" (via library) beforehand.
- inostia 5y agoI actually like `tidy` better and honestly would rather have `cat a.csv | tidy ...`. But it's probably already a thing.
- hnlmorg 5y agoNice idea. I might actually work on this myself
- flusteredBias 5y agoYou don't have to make the alias `tv`. Feel free to make your alias tidy='tidy-viewer'.
- einpoklum 5y agoFirst of all - kudos on tackling this task - it is indeed very annoying to get CSVs to render nicely on a terminal. 1. How does tidy-viewer compare with csvlook? 2. Looking at the demo video, there seems to be an odd fixation with "N/A". The CSV spec, AFAIK, doesn't recognize this phrase. I don't understand why someone would expect a quoted string field whose raw characters are "n/a" should be rendered as anything other than n/a (i.e. lowercase and without the quotes). I'm guessing maybe in your workflow you want to use that phrase a lot, but for a tool for the general public I'd not do this kind of interpretation; and I would leave an empty field as empty. 3. tidy-viewer seems to require "unstable library features", or at least ones which were unstable as of Rust 1.48.0 . It would be nice if you could be compatible with older rust distributions/versions. 4. Many systems, especially older ones, especially ones which you access remotely and don't have root privileges on, won't have a rust installation. It would be even more convenient if you could provide binaries with little or no extra dynamic library dependencies, which could be used on older / rustless systems. I realize this is a tall order, however. 5. What about scrolling? The worst part of viewing CSVs is having to handle wide ones which exceed the terminal width, and having decent horizontal as well as vertical scrolling ability. less doesn't cut it, because it doesn't keep the header row, plus it doesn't recognize field widths. 6. tidy-viewer does not seem to support wrapping longer fields onto multiple terminal lines. 7. When the user doesn't specify the color scheme, are you choosing one based on the terminal colors, or are you using absolute color values? I suggest the former. 8. tidy-viewer loads and parses the entire CSV immediately; and, in fact, seems to keep two copies of it in memory at once. This means it cannot be used with large files without thrashing; and even if your CSV does fit in global memory, it will still be kind of unusable, trying to dump gigabytes onto the terminal. Bottom line: A nice initial effort, but the more serious challenges are yet to be tackled, plus needs to be more robustly cross-platform.
- notafraudster 5y agoThe norm of treating missing data as NA exists in R (which the developer of this is clearly inspired by based on the GitHub readme.). Pandas in Python is stuck with NaN for numeric types (not quite correct) and "" or None for string types. Personally I like the choice to both explicitly render missing data in colour and to apply NA as a placeholder text to display that colour.
- contravariant 5y agoThe NA detection and higlighting is nice but I'm not sure how I feel about showing anything other than the exact textual value. I don't mind abridging quotes when they're not necessary, but showing "N/A", NA,, etc. as the same value is a bit iffy.
- flusteredBias 5y agoIt is rough. There are many ways that different tools put NAs, na, N/A, "", etc. in a file. To chose only "NA" would mean I would be excluding the output of other tools. I chose accessibility over specificity. #trade-offs.
- ayoubElk 5y agoMaybe you could just background highlight the empty cells?
- flusteredBias 5y agoWell, I think being promiscuous with "NA", "N/A", nan, etc. is a separate issue from a blank cell. A blank cell is literally missing. That should be filled with NA.
- nicoburns 5y ago> A blank cell is literally missing. That should be filled with NA. Why? "NA" stands for "Not Applicable", but a blank cell in a CSV could represent any number of things of which "Not Applicable" is only one.
- flusteredBias 5y agohaha. You are right "NA" stands for "Not Applicable". That is not always how people/programs using it though. What are some alternatives that you would suggest? I am happy to learn.
- berlinquin 5y agoCool project! I'm familiar with column, and this looks like a good replacement. Curious, how do you handle formatting on cells with long strings that need to overflow to multiple lines? As soon as you try to optimize the column widths for table length, you start hitting an NP-hard problem. https://quintenkent.com/content/column-problem.html https://quintenkent.com/content/column-problem.html
- flusteredBias 5y agoI actually read that article when I started making the package. You can see some of the input data here https://github.com/alexhallam/tv/blob/main/data/a.csv https://github.com/alexhallam/tv/blob/main/data/a.csv. I let the user chose how long the max column width should me then append "...". The default value is 20 characters.
- asicsp 5y agoHey, if you are able to edit the title of this submission, can you add `Show HN: ` prefix? See https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html for details.
- flusteredBias 5y agoBlast, I am sorry. It looks like I can't edit the title anymore. Otherwise I would make the change.
- sneak 5y agoYour asciinema playback made me twitch. Is the lack of a trailing space in your PS1 intentional?
- GrayShade 5y agoCool project (and written in my favourite language), but I really hate how there's no space at the end of your prompt.
- flusteredBias 5y agoLove the idea! Already opened the issue and will merge a PR before the next release.
- GrayShade 5y agoAh, I didn't mean a newline at the end of the output. Just a space after your shell prompt ($), see e.g. `user@~/code/data$cat` in https://github.com/alexhallam/tv/blob/main/img/column_v_tv2.png https://github.com/alexhallam/tv/blob/main/img/column_v_tv2.....
- flusteredBias 5y agolol. Got it.
- deleted 5y ago[deleted]
- stavros 5y agoI have to agree, I felt dirty watching the recording. The project looks great, though.
- BiteCode_dev 5y agoIt feels like when somebody opens a ( but doesn't close it.
- hoosieree 5y ago)
- 5y ago
- semireg 5y agoConsider adding a code snippet showing a.csv output. I had to watch a video just to see text.
- GrayShade 5y agoNot OP, but there's a screenshot under https://github.com/alexhallam/tv#1-na-comprehension https://github.com/alexhallam/tv#1-na-comprehension. Which is actually worse for screen reader users, I suppose.
- Someone 5y agoAnd a list of options. Reading the first 80 lines of https://github.com/alexhallam/tv/blob/main/src/main.rs https://github.com/alexhallam/tv/blob/main/src/main.rs was in some sense more educational than the readme. It, for example, allowed me to make an educated guess as to the answer to the question “how does this handle huge files?”. It by default only reads 25 lines. (That makes the example from the header: cat diamonds.csv | head -n 35 | tv a bad example. You shouldn’t need that head in-between. However, line 167 says //.take(row_display_option + 1) That seems to indicate this reads the entire file into memory, and that guess wasn’t that educated at all.
- flusteredBias 5y agoI have some work to do on the README. I will show the output better. The difficulty with showing the output only is that it does not capture the coloring. Maybe I will show the output, or add a picture, or have an animated gif. Maybe all three.
- rlue 5y agoWhy not visidata? https://www.visidata.org/ https://www.visidata.org/ https://www.youtube.com/watch?v=N1CBDTgGtOU https://www.youtube.com/watch?v=N1CBDTgGtOU (It does much, much more than pretty printing, but no reason you can't use it for that.)
- certifiedloud 5y agoVisidata is a great interactive tool. TV seems like it would be better when scripting, or in one-liners.
- dotancohen 5y agoFor scripting I would use grep and cut, maybe awk. For scripting with CSV files, at least in my experience, you usually want specific columns from specific lines. If TV had a switch for specifying only certain columns, that would make the job much easier.
- flusteredBias 5y agosounds like you are looking for xsv. I like that tool a lot for selecting specific columns.
- flusteredBias 5y agoI love visidata! But when I want to just glance at a csv file I reach for tv (I used to use `column` which is more of a tv competitor than visidata). This is for a couple reasons. 1. tv gives a quick summary of the count of rows and columns 2. tv shows all columns at the bottom that don't fit in the terminal. With vd I have to scroll on wide data. 3. tv guides the eye to missing data better with NA highlights 4. tv has sigfig logic that is better. I work with files where the decimal dust can become long. Those unnecessary characters pushes remaining columns off the screen. This means the user would need scroll over to see additional columns. I generally think it is better to avoid additional key presses if possible. 5. tv is fast for large files. It does not have to read and format all of the data like vd. tv is focused more on looking at the file and not operating on file. It does not have to do as much as vd. That helps tv with what it is uniquely good at. "Do one thing and do it well" It does not matter if your file is really wide (lots of columns) or really long -- tv will give the user a compact useful pretty print of the data. Why not use vd as a TUI spreadsheet and tv for glancing at csv files. They are both great tools in my eyes with different purposes.
- tambourine_man 5y agoThis looks great. How well does it handle the edge cases of CSV, like escaped commas, quoted text, escaped quotes, and all that fun, fun stuff?
- GrayShade 5y agoPretty well, I'd guess. It uses a well-tested CSV library.
- flusteredBias 5y agoIf you come across an edge case that tv does not handle then let me know. I will add a tests csv file as part of the current portfolio of test csvs. https://github.com/alexhallam/tv/tree/main/data https://github.com/alexhallam/tv/tree/main/data
- flusteredBias 5y agoI have been using tv now for a couple months at work. It has been working well on the data I see. If you find edge cases then please open an issue with an example csv.
- wly_cdgr 5y agoI thought this article was about a new way of understanding actual televisions
- brap 5y agoSame. Read this as some profound observation on what TVs actually are. Even thought to myself "huh, I guess OP does has a point..."
- BrianOnHN 5y ago> "huh, I guess OP does has a point..." LMAO, me too! Guess I've been spending too much time thinking. Edit: this reminds me of Jimmy Kimmel's segment where they "bleep and blur whether they need it or not" so that innocent TV clips appear to have profanity/innuendo etc.
- flusteredBias 5y agoLMAO. I wish.
- c3534l 5y agoWhat is CSV other than a matrix of data? Its matrices all the way down.
- TacticalCoder 5y agoIt's very weird for a project made to "maximize viewer enjoyment" to not put a space after the prompt. The one saved character on the line is definitely not worth the illegible resulting line: this doesn't maximize my enjoyment at all when viewing the examples.
- drcongo 5y agoSome might call this comment petty, but that was making my brain itch too.
- hnlmorg 5y agoThat's a really uncharitable comment considering it the developers prompt and has nothing to do with `tv` aside from appearing in the asciinema demo.
- stavros 5y agoThat's just the shell prompt on the recording machine. You can use your own prompt in your shell.
- dotancohen 5y agoVery nice! How does it handle CSVs that are wider or longer than the terminal? How does it deal with columns that are exceptionally long, or multiline? Often when working with large CSV files, I'll need to show or hide specific columns, especially if they are very long. Also, grepping the output for a specific line will hide the header as well, not to mention make the output unnecessarily wide if non-matching lines have longer fields than do the matching lines. So a built-in grepping feature would make this very useful.
- flusteredBias 5y ago> How does it handle CSVs that are wider or longer than the terminal? Columns that are exceptionally long but cutting and appending an ellipsis if the line is over 20 characters. > a built-in grepping feature would make this very useful. see the following for csv data manipulation: xsv - Command line csv data manipulation. Rust csvtk - Command line csv data manipulation. Go tsv-utils - Command line csv data manipulation toolkit. D q - Command line csv data manipulation query-like. Python miller - Command line data manipulation, statistics, and more. C
- dotancohen 5y agoTerrific, thank you!
- oaiey 5y agoThere should be a new package next to the traditional gnu tools containing the modern needed tools e.g. jq, curl or tv. Sometimes i really miss the extended sw package on some machines?
- flusteredBias 5y agoSounds like a good "awesome" page on github.
- b0afc375b5 5y agoI like httpie as an alternative to curl.
- stavros 5y agoThis is quite nice, but I don't like how it cuts off the output (instead of making it scrollable). Also, why require the use of `cat`? Accepting a filename so I can do `tv foo.csv` would be much more ergonomic, in my opinion.
- davidatbu 5y agoWhat is the value-add when compared to using `xsv` to pretty print? Is it only the fact that it highlights NA values?
- flusteredBias 5y agoxsv is one of my favorite data manipulation tools. Also, the author of that package is one of the best developers I know. I use xsv with tv. I normally pipe the output of xsv to tv. 1. As you noted NA comprehension 2. Column overflow logic for different sized terminals 3. Summary meta data in the header 4. Significant digits logic. This allows users to view more columns than they would otherwise view due to decimal dust shifting the columns over. 5. This is the most import! It looks really pretty!
- davidatbu 5y agoThanks!
- treve 5y agoI've made a Lotus 1-2-3 inspired CSV viewer for the terminal too. Had big plans for it, but it's just a basic viewer now: https://github.com/evert/csv123 https://github.com/evert/csv123
- flusteredBias 5y agoKeep at it. I think it looks great!
- ComputerGuru 5y agoXSV [0] can also pretty-print (minus the colors), but that's just the tip of the iceberg as far as what it can do. It's very handle for quick statistical analysis of CSV input. [0]: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv
- flusteredBias 5y agoI love xsv! I mention in the readme that command line data manipulation tools are great compliments to tv. https://github.com/alexhallam/tv#tools-to-pair-with-tv https://github.com/alexhallam/tv#tools-to-pair-with-tv
- baggiponte 5y agothat's exactly the comment I was looking for! xsv is super powerful and I think you might both draw inspiration from one another. I read above that tv reads everything into memory: maybe you can exploit some xsv tricks to avoid that. I feel tv looks great to visualise the outcome at the end of a pipeline, perhaps with xsv. I am no Ruby expert either, but this can become a cool Homebrew binary: people on macOS will use it too!
- flusteredBias 5y agoI will add some Homebrew installation instructions. That is now an open issue. I want this tool to be highly accessible. Again, xsv is the best. I like the idea of small utilities that specialize in a specific task. From the Unix philosophy: > Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
- baggiponte 5y agoSure! I completely understand and I am really glad you made this great tool to display the data in such a beautiful format. Maybe there can also be a “—head” flag to display only the top n rows by default, given that the data is read into memory?
- frankfrank13 5y agoBrings a tear to my eye. Everything else I've used is so heavy-handed, I just wanted the jq of CSV's
- sigg3 5y agoThis is kinda cool, but why not just use column? The video of catting into tv is equivalent to: column -s, -t FILE
- mixmastamyk 5y agoIt shows why in the readme, the benefits are small but real. Others have recommended improvements.
- 6502nerdface 5y agoA while ago, Two Sigma Investments open-sourced its own curses-based internal tool for pretty printing tabular data: https://github.com/twosigma/ngrid https://github.com/twosigma/ngrid
- claimred 5y agoWanted to mention that Windows PowerShell supports pretty CSV printing out of the box, like so Import-Csv .\Levels.csv | Format-Table Count Level elevation Level name Name Object type Unique ID ----- --------------- ---------- ---- ----------- --------- 1 -600.0000000 Store -0,600 - Store Level dc611fed-1783-d759-053a-b19848c51491 1 2850.0000000 Store +2,850 - Store Level c59f2ae4-0e94-6ea0-bd82-8306971e628c 1 3350.0000000 Roof +3,350 - Roof Level 7b487ac2-e102-dc23-6ad9-81c39124de1d
- slaymaker1907 5y agoPowerShell is actually pretty good at manipulating CSV and JSON. However, I would definitely recommend using v7 (i.e. pwsh) since it has many improvements over v5 (default on Windows). For example, Group-Object seems to be several orders of magnitude faster using the latest version.
- sixothree 5y agoThere's also ConsoleGridView. https://devblogs.microsoft.com/powershell/introducing-consoleguitools-preview/ https://devblogs.microsoft.com/powershell/introducing-consol...
- hnlmorg 5y agoShameless plug, but so does my shell, https://github.com/lmorg/murex https://github.com/lmorg/murex $ open test/example.csv | format generic Login email Identifier One-time password Recovery code First name Last name Department Location rachel@example.com 9012 12se74 rb9012 Rachel Booker Sales Manchester laura@example.com 2070 04ap67 lg2070 Laura Grey Depot London craig@example.com 4081 30no86 cj4081 Craig Johnson Depot London mary@example.com 9346 14ju73 mj9346 Mary Jenkins Engineering Manchester jamie@example.com 5079 09ja61 js5079 Jamie Smith Engineering Manchester My shell also aims to have closer compatibility with POSIX (albeit it's not a POSIX shell) so you can use all the same command line tools you're already familiar with too (which, for me at least, was the biggest hurdle in my adoption of PowerShell). It also supports other file types out of the box too. eg jsonlines $ open test/example.csv | format jsonl ["Login email","Identifier","One-time password","Recovery code","First name","Last name","Department","Location"] ["rachel@example.com","9012","12se74","rb9012","Rachel","Booker","Sales","Manchester"] ["laura@example.com","2070","04ap67","lg2070","Laura","Grey","Depot","London"] ["craig@example.com","4081","30no86","cj4081","Craig","Johnson","Depot","London"] ["mary@example.com","9346","14ju73","mj9346","Mary","Jenkins","Engineering","Manchester"] ["jamie@example.com","5079","09ja61","js5079","Jamie","Smith","Engineering","Manchester"]
- gurgeous 5y agoHey, great start. I spend half my day in CSVs and I am definitely your target audience. Most of the time I use bat, visidata or tabview. In many ways tabview is the best, though recently the project has been abandoned. tv looks excellent. Fun name. I think if you added a couple of features it would ascend to my toolbox: (1) scrolling (horizontal and vertical) (2) better command line parsing. Running "tv" without stdin or arguments should produce an error/help. Running "tv xyz.csv" should read that file. Good luck!
- udkl 5y agoWhat you need is https://www.visidata.org/ https://www.visidata.org/
- thamer 5y agoSome (most?) tools that output data in columns and fit each one to the largest value in that column need to scan the whole file as a first pass just to start displaying data. Not only is it the case with this tool, but from what I'm reading in main.rs it looks like it's also loading the whole file in memory. I was going to say that scanning the file was a deal-breaker, but if true this is much more resource-intensive. This looks like a nice tool, but these design choices seem to limit its use to relatively small files. It could be updated to have a read-ahead buffer instead and adjust its output as new lines are discovered with values of different width, although doing this without a jarring resize could be challenging. Could someone with better knowledge of Rust than mine confirm this? I see the full dataset being loaded here[1] and the column widths being computed here.[2] [1] https://github.com/alexhallam/tv/blob/main/src/main.rs#L183-L192 https://github.com/alexhallam/tv/blob/main/src/main.rs#L183-... [2] https://github.com/alexhallam/tv/blob/main/src/main.rs#L218-L221 https://github.com/alexhallam/tv/blob/main/src/main.rs#L218-...
- eevilspock 5y agoI like this idea. I don't think it would be jarring if the read-ahead buffer was a minimal number of lines, i.e. looking like distinct pages. The default could be at least the line height of the terminal, or some multiple. There could be an option to redisplay the header row for resized "pages". There could be a CLI switch giving the user control, i.e. make everyone happy.
- killjoywashere 5y agoI think data scientists will recognize this problem, and there's a well-used solution: .head() Just show me the top 5 rows. That's all most people are looking for. cat data/a.csv | tv --head
- fiddlerwoaroof 5y agoOr: head -n5 data/a.csv | tv
- mejari 5y agoDoes it really need libc6 >= 2.31? Having problems installing it on Ubuntu 16.04 LTS because of that dependency.
- flusteredBias 5y agoOpen an issue. I would have to look into that.
- timwis 5y agocsvkit’s csvlook works similarly but outputs a markdown table.
- leptoniscool 5y agoCSV is still a major part of the development ecosystem, amazing that it has such staying power after all these years.
- shantun007 5y agoThere is something like on spread sheet https://www.coodingdessign.com/javascript/api-tutorial-for-beginners-with-google-sheets/ https://www.coodingdessign.com/javascript/api-tutorial-for-b...
- watersb 5y agoThis is beautiful. Perl could do this, but I have to think about it every time. Often, I don't want to think.
- xtat 5y agoVery handy thanks for sharing.
- flusteredBias 5y agoThanks!
- robbiejs 5y agoSeems like a great command line tool. Congrats! FYI, I have made a browser-based tool for viewing and editing CSV files. Super fast, with Excel-like controls: https://editcsvonline.com https://editcsvonline.com.
- flusteredBias 5y agoThanks and great work!