3 ms·

I spend a lot of time in the terminal and want to quickly glance at a csv files without making a new script, opening excel, or using a tui. I made tidy-viewer (

I spend a lot of time in the terminal and want to quickly glance at a csv files without making a new script, opening excel, or using a tui. I made tidy-viewer (tv) because current tools like cat and column were not pretty enough. tv modifies raw files in the following ways: 1. NA detection and highlighting 2. Printing only significant digits 3. Header and footer meta data I have been using this a lot at work. There is a lot more work to do, but it is in a usable state. Give it a try! If you like it then star on Github!

IgorPartola 5y ago
This looks great! I wonder how long it’ll be until someone posts a long ask snippet that will do something similar and claim this isn’t progress, but rest assured that they are wrong. I’m adding tv to my toolbox.
- flusteredBias 5y ago
  Thanks! I appreciate the compliment!
mileza 5y ago
I think it's a great first effort, but there are a number of possible improvements to do. The most obvious one would be to support passing the file as an argument instead of using cat or the redirection operator every time. It's great that it works with stdin to allow piping into it, but it's cumbersome if you just want to take a file and print it, which will no doubt be a common use case.
- MobiusHorizons 5y ago
  Do you think ‘tv <file.csv’ does what you want well enough? What is the behavior when you run ‘tv file.csv’ does it just block waiting for input? I think it’s great for a visualizer like this to encourage people to get used to the power of shell pipelines if possible.
  - Someone 5y ago
    I think it’s a bad idea to go against half a century of conventions without good reason. It’s very surprising to see a tool that only works as a filter, and doesn’t take file paths as arguments.
  - GuB-42 5y ago
    It works, but almost all UNIX commands that work on pipelines can take a list of files as arguments. Out of the commands I use regularly, "patch" is the only one that works exclusively from stdin, probably because file arguments have a different, somewhat obscure, and probably historical meaning. If appropriate, using files as arguments instead of using shell pipelines is a best practice. Commands can optimize for that use case, print better error messages, etc... And it is not a good thing to encourage useless use of cat. If you goal is to show how your tool is to be used with pipelines, show an actually useful pipeline for example "sed '1b;/abc/!d' file.csv | tv". The "sed" command prints the first line (header), and all lines containing "abc".
    - MobiusHorizons 5y ago
      Fair enough.
moritonal 5y ago
Man.. TV is such a good name for a visualiser tool that it'd be excellent if it could pretty-print any content given to it. Does your framework support the idea of pre-parsing the file's content and selecting an appropriate renderer, or is it fairly tied to CSVs?
- flusteredBias 5y ago
  I would need more details on what you mean by "pre-parsing". It works with any deliminator, it could be comma-separated, pipe-separated, etc.
  - mixmastamyk 5y ago
    Doesn't need to be pre-parsed. Perhaps give the filename to the utility instead of content via stdin. Then filename gives a hint. If there is none, run "file filename" (via library) beforehand.
- inostia 5y ago
  I actually like `tidy` better and honestly would rather have `cat a.csv | tidy ...`. But it's probably already a thing.
  - hnlmorg 5y ago
    Nice idea. I might actually work on this myself
  - flusteredBias 5y ago
    You don't have to make the alias `tv`. Feel free to make your alias tidy='tidy-viewer'.
einpoklum 5y ago
First of all - kudos on tackling this task - it is indeed very annoying to get CSVs to render nicely on a terminal. 1. How does tidy-viewer compare with csvlook? 2. Looking at the demo video, there seems to be an odd fixation with "N/A". The CSV spec, AFAIK, doesn't recognize this phrase. I don't understand why someone would expect a quoted string field whose raw characters are "n/a" should be rendered as anything other than n/a (i.e. lowercase and without the quotes). I'm guessing maybe in your workflow you want to use that phrase a lot, but for a tool for the general public I'd not do this kind of interpretation; and I would leave an empty field as empty. 3. tidy-viewer seems to require "unstable library features", or at least ones which were unstable as of Rust 1.48.0 . It would be nice if you could be compatible with older rust distributions/versions. 4. Many systems, especially older ones, especially ones which you access remotely and don't have root privileges on, won't have a rust installation. It would be even more convenient if you could provide binaries with little or no extra dynamic library dependencies, which could be used on older / rustless systems. I realize this is a tall order, however. 5. What about scrolling? The worst part of viewing CSVs is having to handle wide ones which exceed the terminal width, and having decent horizontal as well as vertical scrolling ability. less doesn't cut it, because it doesn't keep the header row, plus it doesn't recognize field widths. 6. tidy-viewer does not seem to support wrapping longer fields onto multiple terminal lines. 7. When the user doesn't specify the color scheme, are you choosing one based on the terminal colors, or are you using absolute color values? I suggest the former. 8. tidy-viewer loads and parses the entire CSV immediately; and, in fact, seems to keep two copies of it in memory at once. This means it cannot be used with large files without thrashing; and even if your CSV does fit in global memory, it will still be kind of unusable, trying to dump gigabytes onto the terminal. Bottom line: A nice initial effort, but the more serious challenges are yet to be tackled, plus needs to be more robustly cross-platform.
- notafraudster 5y ago
  The norm of treating missing data as NA exists in R (which the developer of this is clearly inspired by based on the GitHub readme.). Pandas in Python is stuck with NaN for numeric types (not quite correct) and "" or None for string types. Personally I like the choice to both explicitly render missing data in colour and to apply NA as a placeholder text to display that colour.
contravariant 5y ago
The NA detection and higlighting is nice but I'm not sure how I feel about showing anything other than the exact textual value. I don't mind abridging quotes when they're not necessary, but showing "N/A", NA,, etc. as the same value is a bit iffy.
- flusteredBias 5y ago
  It is rough. There are many ways that different tools put NAs, na, N/A, "", etc. in a file. To chose only "NA" would mean I would be excluding the output of other tools. I chose accessibility over specificity. #trade-offs.
  - ayoubElk 5y ago
    Maybe you could just background highlight the empty cells?
    - flusteredBias 5y ago
      Well, I think being promiscuous with "NA", "N/A", nan, etc. is a separate issue from a blank cell. A blank cell is literally missing. That should be filled with NA.
      - nicoburns 5y ago
        > A blank cell is literally missing. That should be filled with NA. Why? "NA" stands for "Not Applicable", but a blank cell in a CSV could represent any number of things of which "Not Applicable" is only one.
        flusteredBias 5y ago
        haha. You are right "NA" stands for "Not Applicable". That is not always how people/programs using it though. What are some alternatives that you would suggest? I am happy to learn.
berlinquin 5y ago
Cool project! I'm familiar with column, and this looks like a good replacement. Curious, how do you handle formatting on cells with long strings that need to overflow to multiple lines? As soon as you try to optimize the column widths for table length, you start hitting an NP-hard problem. https://quintenkent.com/content/column-problem.html https://quintenkent.com/content/column-problem.html
- flusteredBias 5y ago
  I actually read that article when I started making the package. You can see some of the input data here https://github.com/alexhallam/tv/blob/main/data/a.csv https://github.com/alexhallam/tv/blob/main/data/a.csv. I let the user chose how long the max column width should me then append "...". The default value is 20 characters.
asicsp 5y ago
Hey, if you are able to edit the title of this submission, can you add `Show HN: ` prefix? See https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html for details.
- flusteredBias 5y ago
  Blast, I am sorry. It looks like I can't edit the title anymore. Otherwise I would make the change.
sneak 5y ago
Your asciinema playback made me twitch. Is the lack of a trailing space in your PS1 intentional?

IgorPartola 5y ago

This looks great! I wonder how long it’ll be until someone posts a long ask snippet that will do something similar and claim this isn’t progress, but rest assured that they are wrong. I’m adding tv to my toolbox.

flusteredBias 5y ago
Thanks! I appreciate the compliment!

mileza 5y ago

I think it's a great first effort, but there are a number of possible improvements to do. The most obvious one would be to support passing the file as an argument instead of using cat or the redirection operator every time. It's great that it works with stdin to allow piping into it, but it's cumbersome if you just want to take a file and print it, which will no doubt be a common use case.

MobiusHorizons 5y ago
Do you think ‘tv <file.csv’ does what you want well enough? What is the behavior when you run ‘tv file.csv’ does it just block waiting for input? I think it’s great for a visualizer like this to encourage people to get used to the power of shell pipelines if possible.
- Someone 5y ago
  I think it’s a bad idea to go against half a century of conventions without good reason. It’s very surprising to see a tool that only works as a filter, and doesn’t take file paths as arguments.
- GuB-42 5y ago
  It works, but almost all UNIX commands that work on pipelines can take a list of files as arguments. Out of the commands I use regularly, "patch" is the only one that works exclusively from stdin, probably because file arguments have a different, somewhat obscure, and probably historical meaning. If appropriate, using files as arguments instead of using shell pipelines is a best practice. Commands can optimize for that use case, print better error messages, etc... And it is not a good thing to encourage useless use of cat. If you goal is to show how your tool is to be used with pipelines, show an actually useful pipeline for example "sed '1b;/abc/!d' file.csv | tv". The "sed" command prints the first line (header), and all lines containing "abc".
  - MobiusHorizons 5y ago
    Fair enough.

moritonal 5y ago

Man.. TV is such a good name for a visualiser tool that it'd be excellent if it could pretty-print any content given to it. Does your framework support the idea of pre-parsing the file's content and selecting an appropriate renderer, or is it fairly tied to CSVs?

flusteredBias 5y ago
I would need more details on what you mean by "pre-parsing". It works with any deliminator, it could be comma-separated, pipe-separated, etc.
- mixmastamyk 5y ago
  Doesn't need to be pre-parsed. Perhaps give the filename to the utility instead of content via stdin. Then filename gives a hint. If there is none, run "file filename" (via library) beforehand.
inostia 5y ago
I actually like `tidy` better and honestly would rather have `cat a.csv | tidy ...`. But it's probably already a thing.
- hnlmorg 5y ago
  Nice idea. I might actually work on this myself
- flusteredBias 5y ago
  You don't have to make the alias `tv`. Feel free to make your alias tidy='tidy-viewer'.

einpoklum 5y ago

First of all - kudos on tackling this task - it is indeed very annoying to get CSVs to render nicely on a terminal. 1. How does tidy-viewer compare with csvlook? 2. Looking at the demo video, there seems to be an odd fixation with "N/A". The CSV spec, AFAIK, doesn't recognize this phrase. I don't understand why someone would expect a quoted string field whose raw characters are "n/a" should be rendered as anything other than n/a (i.e. lowercase and without the quotes). I'm guessing maybe in your workflow you want to use that phrase a lot, but for a tool for the general public I'd not do this kind of interpretation; and I would leave an empty field as empty. 3. tidy-viewer seems to require "unstable library features", or at least ones which were unstable as of Rust 1.48.0 . It would be nice if you could be compatible with older rust distributions/versions. 4. Many systems, especially older ones, especially ones which you access remotely and don't have root privileges on, won't have a rust installation. It would be even more convenient if you could provide binaries with little or no extra dynamic library dependencies, which could be used on older / rustless systems. I realize this is a tall order, however. 5. What about scrolling? The worst part of viewing CSVs is having to handle wide ones which exceed the terminal width, and having decent horizontal as well as vertical scrolling ability. less doesn't cut it, because it doesn't keep the header row, plus it doesn't recognize field widths. 6. tidy-viewer does not seem to support wrapping longer fields onto multiple terminal lines. 7. When the user doesn't specify the color scheme, are you choosing one based on the terminal colors, or are you using absolute color values? I suggest the former. 8. tidy-viewer loads and parses the entire CSV immediately; and, in fact, seems to keep two copies of it in memory at once. This means it cannot be used with large files without thrashing; and even if your CSV does fit in global memory, it will still be kind of unusable, trying to dump gigabytes onto the terminal. Bottom line: A nice initial effort, but the more serious challenges are yet to be tackled, plus needs to be more robustly cross-platform.

notafraudster 5y ago
The norm of treating missing data as NA exists in R (which the developer of this is clearly inspired by based on the GitHub readme.). Pandas in Python is stuck with NaN for numeric types (not quite correct) and "" or None for string types. Personally I like the choice to both explicitly render missing data in colour and to apply NA as a placeholder text to display that colour.

contravariant 5y ago

The NA detection and higlighting is nice but I'm not sure how I feel about showing anything other than the exact textual value. I don't mind abridging quotes when they're not necessary, but showing "N/A", NA,, etc. as the same value is a bit iffy.

flusteredBias 5y ago
It is rough. There are many ways that different tools put NAs, na, N/A, "", etc. in a file. To chose only "NA" would mean I would be excluding the output of other tools. I chose accessibility over specificity. #trade-offs.
- ayoubElk 5y ago
  Maybe you could just background highlight the empty cells?
  - flusteredBias 5y ago
    Well, I think being promiscuous with "NA", "N/A", nan, etc. is a separate issue from a blank cell. A blank cell is literally missing. That should be filled with NA.
    - nicoburns 5y ago
      > A blank cell is literally missing. That should be filled with NA. Why? "NA" stands for "Not Applicable", but a blank cell in a CSV could represent any number of things of which "Not Applicable" is only one.
      - flusteredBias 5y ago
        haha. You are right "NA" stands for "Not Applicable". That is not always how people/programs using it though. What are some alternatives that you would suggest? I am happy to learn.

berlinquin 5y ago

Cool project! I'm familiar with column, and this looks like a good replacement. Curious, how do you handle formatting on cells with long strings that need to overflow to multiple lines? As soon as you try to optimize the column widths for table length, you start hitting an NP-hard problem. https://quintenkent.com/content/column-problem.html https://quintenkent.com/content/column-problem.html

flusteredBias 5y ago
I actually read that article when I started making the package. You can see some of the input data here https://github.com/alexhallam/tv/blob/main/data/a.csv https://github.com/alexhallam/tv/blob/main/data/a.csv. I let the user chose how long the max column width should me then append "...". The default value is 20 characters.

asicsp 5y ago

Hey, if you are able to edit the title of this submission, can you add `Show HN: ` prefix? See https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html for details.

flusteredBias 5y ago
Blast, I am sorry. It looks like I can't edit the title anymore. Otherwise I would make the change.

sneak 5y ago

Your asciinema playback made me twitch. Is the lack of a trailing space in your PS1 intentional?