17 ms·
Modernizing AWK, a 45-year old language, by adding CSV support
- mattewong 4y agothanks for this. am looking at the benchmarks. how do I get huge.csv? Don't see how to fetch or generate
- mattewong 4y agoFYI I ran on the worldcities data at https://github.com/petewarden/dstkdata https://github.com/petewarden/dstkdata (credit to xsv for choosing that dataset) against https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv and https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv (full disclosure: I am one of the zsv authors). Here's what I got. fastest to slowest: zsv (0.07), xsv (0.16), goawk (0.42), python (~1.6) Obviously, does not tell the whole story as this test was limited to "count" and an interpreted language is expected to always be slower compared to a precompiled command, but, it might be relevant to a user deciding what tool to use. Also, might be instructive as to room for improvement in the go code (or possibly the go code could use the c lib)-- I note that even if the goawk command is '{}' the runtime is still about the same. full results: --- goawk: 1000001 7000007 real 0m0.435s user 0m0.435s sys 0m0.031s 1000001 7000007 real 0m0.413s user 0m0.419s sys 0m0.024s 1000001 7000007 real 0m0.425s user 0m0.430s sys 0m0.024s xsv: 1000000 real 0m0.157s user 0m0.141s sys 0m0.013s 1000000 real 0m0.156s user 0m0.141s sys 0m0.012s 1000000 real 0m0.158s user 0m0.142s sys 0m0.013s zsv: 1000000 real 0m0.066s user 0m0.053s sys 0m0.010s 1000000 real 0m0.077s user 0m0.060s sys 0m0.012s 1000000 real 0m0.069s user 0m0.056s sys 0m0.010s python: 1000001 7000007 real 0m1.589s user 0m1.553s sys 0m0.026s 1000001 7000007 real 0m1.583s user 0m1.550s sys 0m0.025s 1000001 7000007 real 0m2.122s user 0m1.675s sys 0m0.037s --- The script for this was: --- echo 'goawk:' (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs echo 'xsv:' (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs echo 'zsv:' (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs echo 'python:' (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs
- benhoyt 4y agoThanks for that! zsv looks amazing. Yeah, it's definitely going to whip GoAWK, what with SIMD parsing and careful memory handling. I've done a couple of basic things for GoAWK's CSV performance, but haven't profiled or looked at allocation bottlenecks (absolute performance definitely wasn't my first focus). Yeah, sorry about huge.csv -- I found it online somewhere originally by searching for something like "large csv example", but can't for the life of me find it now. It's a monster 1.5GB file with 286 columns including quoted fields (whereas worldcitiespop only has a few columns and it doesn't look like it has quoted fields). I can upload to a file transfer service and send a link to you if you want ... though I should really update my benchmarks to use an easily-downloadable file instead.
- mattewong 4y agoNot a problem at all. Nice work with goawk!
- benhoyt 4y agoI've now updated the benchmarks to avoid huge.csv: the write benchmarks now write a big 1GB, 20-column CSV and the read benchmarks use that same CSV: https://github.com/benhoyt/goawk/commit/07eb4505a7f64cffceeb7d8579acc70ce3e96689 https://github.com/benhoyt/goawk/commit/07eb4505a7f64cffceeb...
- rgoulter 4y agoI'd be curious to see a comparison with the csvkit suite. https://csvkit.readthedocs.io/en/latest/index.html https://csvkit.readthedocs.io/en/latest/index.html
- benhoyt 4y agoThat's written in Python and uses the "agate" library which uses Python's built-in "csv" module. I did a couple of simple benchmarks against Python's csv module in the article: https://benhoyt.com/writings/goawk-csv/#performance https://benhoyt.com/writings/goawk-csv/#performance (Go/GoAWK is a bit more than 3x as fast) I also did a quick test using csvcut to pull out a single field, and compared it to GoAWK. Looks like GoAWK is about 4x as fast here: $ time csvcut -c agency_id huge.csv >/dev/null real 0m25.977s user 0m25.240s sys 0m0.424s $ time goawk -i csv -H -o csv '{ print @"agency_id" }' huge.csv >/dev/null real 0m6.584s user 0m7.434s sys 0m0.480s
- rgoulter 4y agoI meant more in terms of user/developer experience. I imagine there's some use case where AWK will be able to come up with output that csvkit can't. But for simple cases, csvkit's invocation is easier to remember.
- benhoyt 4y agoAh, true -- that would be a good comparison to do, or section to add the the docs (csv.md in the repo). I'll add it to my to-do list!
- malkocoglu 4y agoModernizing ... by adding CSV support !?!
- benhoyt 4y agoFair comeback. I think of CSV as modern, but Wikipedia tells me it's almost as old as AWK (depending on how you count). It seems to me it's used more heavily now as an exchange format, compared to say 15-20 years ago, but I could be wrong.
- londons_explore 4y agoJSON is an exchange format... sqlite is an exchange format... even protocol buffers are an exchange format... CSV is only an exchange format if there is no user generated strings in the data... If there are, then you'll almost certainly screw up the encoding when someones name has a newline or comma or speech mark in it, or some obscure unicode etc. Even moreso if awk is part of your toolkit.
- benhoyt 4y agoThat may have been more true years ago, but now quoting is pretty well defined with RFC 4180, and most tools seem to produce and consume RFC 4180-compatible CSV (which properly handles commas, quotes, and even newlines in fields). That said, there still are too many non-standard or quirky CSV files out there.
- dspillett 4y ago> and most tools seem to produce and consume RFC 4180-compatible CSV Laughs in SSIS… There are some significant tools (or common add-ins for them) that don't entirely respect RFC4180. Though I see few files that breach it these days, thre are tools that break with conforming files (looking at you, Excel, trying to be clever about anything isn't conclusively provable not to be a date). Our clients use it all the time, to the point where we'd lose sales if we didn't support it, but CSV is far from a safe way to transport data IMO. Each time a new requirement to deal with CSV comes in I treat it as a custom format that may or may not be something like RFC4180.
- skanga 4y agoI always use mawk for its performance. This may be worth a try ...
- benhoyt 4y agoIf you're just using AWK features, mawk is still the fastest. GoAWK is faster than awk in most cases, on a par with gawk, but still not as fast as mawk (see https://benhoyt.com/writings/goawk-compiler-vm/#virtual-machine-results https://benhoyt.com/writings/goawk-compiler-vm/#virtual-mach...) ... except for scripts that make heavy use of regexen. Unfortunately Go's regexp package is still quite slow. You could also try frawk, which is a JIT-optimized AWK written in Rust -- it's really fast, and shares some of the CSV features (but not the @"named-field" syntax). But for most everyday usage, even for large inputs, GoAWK's performance is quite sufficient. The CSV support, and its use as a Go library -- they're more important to me than raw speed at this point.
- skanga 4y agoThanks for such a detailed and informative answer. Much appreciated.
- jeroenjanssens 4y agoI often use csvquote [1] whenever I need to process CSV with a command-line tool that doesn't support it. For example: csvquote test.csv | awk '{print $1, $2}' | csvquote -u [1] https://github.com/dbro/csvquote https://github.com/dbro/csvquote
- throwawayboise 4y agoYep, was going to post the same if someone hadn't already. This is the unix way. Small tools with narrow focus, strung together in pipelines.
- sophacles 4y agoawk is a tool for handling records in text. csv is a textual format for records. I'd argue that needing a different tool for handling records in text so that you can pass it to a tool for handling records in text is a bit too far.
- jcelerier 4y agoah yes, awk's famously narrow focus, covering things such as networking (https://www.gnu.org/software/gawk/manual/html_node/TCP_002fIP-Networking.html https://www.gnu.org/software/gawk/manual/html_node/TCP_002fI...) or video games (https://github.com/mikkun/AWKTC https://github.com/mikkun/AWKTC). What do you mean your blog isn't served by awk (https://www.gnu.org/software/gawk/manual/gawkinet/html_node/Primitive-Service.html#Primitive-Service https://www.gnu.org/software/gawk/manual/gawkinet/html_node/...) as a VRML world (https://www.gnu.org/software/gawk/manual/gawkinet/html_node/MAZE.html#MAZE https://www.gnu.org/software/gawk/manual/gawkinet/html_node/...) ?
- MonkeyClub 4y ago>> This is the unix way. Small tools with narrow focus, strung together in pipelines. > ah yes, awk's famously narrow focus, [...] "GNU's Not Unix"
- 4y ago
- ognyankulev 4y agoInstead of yet another limited parser, it would be best if universal tabular data parsing is supported by allowing one to specify all important parsing parameters, as described in https://www.w3.org/TR/2015/REC-tabular-metadata-20151217/#dialect-descriptions https://www.w3.org/TR/2015/REC-tabular-metadata-20151217/#di...
- junon 4y agoThose parameters are URL/Header/descriptor parameters. They don't live in the CSV itself.
- ognyankulev 4y agoI meant to have these parameters as awk options, supplied on command line, from envvar, or maybe even as awk variables.
- asicsp 4y ago>A big thank-you to the library of the University of Antwerp, who sponsored this feature. They’re one of two major teams or projects I know of that use GoAWK – the other one is the Benthos stream processor. That's great to hear. Are you planning to add support for xml, json, etc next? Something like Python's `json` module that gives you a dictionary object.
- benhoyt 4y agoI'm not considering adding general structured formats like XML or JSON, as they don't fit AWK's "fields and records" model or its simplistic data structure (the associative array) very well. However, I have considered adding JSON Lines support, where each line is a record, and fields are indexable using the new @"named-field" syntax (possibly nested like @"foo.bar").
- asicsp 4y agoThanks, JSON Lines support would be welcome too :)
- andi999 4y agoCan you also set the decimal separator? Some countries use ',' in numbere like 10,5
- ta8645 4y agoCan we please, please, stop that? If we go Metric, will you please standardize on period as decimal point? Shake on it. Let's make it happen.
- bloak 4y agoSomebody told me that the official international standard prefers comma; a period is just a permitted alternative. Of course, comma as decimal separator isn't very compatible with programming languages. However, the decimal separator that I was taught at primary school and have used in handwriting all my life is neither comma, nor period, but '·'.
- Frost1x 4y agoWhat, why? I've seen comma decimal separated values but I thought it was a weird English thing. What do these places use to separate larger values (e.g. each 10^3 magnitude increase) to make manual counting easier. 828.497.171.614,2?
- franga2000 4y agoExactly. The "official way" of writing big numbers around here (I know at least the balkans, germany and austria) is `1.000.000,00 €` (1 million). So what you wrote, but always at least 2 decimal places (or zero if it's not a financial document).
- Chris2048 4y agoDoh, Having two opposite conventions is a PITA. But if I add my 2 cents - I prefer 1,000.00 because I think it is similar to commas and full-stops in writing, where a comma is mainly a reading/speaking guide to help break up a sentence (or in this case a number) and a full stop is a much harder termination of a sentence (in the case of a number the official origin where int/non-fractional part ends).
- jph 4y agoBen this is great, thank you. Would you consider adding Unicode Separated Values (USV)? https://github.com/sixarm/usv https://github.com/sixarm/usv USV is like CSV and simpler because of no escaping and no quoting. I can donate $50 to you or your charity of choice as a token of thanks and encouragement.
- tomsmeding 4y ago> no escaping That makes no sense. Sure, they've chosen significantly less common separator characters than something like ','. but they are still characters that may appear in the data. How do you represent a value containing ␟ Unit Separator in USV? In-band signalling is ever going to remain in-band signalling. And in-band signalling will need escaping.
- jph 4y agoUSV has no escaping on purpose because it's simpler and faster. USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. If you need escaping, then I can suggest you consider USVX which is USV + extensions for escaping and other conveniences, or you can use your content's own escaping such as ampersand escaping for HTML text, or you can use any other format such as JSON, XML, SQL, etc.
- layer8 4y agoThis just creates another “CSV” were you don’t know which format variation was actually used.
- tomsmeding 4y ago> USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. I agree that wanting to put one of these characters in a csv is super rare. The vast majority of use cases would not notice that certain inout characters are prohibited. But certain inout characters _are_ prohibited, which means that encoders must check this case, and tools using those encoders neex to handle its failure. In a way, the fact that failure will be super rare will make it more dangerous, because people will omit these checks because "it won't happen for them" -- and then at some point it will wreak havoc. And with USVX all we've gained is less backslashes but more bytes per separator, so for usual data that doesn't contain a comma in every field, the USVX encoding will even increase file size, without requiring less code anywhere. I admit there is no ideal solution, though; while out-of-band signalling (i.e. length-prefixing) avoids all of these issues (which is why binary formats are universally length-prefixing formats), escaped formats are work much better for humans. And if a human wouldn't need to read it, one wouldn't use csv anyway, usually.
- hawski 4y agoIt is a nice addition, but I would like to see this taken further - structural regular expression awk. It is waiting to be implemented for 35 years now.
- mro_name 4y agoI recently learned via https://news.ycombinator.com/item?id=31257248 https://news.ycombinator.com/item?id=31257248 that ASCII has the idea of records and fields ever since. It's just not used, but workaround CSV. No improvement of CSV handling will ever improved on that.
- cb321 4y agoSee the "Conventions for lossless conversion to TSV" on the current version of https://en.wikipedia.org/wiki/Tab-separated_values https://en.wikipedia.org/wiki/Tab-separated_values . There are really only 3 chars to escape - the escape char, and 2 delimiters - to make everything easy to deal with (even binary data in fields - what "lossless" means here).
- belter 4y agoDiscussed here before, not everybody was seeing only positives. "ASCII Delimited Text – Not CSV or TAB delimited text": https://news.ycombinator.com/item?id=7474600 https://news.ycombinator.com/item?id=7474600
- Chris2048 4y agoA big problem is tooling - what they currently support, and the fact that excel will f*ck up anything. I suppose the best way would be a suite of tools to edit them that are compatible with existing editors? TBH a great value-add for adoption would be versioning s.t. you can see (and revert) when other tools mess up your files.
- RhysU 4y agoCould one get away with regular AWK using these combined with a streaming to/from CSV converter?
- cb321 4y agoNot just "regular awk", but "regular most things", like head, tail, grep, etc. The "things" just need to be 8-bit/NUL handling clean and the data could even have binary fields. I mean, if there is a library like Python/Go/etc. that you especially trust to write a streaming converter then that is a fine approach, but otherwise see both https://news.ycombinator.com/item?id=31352517 https://news.ycombinator.com/item?id=31352517 and https://news.ycombinator.com/item?id=31352704 https://news.ycombinator.com/item?id=31352704 (at least!)
- cb321 4y agoWhen you have "format wars", the best idea is usually to have a converter program change to the easiest to work with format - unless this incurs a space explosion as per some image/video formats. With CSV-like data, bulk conversion from quoted-escaped RFC4180 CSV to a simpler-to-parse format is the best plan for several reasons. First, it may "catch on", help Microsoft/R/whoever embrace the format and in doing so squash many bugs written by "data analyst/scientist coders". Second, in a shell "a|b" runs programs a & b in parallel on multi-core and allow things like csv2x|head -n10000|b or popen("csv2x foo.csv"). Third, bulk conversion to a random access file where literal delimiters cannot occur as non-delimiters allows trivial file segmentation to be nCores times faster (under often satisfied assumptions). There are some D tools for this bulk convert in https://github.com/eBay/tsv-utils https://github.com/eBay/tsv-utils and a much smaller stand-alone Nim tool https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim . Optional quoting was always going to be a PITA due to its non-locality. What if there is no quote anywhere? Fourth, by using a program as the unit of modularity in this case, you make things programming language agnostic. Someone could go to town and write a pure SIMD/AVX512 converter in assembly even and solve the problem "once and for all" on a given CPU. The problem is actually just simple enough that this smells possible. I am unaware of any "document" that "standardizes" this escaped/lossless TSV format. { Maybe call it "DSV" for delimiter separated values where "delimiters actually separate"? Ironically redundant. ;-) } Someone want to write an RFC or point to one? It can be just as "general/lossless" (see https://news.ycombinator.com/item?id=31352170 https://news.ycombinator.com/item?id=31352170). Of course, if you are going to do a lot of data processing against some data, it is even better to parse all the way to down to binary so that you never have to parse again (well, unless you call CPUs loading registers "parsing") which is what database systems have been doing since the 1960s.
- hedora 4y agoSomeone linked a wikipedia format guide for TSV, but the world seems to have settled on using the escape codes \\, \t, \n with their obvious meanings, then allowing arbitrary binary. That should be parallelism friendly, even with UTF-8, where an ascii tab or newline byte always mean tab and newline.
- torginus 4y agoI can't tell whether the UNIX people have lost their way, or just the demands of modern shell scripts cannot be met by typical shell philosophy - that is, piping together the output of small, orthogonal utilities. The emergence and constantly increasing complexity of these small, bespoke DSLs like this or jq does not inspire confidence in me.
- jjtheblunt 4y agoyou made me think a possible corollary (?) question would be if the json people don't perfectly overlap with the unix people.
- anthk 4y agoA single tool to parse JSON and pipe it? That's Unix too.
- tonyg 4y ago> demands of modern shell scripts cannot be met by typical shell philosophy That. Pipes and unstructured binary data isn't compositional enough, making the divide between the kinds of things you can express in the language you use to write a stage in a pipeline and the kinds of things you can express by building a pipeline too large.
- igtztorrero 4y agoone step further towards GoUnix, Gonux or any name Gophers like!
- anthk 4y agoJust use 9front. The plan9 C borrows a lot of philosophy from Golang (well, the reverse), and it's from the same creators.
- demouser345 4y agono thank you. i will vehemently fight this hivemind
- motohagiography 4y agoThank you! In non-technical environments, shell scripting with awk is a superpower, and it's almost always on csv data.
- gpvos 4y agoDuring a recent HN discussion on pipes and text versus structured objects to transfer data between programs, I started wondering if CSV wouldn't be a nice middle ground.
- carlmr 4y agoI still think objects are great, but PowerShell makes it so hard to deal with them. I think F#-interactive (FSI) with its Hindley-Milner type-inference, would have been a much better base for a shell.
- gpvos 4y agoI'm not familiar with F#, but I do hate CSV tools that try type inference on data; in my opinion the csvkit tools should have the -I option on by default.
- carlmr 4y agoF# does type inference like this: let x = 1; x is now an integer. Now you can't do printfn "%s" x; Only printfn "%i" x; So it has strong static typing, but most of the time you don't need to be explicit about them. It can even infer function types. If you use FSI (F# interactive( it will always print the signatures in between, so that it's really easy to explore interfaces.
- majkinetor 4y ago> I still think objects are great, but PowerShell makes it so hard to deal with them. LOL, this is amazing...
- carlmr 4y agoThe thing is, I've used .NET a lot, and C# and F# I can code in my sleep. The same object system, integrated in PowerShell, makes it really hard to use.
- 4y ago
- uhtred 4y agoWhy not use FPAT: https://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html https://www.gnu.org/software/gawk/manual/html_node/Splitting...
- forgotpwd16 4y agoFrom same documentation but the "more CSV" link: >In general, using FPAT to do your own CSV parsing is like having a bed with a blanket that’s not quite big enough. There’s always a corner that isn’t covered. We recommend, instead, that you use Manuel Collado’s CSVMODE library for gawk.
- forgotpwd16 4y agoA good and useful addition. There's a mention to CSVMODE, a gawk library. I wonder if it could be extended to support the functionality that goawk's `-i csv` has.
- db65edfc7996 4y agoI have grown fond of using miller[0] to handle command line data processing. Handles the standard tabular formats (csv, tsv, json) and has all of the standard data cleanup options. Works on streams so (most operations) are not limited by memory. [0]: https://github.com/johnkerl/miller https://github.com/johnkerl/miller
- pstuart 4y agoJust starred the repo -- thanks for the tip!
- llimllib 4y agoxsv does similar stuff for CSV, and very rapidly: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv https://miller.readthedocs.io/en/latest/why/ https://miller.readthedocs.io/en/latest/why/ has a nice section on "why miller": > First: there are tools like xsv which handles CSV marvelously and jq which handles JSON marvelously, and so on -- but I over the years of my career in the software industry I've found myself, and others, doing a lot of ad-hoc things which really were fundamentally the same except for format. So the number one thing about Miller is doing common things while supporting multiple formats: (a) ingest a list of records where a record is a list of key-value pairs (however represented in the input files); (b) transform that stream of records; (c) emit the transformed stream -- either in the same format as input, or in a different format.
- cb321 4y agoOut of curiosity, I tried these two on that same data & computer as https://news.ycombinator.com/item?id=31356573 https://news.ycombinator.com/item?id=31356573 , mlr --c2t cat takes 2.96 seconds while xsv cat rows to /dev/null takes 0.434 seconds. So, 14.8X and 2.17X slower than that c2tsv Nim program to do exactly (and only) that conversion. But, yes, yes I am sure perf varies depending on quoting/escaping/column/etc. densities.
- mattewong 4y agoThis is not an outlier. `mlr` is quite slow, literally off-the-charts slow for our purposes when we benchmarked vs xsv and zsv (see https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv. disclaimer: I'm one of its authors)
- adamgordonbell 4y agoIt's somewhat a chore to use but gawkextlib has a CSV extension. so you can do this in gawk if the extension is loaded. @include "csv" BEGIN { CSVMODE = 1 } { print $2 } https://earthly.dev/blog/awk-csv/#gawkextlib https://earthly.dev/blog/awk-csv/#gawkextlib
- rufugee 4y agoWhat’s the best resource for learning modern awk these days? I’ve used it for decades, but only via memorized snippets…
- chasil 4y agoThe full PDF of "The AWK Programming Language" has been online for some time. Chapter 2 is the complete description of the language in 40 pages. I think this link works... https://ia803404.us.archive.org/0/items/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf https://ia803404.us.archive.org/0/items/pdfy-MgN0H1joIoDVoIC...
- altairprime 4y agoFor non-awk tools, csvformat (from csvkit) will unquote and re-delimeter a CSV file (-D\034 -U -B) into something that UNIX pipes can handle (cut -d\034, etc). It’s worth setting up as an alias, and you can store \034 in $D or whatever for convenience.
- adolph 4y agoFor anything down and dirty, what's wrong with -F'"'? For anything fancy there are plenty of things like the below. eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more. includes csv to tsv: https://github.com/eBay/tsv-utils https://github.com/eBay/tsv-utils HT: https://simonwillison.net/ https://simonwillison.net/
- WFHRenaissance 4y agoAlways had CSV support, it's called `awk --field-separator=","`.
- __jem 4y agoI see you like to live dangerously.
- WFHRenaissance 4y agoAWK is my hammer and I will abuse it to every end
- gshubert17 4y agoI think the article already showed that with the example -F, The problem is commas inside quoted strings, as in the example "Smith, Bob". I believe the field-separator option on awk will break the quoted string at the interior comma.
- asdff 4y agoYou could just replace those commas within the quote with some other string or remove them if you wanted with some awk or sed. Here is a somewhat relevant example from stackoverflow: awk -F'"' -v OFS='' '{ for (i=2; i<=NF; i+=2) gsub(",", "", $i) } 1' infile 1. https://unix.stackexchange.com/questions/48672/remove-comma-between-the-quotes-only-in-a-comma-delimited-file https://unix.stackexchange.com/questions/48672/remove-comma-...
- WFHRenaissance 4y agoLOL - you guys actually read the articles?
- tyingq 4y agoGnu awk also has a csv extension that comes with gawkextlib. I think it may even be installed on many Linux distros by default.
- parasense 4y agoI alway just use awk to process csv files. awk -F '^"|","|"$|,' '{print $2,$3}' whatever.csv The above works perfectly well, it handles quoted fields, or even just unquoted fields.... This snippet is taken from a presentation I give on AWK and BASH scripting. That's the thing about AWK, it's already does everything. No need to extended it much at all.
- CRConrad 4y agoWhat about fields with "," in them?
- w0de0 4y ago-F ','
- wef 4y agoHere's another library I've been using for several years: http://lorance.freeshell.org/csv/ http://lorance.freeshell.org/csv/