mattewong

26 ms·

91.
by mattewong 4y ago
It doesn't detect column types automatically-- they are imported as text. You can, however, use math functions on them and sqlite3 will dynamically convert where possible (e.g. "select number_1_stored_as_text + 1 from mytable&quot
92.
by mattewong 4y ago
Over all these CLIs, I prefer zsv ( https://github.com/liquidaty/zsv )-- then again, I wrote it so my preference should come as no surprise. On my Mac was 340% faster than the OP command, and does a lot more than just
93.
by mattewong 4y ago
Nice article. I'm left wondering: - Are there any benchmarks available? - Is any full code example available (as opposed to a snippet)? - Does it handle real-world edge cases (unix vs dos line ends, non-"normal" quoting etc)
94.
Bitcoin falls below 20k and Ethereum sinks past 1k(techcrunch.com)
3 points by mattewong 4y ago|2 comments
95.
by mattewong 4y ago
The rate itself is nothing new; 5.78 is still lower than the average 30-yr fixed rate between 1972 and today (at least per https://www.bankrate.com/mortgages/historical-mortgage-rates... )-- and possibly since the begi
96.
by mattewong 4y ago
Glad to see this. I'm in the midst of evaluating "modern cloud" ETL approaches and it seems that Airflow comes up everywhere I look, yet Step Functions seems better-- glad to see others with that perspective too. Major advant
97.
by mattewong 4y ago
if you can combine this with cloud services (for example, each node is an S3 bucket or a lambda function) and now we're cookin with gas
98.
Will data model/schema/structure variance widen or narrow over time?
1 points by mattewong 4y ago|0 comments
99.
by mattewong 4y ago
Sorry for the delay, didn't see your reply so thought the thread was dead. Whether you want to add a new column to the input data, or add a new column to the report, I'm not seeing any need for backfilling either way.
100.
Show HN: CSV to/fm Sqlite3 via JSON(github.com)
1 points by mattewong 4y ago|0 comments
101.
by mattewong 4y ago
There is a better way to do this. An example of output that fits this paradigm you describe-- but to a much further degree-- would be the dozens of tables shown in the securities offering described at https://www.sec.gov/Arc
102.
by mattewong 4y ago
are you using the same categories AMZN provides or using another approach to categorize? You mention open sourcing the code, I would have checked there for an answer but couldn't find the link to it
103.
by mattewong 4y ago
BTW does c2tsv handle multibyte UTF8, \r\n vs \n, regular escapes e.g. embedded dbl-quote/nl/lf/comma, as well as CSV garbage that doesn't exist in theory but is abundant in real world (e.g. dbl-quote inside a cell that
104.
by mattewong 4y ago
thanks for mentioning. will try out. did you use the default build settings for zsv (i.e. just plain old "make install")? also do you have copy or location for the dataset you used to test on? also what hardware / os if I may
105.
by mattewong 4y ago
This is not an outlier. `mlr` is quite slow, literally off-the-charts slow for our purposes when we benchmarked vs xsv and zsv (see https://github.com/liquidaty/zsv . disclaimer: I'm one of its authors)
106.
by mattewong 4y ago
Not a problem at all. Nice work with goawk!
107.
by mattewong 4y ago
FYI I ran on the worldcities data at https://github.com/petewarden/dstkdata (credit to xsv for choosing that dataset) against https://github.com/BurntSushi/xsv and https://github.com&#x
108.
by mattewong 4y ago
thanks for this. am looking at the benchmarks. how do I get huge.csv? Don't see how to fetch or generate
109.
by mattewong 4y ago
Looks cool. But why use this instead of jq? You mention it's "like jq on steroids"; could you provide an example of a real-world use case where it solves a problem that couldn't be solved with jq?
110.
by mattewong 5y ago
Assuming you want to do this on a single machine with finite memory, a feasible approach would be: 1. Split into multiple shard using e.g. `split`, where each piece is small enough to fit comfortably into memory 2. Sort each shard. You coul
111.
by mattewong 5y ago
Are you asking how to do this with a single machine, or how to do this assuming you have access to cloud resources?
112.
by mattewong 5y ago
You are in luck. My company is launching a free solution to exactly this problem. You will be able to normalize tables of data (from csv, txt, xlsx and more) that have the same subject matter but different formats. Variations in names, scal
113.
by mattewong 5y ago
jq
114.
by mattewong 5y ago
I never really understood why anyone thinks this is a good idea. Popular, maybe. Smart? Not so sure. Instead of just modifying your parser, you change your entire input format to be incompatible with what most of the rest of the world uses
115.
by mattewong 5y ago
I spent 10 seconds on it. looks pretty cool. But, some questions before I'd try to use it instead of jquery datatables or handsontable: - why did you build it and what alternatives did you try first? What unmet need does it fulfill? ho
116.
by mattewong 5y ago
My 2 cents: This is a lot of work just to see it in action. I have to click, fill out a form to save a scenario, then start entering data. I can understand why it was built that way-- it makes sense from an engineering perspective to do so-
117.
Show HN: Generate SQL table definition from CSV (in-browser and WASM)(zsvhub.com)
3 points by mattewong 5y ago|0 comments
118.
by mattewong 5y ago
Not being familiar with NSGA, I would find it very interesting if your link provided some context/example as to why this intriguing problem matters in the real world. I'm sure it does, I just can't fathom it and googling &quo
119.
by mattewong 5y ago
For relatively small data sets in CSV format, you can use the in-browser utility-- which is quite convenient (no installation, no security concerns) at https://zsvhub.com/playground/ . A subset of the features there are
120.
by mattewong 5y ago
Will take a look. Does it come with a sample main() that can either spit out some specified columns, or simply count rows? and/or plug into https://bitbucket.org/ewanhiggs/csv-game . Feel free to post this as an is

More ›