4 ms·

In a recent job, I was a software engineer who worked with the data science people in an "analytics" group. A big PITA for us was the data we'd get from variou

In a recent job, I was a software engineer who worked with the data science people in an "analytics" group. A big PITA for us was the data we'd get from various sources (vendors/clients/whomever). Oh, there's a rando string in a "number" field in row 9,100,500, huh, that sucks. You can't always build the validation into your ETL pipeline for various reasons. Or it fails the whole file (b/c the data science folks built it that way really aren't that experienced at handling exceptions in the file loads, etc) Anyway that rambling is trying to lead up to me saying that there's a chasm between your supposed target market (marketers, BSAs, scientists) and the engineers/devs etc that could probably use some better tools for CSV/JSON munging. Excel is terrible at this job. What I often did was load the csv into sqlite to and then ran queries on the data to figure out how many records were horked. Or if it was a process we were testing, to verify that we had the right data or a good approximation. That works for csv, but with JSON, it's even harder. Anyway, I'm just not sure that your target market knows to look for a tool like this or would really know what to do with it. On the other hand, an engineer might look at this tool as too down level. - No cli options - No api? I think there's a market for tools to help "data plumbers", software devs, engineers, support personal, etc. There's all kinds of application consultants/support etc that are moving data between systems for migrations, upgrades, custom exports, etc.

rusticpenn 7y ago
I can tell you from experience that this is primarily a Business Transformation problem more than a technical one. A tool would of course be helpful but solves only 5% of the real problem.
- hermitcrab 7y ago
  And what are the 95% of 'real problem'? Are you talking about soft/people issues? Is it even possible to address these things with a tool?
hermitcrab 7y ago
Real data is almost always dirty. But the filtering and replace transforms could help with that. >I'm just not sure that your target market knows to look for a tool like this Perhaps not. I am getting traffic for things such as "merge csv files" or "join excel sheets". But they mostly bounce without downloading the free trial. Even when I send them to a dedicated landing page. Most people really hate change. >That works for csv, but with JSON, it's even harder. Currently I am only exporting JSON (not importing). Importing JSON would involve potentially flattening a tree into a table, which is harder. >- No cli options - No api? Command line is high on the wish list. Given that I am aiming mostly at non-programmers, I'm not sure an API would be useful. Note that there is a 'Javascript' transform, which adds a lot of flexibility if you are able to do some scripting.

rusticpenn 7y ago

I can tell you from experience that this is primarily a Business Transformation problem more than a technical one. A tool would of course be helpful but solves only 5% of the real problem.

hermitcrab 7y ago
And what are the 95% of 'real problem'? Are you talking about soft/people issues? Is it even possible to address these things with a tool?

hermitcrab 7y ago

Real data is almost always dirty. But the filtering and replace transforms could help with that. >I'm just not sure that your target market knows to look for a tool like this Perhaps not. I am getting traffic for things such as "merge csv files" or "join excel sheets". But they mostly bounce without downloading the free trial. Even when I send them to a dedicated landing page. Most people really hate change. >That works for csv, but with JSON, it's even harder. Currently I am only exporting JSON (not importing). Importing JSON would involve potentially flattening a tree into a table, which is harder. >- No cli options - No api? Command line is high on the wish list. Given that I am aiming mostly at non-programmers, I'm not sure an API would be useful. Note that there is a 'Javascript' transform, which adds a lot of flexibility if you are able to do some scripting.