4 ms·

I haven't tried the thing yet, but if it made my life easier, I would most certainly pay for and use it. Or get my customers to do so. Data pipelining, cleanin

I haven't tried the thing yet, but if it made my life easier, I would most certainly pay for and use it. Or get my customers to do so. Data pipelining, cleaning and feature construction is the most time consuming part of data science. Its almost always a struggle, and the process usually produces fragile and ephemeral code that will need to be rewritten to put into production. If you could provide a labview-like GUI thing to remove some of this drudgery, assuming you could hook it up to database and csv back ends, or if there were a target which could do this, and the result were robust and could be deployed, it would make me much more productive than fiddling around with pandas or R data tables. Maybe it isn't what you built, but I've said many times this is the most useful data science product; the one that is 10x easier than writing your own every damn time. Fancy woo-woo classifiers with alleged superpowers don't even begin to compare to tooling like this.

hermitcrab 7y ago
Interesting. I'd love to know close or far it is from what you are looking for. Currently it can input XLS/XLSX (on Windows) and delimited text (e.g. CSV) and can output delimited text. I am looking to support other inputs and outputs, e.g. JSON / ODBC / SQLite, depending on feedback.
- scottlocklin 7y ago
  well, I don't work on osx or windows, like, ever. ;-) csv is the lingua franca for prototyping of course. For deploying (which could make your code more sticky in an enterprise setup), you need to hook up to real databases. I'd suggest finding a local senior data scientist or consulting group and use them to drive your product feature development. I'm not sure who you had in mind for your end user, but I do think data science types (the ones who get paid; not smurfs who want a DS job some day) are a viable market for something like this.
  - hermitcrab 7y ago
    >I'm not sure who you had in mind for your end user People who have data they need to transform, but probably aren't programmers or data science professionals. I would obviously be delighted if data science professionals want to buy it. But I assume they have lots of powerful tools already at their disposal. But I am still learning about the market, so I could be completely wrong about where the opportunity is.
    - scottlocklin 7y ago
      >People who have data they need to transform, but probably aren't programmers or data science professionals. I can't imagine who that would be. Excel power users maybe?
      - hermitcrab 7y ago
        Marketing people for a start. They often have vast reams of data (email lists, postal lists, analytics data) that they need to clean, reformat and analyze.

hermitcrab 7y ago

Interesting. I'd love to know close or far it is from what you are looking for. Currently it can input XLS/XLSX (on Windows) and delimited text (e.g. CSV) and can output delimited text. I am looking to support other inputs and outputs, e.g. JSON / ODBC / SQLite, depending on feedback.

scottlocklin 7y ago
well, I don't work on osx or windows, like, ever. ;-) csv is the lingua franca for prototyping of course. For deploying (which could make your code more sticky in an enterprise setup), you need to hook up to real databases. I'd suggest finding a local senior data scientist or consulting group and use them to drive your product feature development. I'm not sure who you had in mind for your end user, but I do think data science types (the ones who get paid; not smurfs who want a DS job some day) are a viable market for something like this.
- hermitcrab 7y ago
  >I'm not sure who you had in mind for your end user People who have data they need to transform, but probably aren't programmers or data science professionals. I would obviously be delighted if data science professionals want to buy it. But I assume they have lots of powerful tools already at their disposal. But I am still learning about the market, so I could be completely wrong about where the opportunity is.
  - scottlocklin 7y ago
    >People who have data they need to transform, but probably aren't programmers or data science professionals. I can't imagine who that would be. Excel power users maybe?
    - hermitcrab 7y ago
      Marketing people for a start. They often have vast reams of data (email lists, postal lists, analytics data) that they need to clean, reformat and analyze.