4 ms·

I'm not totally sure what the question is but currently these are the inputs: delimited, excel, json, feather, parquet, sqlite, postgresql, mysql, googleSheets

I'm not totally sure what the question is but currently these are the inputs: delimited, excel, json, feather, parquet, sqlite, postgresql, mysql, googleSheets Outputs are the same (excluding databases currently). This is probably the most time consuming piece of the project if I was to expand this set dramatically given the quantity of possible connections, although here is where open source could help. It's hard to know how technical or not technical to take this. e.g. could there be a python input e.g. arbitrary python script to pull in data? That would allow for basically any input or output. But for non technical users thats a harder sell... open to ideas.

0xbadcafebee 4y ago
Target data scientists and you'll have a very large segment of users. They aren't programmers but they deal with data a lot.
- mousetree 4y ago
  I think that is OP's intent. Data Scientists, Data Engineers, Data Analysts, Data X
  - gthompson1 4y ago
    Yeh I still appreciate the callout / opinion though. It's not obvious to me who is best to target. As a full stack engineer with analytics experience one of the main uses of the tool is scheduling, running pipelines remotely vs the UI for data munging which I can write pretty easily. But for non technical users the data munging piece might be really helpful (e.g. no code). Its a little hard to serve both a marketer and a developer given the needs are so different. But then on the other hand I assume a marketer is more likely to pay for the tool (and thus keep it going) vs a dev who might be a harder sell. I built this tool with the hope of being able to serve both which is where maybe I bit off more than I could chew.
    - 0xbadcafebee 4y ago
      I'd say you should still work toward whatever your own interest with the tool is. If it seems like too much work, make a list of all the functionality you want, figure out the easiest way to implement each, and work on the least-effort/highest-reward things first. And do whatever feels fun!
programmarchy 4y ago
Would you be able to leverage singer taps / targets? https://www.singer.io https://www.singer.io
- gthompson1 4y ago
  Oh this is very interesting, I had never heard of this. Yeh I could potentially use this. I do think using a third party library or provider makes sense to massively expand the input / output options. Seems like there is a standard schema for the different tap configs that I could pull and wrap in UI forms in a generalized fashion. I don't know if you think these are fair comment about the state of the project? - https://www.youtube.com/watch?v=TBrSOPNEg-g&ab_channel=Restack https://www.youtube.com/watch?v=TBrSOPNEg-g&ab_channel=Resta...
  - programmarchy 4y ago
    I actually use Stitch so I don’t have to host orchestration, which is how I came to know about Singer in the first place. It’s true that Stitch got acquired by Talend, but Singer seems independent enough. I don’t understand why Stitch would “abandon” Singer. They’re a sponsor, and seems like they have an incentive to keep that ball rolling. Their bread and butter seems to be hosting orchestration, so more taps means more customers for them. There’s other stakeholders in Singer, too, like Meltano, and all the self-hosters. The GitHub activity seems robust to me, but I don’t really have any in depth knowledge about the Singer community. Saying it’s “dying” doesn’t seem accurate, though.

0xbadcafebee 4y ago

Target data scientists and you'll have a very large segment of users. They aren't programmers but they deal with data a lot.

mousetree 4y ago
I think that is OP's intent. Data Scientists, Data Engineers, Data Analysts, Data X
- gthompson1 4y ago
  Yeh I still appreciate the callout / opinion though. It's not obvious to me who is best to target. As a full stack engineer with analytics experience one of the main uses of the tool is scheduling, running pipelines remotely vs the UI for data munging which I can write pretty easily. But for non technical users the data munging piece might be really helpful (e.g. no code). Its a little hard to serve both a marketer and a developer given the needs are so different. But then on the other hand I assume a marketer is more likely to pay for the tool (and thus keep it going) vs a dev who might be a harder sell. I built this tool with the hope of being able to serve both which is where maybe I bit off more than I could chew.
  - 0xbadcafebee 4y ago
    I'd say you should still work toward whatever your own interest with the tool is. If it seems like too much work, make a list of all the functionality you want, figure out the easiest way to implement each, and work on the least-effort/highest-reward things first. And do whatever feels fun!

programmarchy 4y ago

Would you be able to leverage singer taps / targets? https://www.singer.io https://www.singer.io

gthompson1 4y ago
Oh this is very interesting, I had never heard of this. Yeh I could potentially use this. I do think using a third party library or provider makes sense to massively expand the input / output options. Seems like there is a standard schema for the different tap configs that I could pull and wrap in UI forms in a generalized fashion. I don't know if you think these are fair comment about the state of the project? - https://www.youtube.com/watch?v=TBrSOPNEg-g&ab_channel=Restack https://www.youtube.com/watch?v=TBrSOPNEg-g&ab_channel=Resta...
- programmarchy 4y ago
  I actually use Stitch so I don’t have to host orchestration, which is how I came to know about Singer in the first place. It’s true that Stitch got acquired by Talend, but Singer seems independent enough. I don’t understand why Stitch would “abandon” Singer. They’re a sponsor, and seems like they have an incentive to keep that ball rolling. Their bread and butter seems to be hosting orchestration, so more taps means more customers for them. There’s other stakeholders in Singer, too, like Meltano, and all the self-hosters. The GitHub activity seems robust to me, but I don’t really have any in depth knowledge about the Singer community. Saying it’s “dying” doesn’t seem accurate, though.