4 ms·

How do you plan to implement the transforms? Are you going to expose the subset of programming language or a drop down menu or something else?

joshxyz 4y ago
Right now I let the user modify a transform function that accepts the csv row then returns the table row. Screenshot here: https://ibb.co/09KJX18 https://ibb.co/09KJX18 Currently it is in JavaScript since it is the easiest to emulate in the browser (I just run the code in a Worker instance, then display the results back to the user). Kinda ideal for moving from one source to one table, I'm still figuring out how it should work if the user wants to combine data from multiple sources. What do you think?
- freemint 4y ago
  I find the JavaScript code not very useful for human input, if the intended audience is not restricted to web developers (a code like interface might still be a good idea). If your intended offering is for devs it would neat some more advanced features to be useful. More interesting is the question whether you want to support joins. If you do not then you are restricted to concatenation (either in length or in width). Another question is do you want to allow filtering too? If you want joins i would recommend writing a small query planner because there are some hugely performant optimisations with joins (and filters). If you restrict the possible calculations across loops to simple things like sums, max one could parallelize it between workers with some workm An interesting avenue for small scale monetisation would be to allow paying users to run them "in the cloud" in JavaScript edge VMs (not sure how they are called exactly cloudflare and others have them). One premium feature could be to pick one of multiple locations where to start them between closest to the data sources or closest data consumer based on the expected ratio of input/output. For large local files and poor connections client side calculations could be used automatically.

joshxyz 4y ago

Right now I let the user modify a transform function that accepts the csv row then returns the table row. Screenshot here: https://ibb.co/09KJX18 https://ibb.co/09KJX18 Currently it is in JavaScript since it is the easiest to emulate in the browser (I just run the code in a Worker instance, then display the results back to the user). Kinda ideal for moving from one source to one table, I'm still figuring out how it should work if the user wants to combine data from multiple sources. What do you think?

freemint 4y ago
I find the JavaScript code not very useful for human input, if the intended audience is not restricted to web developers (a code like interface might still be a good idea). If your intended offering is for devs it would neat some more advanced features to be useful. More interesting is the question whether you want to support joins. If you do not then you are restricted to concatenation (either in length or in width). Another question is do you want to allow filtering too? If you want joins i would recommend writing a small query planner because there are some hugely performant optimisations with joins (and filters). If you restrict the possible calculations across loops to simple things like sums, max one could parallelize it between workers with some workm An interesting avenue for small scale monetisation would be to allow paying users to run them "in the cloud" in JavaScript edge VMs (not sure how they are called exactly cloudflare and others have them). One premium feature could be to pick one of multiple locations where to start them between closest to the data sources or closest data consumer based on the expected ratio of input/output. For large local files and poor connections client side calculations could be used automatically.