6 ms·

Show HN: Data Painter – A Different Way to Interact with Your Data

179 points by loa_observer 4y ago

Data Painter is RATH's new feature to interact with your data. It helps you to specify complex visual patterns and find explanations and potential causals. It allows you to do on-flight data labeling, cleaning or even create new features does not exited in original dataset. Everything can be done with a brush tool(painter), You can even play with your data with your fingers on mobile. RATH is an open-source alternative to Tableau, but with more automations. Feedbacks and suggestions are appreciated. RATH's Github: https://github.com/Kanaries/Rath https://github.com/Kanaries/Rath RATH's HomePage: https://kanaries.net/ https://kanaries.net/

gbasin 4y ago
Wow I would have killed for this in my algo trading days :) Great looking tool
- Yoric 4y ago
  Children, this is why you to have be very careful when you hire someone for algo trading. Some of these people are killers.
  - gbasin 4y ago
    ngl would've killed for more alpha back then
Hakkin 4y ago
I was really impressed that it seems to be doing everything locally in the browser, but I see once you start trying to do more complex visualizations it POSTs the dataset to the cloud. Still neat.
- loa_observer 4y ago
  I tested 1 million rows * 12 columns. It works fine with pure client's web-workers and indexedDB. For larger datasets(like CSV file > 150MB), especially for the "mega automation" feature, It turns to be slow and sometimes lead crash in browsers. For now, I have some temporally solutions, setting a sample size is the easiest. Another way is put some heavy computation tasks to scaleable OLAP services, like a clickhouse clusters. I will continue to research more ways to make the client handle as much data as possible and fast enough so more people can deploy it by their own with few costs.
albert_e 4y ago
I am hearing about this tool first time and the git repo examples look impressive. Is this free and open-source in the sense that we can host it on our own infra with no license fees? Is there any guide or example for deploying this in a AWS environment with typical data sources like S3 data lake (CSV/Parquet/Delta) and Redshift?
- loa_observer 4y ago
  RATH is now under AGPL license, we are considering move to a less limited license like apache if there are more open-source users and developer in its community. The related docs will be published soon, we are working on connecting RATH to more infra now, you can join the discord to the latest discussions. https://discord.com/invite/Z4ngFWXz2U https://discord.com/invite/Z4ngFWXz2U
  - dec0dedab0de 4y ago
    I think AGPL is probably your best bet if there is a chance a competitor will extend your code and release it as a SAAS. But also, the git repo has LICENSE with the AGPL, as well as LICENSE2 which just has some names followed by all rights reserved. It's a bit confusing, if there is dual licensing or some of the files are licensed differently maybe you should explain that in the LICENSE file, even if it means referencing a separate AGPL file. In any case it looks pretty neat, I think I'll give it a try.
aliqot 4y ago
I think the 'k6s' title is an unintentional misdirection
- pimlottc 4y ago
  I was confused why the title bar says Kanaries but the product name is RATH. As someone coming in via a deep link with no knowledge of this product or company, it was confusing to figure out what was the actual product was called. I also couldn’t easily find an explanation of what RATH stands for.
  - loa_observer 4y ago
    Sorry for lack of information. Kanaries is the github organization's name, with a shorthand k6s. RATH is one of an open source project under Kanaries. As for the meaning of RATH, it actually came from an animation called Sword Art Online. I was a little bit regret to named this repo with RATH and had deleted descriptions about the repo's name, but with no more good ideas for now.....
    - aliqot 4y ago
      Brand cohesion is important. I'm in your target demographic, and arrived knowing that, but I see stuff like (k6s) up top and immediately I'm bouncing because it's confusing and I have a lot of tabs opened I wanted to see.
      - loa_observer 4y ago
        thanks for sharing your thoughts and experience! I am fixing the descriptions.
        mkl 4y ago
        You should probably use a spellchecker too. I saw "Embeded" on the Kanaries homepage and "comming" on https://kanaries.net/products https://kanaries.net/products. Also, when navigating from the Kanaries homepage to https://kanaries.net/products https://kanaries.net/products it's not a proper page load and doesn't reset the scroll position.
untangle 4y ago
Interesting product. I first encountered this "data painter" paradigm in the 80's in a product called DataDesk. That software was created by Cornell professor Paul Velleman. Paul was a student of John Tukey and credits his influence on the product. https://datadescription.com/ https://datadescription.com/
thedudeabides5 4y ago
Terrific, have been looking for something like this since Hadley's work on the grand tourr https://github.com/ggobi/tourr https://github.com/ggobi/tourr https://www.jstatsoft.org/article/view/v040i02 https://www.jstatsoft.org/article/view/v040i02 https://www.researchgate.net/publication/346903245_High-dimensional_data_visualisation_with_the_grand_tour https://www.researchgate.net/publication/346903245_High-dime...
- paulgb 4y ago
  ggobi is very cool. I was always surprised that the ideas it exemplified were so slow to catch on. Glad OP is working on rectifying that.
  - thedudeabides5 4y ago
    Same, but after working with folks on these issues, turns out mostly they just don't have clean, well formatted data to even put into these tools. Still in the stone age etc.
- deleted 4y ago
  [deleted]
samstave 4y ago
I need this! I look forward to testing it out.
dr_kiszonka 4y ago
Nicely done! Tangent: I am looking for a different "data painter," i.e., a simple GUI where I can draw data points and have it generate a table or CSV file with them.
- loa_observer 4y ago
  I thought about "draw to create data" before, but it brings problems when your data contains high dimensions (lots of columns). Painted data only defines limited number of dimensions, for other columns, Some prediction model can be used to infer the values. but with higher dimension in data, the prediction gets less convincing.
  - dr_kiszonka 4y ago
    Great point. For my use case, drawing points on a 2d plane would be good enough. I wasn't considering higher dimensions. Maybe it would be possible to "sculp" 3 or 4d data using a cad-inspired direct manipulation GUI like SculptGL or Meshmixer [0,1, respectively]. 0. https://stephaneginier.com/sculptgl/ https://stephaneginier.com/sculptgl/ 1. https://meshmixer.com/ https://meshmixer.com/
- e12e 4y ago
  I found this... Something like what you have in mind? (not Foss) https://plotdigitizer.com/ https://plotdigitizer.com/ Ed: or maybe this (gpl) - didn't realize this existed... https://markummitchell.github.io/engauge-digitizer/ https://markummitchell.github.io/engauge-digitizer/ Ed2: another along similar lines (pun not intended). Looks like most assume you have a bitmap of a graph, rather than let you just draw datapoints. https://automeris.io/WebPlotDigitizer/ https://automeris.io/WebPlotDigitizer/
  - dr_kiszonka 4y ago
    Thanks for links! I was thinking about having the drawing part integrated into such a tool, but what you shared will work too. Cheers!
boredemployee 4y ago
Dope tool, will try it exhaustively! One thing tho, when I try the causal analysis feature, it shows in japanese to me (github show all in english tho) is there a knob to turn it to english?
- loa_observer 4y ago
  If you are on Kanaries homepage (kanaries.net), the language setting is on the top right corner near login button. The language switch button icon looks like this one: https://www.flaticon.com/free-icon/language_4509733 https://www.flaticon.com/free-icon/language_4509733 If you are on RATH product page (rath.kanaries.net), the language setting is on the left bottom of the page.
  - boredemployee 4y ago
    I did it, the insiders > casual analysis is partial english/chinese here: https://i.imgur.com/5QCXdoG.jpeg https://i.imgur.com/5QCXdoG.jpeg
    - loa_observer 4y ago
      Just fixed this, thanks for feedback.
mighty_donkey 4y ago
Yessss this is really great! I have a bunch of time-series for climate and remote sensing data sources I could use for this

179 points by loa_observer 4y ago

gbasin 4y ago

Wow I would have killed for this in my algo trading days :) Great looking tool

Yoric 4y ago
Children, this is why you to have be very careful when you hire someone for algo trading. Some of these people are killers.
- gbasin 4y ago
  ngl would've killed for more alpha back then

Hakkin 4y ago

I was really impressed that it seems to be doing everything locally in the browser, but I see once you start trying to do more complex visualizations it POSTs the dataset to the cloud. Still neat.

loa_observer 4y ago
I tested 1 million rows * 12 columns. It works fine with pure client's web-workers and indexedDB. For larger datasets(like CSV file > 150MB), especially for the "mega automation" feature, It turns to be slow and sometimes lead crash in browsers. For now, I have some temporally solutions, setting a sample size is the easiest. Another way is put some heavy computation tasks to scaleable OLAP services, like a clickhouse clusters. I will continue to research more ways to make the client handle as much data as possible and fast enough so more people can deploy it by their own with few costs.

albert_e 4y ago

I am hearing about this tool first time and the git repo examples look impressive. Is this free and open-source in the sense that we can host it on our own infra with no license fees? Is there any guide or example for deploying this in a AWS environment with typical data sources like S3 data lake (CSV/Parquet/Delta) and Redshift?

loa_observer 4y ago
RATH is now under AGPL license, we are considering move to a less limited license like apache if there are more open-source users and developer in its community. The related docs will be published soon, we are working on connecting RATH to more infra now, you can join the discord to the latest discussions. https://discord.com/invite/Z4ngFWXz2U https://discord.com/invite/Z4ngFWXz2U
- dec0dedab0de 4y ago
  I think AGPL is probably your best bet if there is a chance a competitor will extend your code and release it as a SAAS. But also, the git repo has LICENSE with the AGPL, as well as LICENSE2 which just has some names followed by all rights reserved. It's a bit confusing, if there is dual licensing or some of the files are licensed differently maybe you should explain that in the LICENSE file, even if it means referencing a separate AGPL file. In any case it looks pretty neat, I think I'll give it a try.

aliqot 4y ago

I think the 'k6s' title is an unintentional misdirection

pimlottc 4y ago
I was confused why the title bar says Kanaries but the product name is RATH. As someone coming in via a deep link with no knowledge of this product or company, it was confusing to figure out what was the actual product was called. I also couldn’t easily find an explanation of what RATH stands for.
- loa_observer 4y ago
  Sorry for lack of information. Kanaries is the github organization's name, with a shorthand k6s. RATH is one of an open source project under Kanaries. As for the meaning of RATH, it actually came from an animation called Sword Art Online. I was a little bit regret to named this repo with RATH and had deleted descriptions about the repo's name, but with no more good ideas for now.....
  - aliqot 4y ago
    Brand cohesion is important. I'm in your target demographic, and arrived knowing that, but I see stuff like (k6s) up top and immediately I'm bouncing because it's confusing and I have a lot of tabs opened I wanted to see.
    - loa_observer 4y ago
      thanks for sharing your thoughts and experience! I am fixing the descriptions.
      - mkl 4y ago
        You should probably use a spellchecker too. I saw "Embeded" on the Kanaries homepage and "comming" on https://kanaries.net/products https://kanaries.net/products. Also, when navigating from the Kanaries homepage to https://kanaries.net/products https://kanaries.net/products it's not a proper page load and doesn't reset the scroll position.

untangle 4y ago

Interesting product. I first encountered this "data painter" paradigm in the 80's in a product called DataDesk. That software was created by Cornell professor Paul Velleman. Paul was a student of John Tukey and credits his influence on the product. https://datadescription.com/ https://datadescription.com/

thedudeabides5 4y ago

Terrific, have been looking for something like this since Hadley's work on the grand tourr https://github.com/ggobi/tourr https://github.com/ggobi/tourr https://www.jstatsoft.org/article/view/v040i02 https://www.jstatsoft.org/article/view/v040i02 https://www.researchgate.net/publication/346903245_High-dimensional_data_visualisation_with_the_grand_tour https://www.researchgate.net/publication/346903245_High-dime...

paulgb 4y ago
ggobi is very cool. I was always surprised that the ideas it exemplified were so slow to catch on. Glad OP is working on rectifying that.
- thedudeabides5 4y ago
  Same, but after working with folks on these issues, turns out mostly they just don't have clean, well formatted data to even put into these tools. Still in the stone age etc.
deleted 4y ago
[deleted]

samstave 4y ago

I need this! I look forward to testing it out.

dr_kiszonka 4y ago

Nicely done! Tangent: I am looking for a different "data painter," i.e., a simple GUI where I can draw data points and have it generate a table or CSV file with them.

loa_observer 4y ago
I thought about "draw to create data" before, but it brings problems when your data contains high dimensions (lots of columns). Painted data only defines limited number of dimensions, for other columns, Some prediction model can be used to infer the values. but with higher dimension in data, the prediction gets less convincing.
- dr_kiszonka 4y ago
  Great point. For my use case, drawing points on a 2d plane would be good enough. I wasn't considering higher dimensions. Maybe it would be possible to "sculp" 3 or 4d data using a cad-inspired direct manipulation GUI like SculptGL or Meshmixer [0,1, respectively]. 0. https://stephaneginier.com/sculptgl/ https://stephaneginier.com/sculptgl/ 1. https://meshmixer.com/ https://meshmixer.com/
e12e 4y ago
I found this... Something like what you have in mind? (not Foss) https://plotdigitizer.com/ https://plotdigitizer.com/ Ed: or maybe this (gpl) - didn't realize this existed... https://markummitchell.github.io/engauge-digitizer/ https://markummitchell.github.io/engauge-digitizer/ Ed2: another along similar lines (pun not intended). Looks like most assume you have a bitmap of a graph, rather than let you just draw datapoints. https://automeris.io/WebPlotDigitizer/ https://automeris.io/WebPlotDigitizer/
- dr_kiszonka 4y ago
  Thanks for links! I was thinking about having the drawing part integrated into such a tool, but what you shared will work too. Cheers!

boredemployee 4y ago

Dope tool, will try it exhaustively! One thing tho, when I try the causal analysis feature, it shows in japanese to me (github show all in english tho) is there a knob to turn it to english?

loa_observer 4y ago
If you are on Kanaries homepage (kanaries.net), the language setting is on the top right corner near login button. The language switch button icon looks like this one: https://www.flaticon.com/free-icon/language_4509733 https://www.flaticon.com/free-icon/language_4509733 If you are on RATH product page (rath.kanaries.net), the language setting is on the left bottom of the page.
- boredemployee 4y ago
  I did it, the insiders > casual analysis is partial english/chinese here: https://i.imgur.com/5QCXdoG.jpeg https://i.imgur.com/5QCXdoG.jpeg
  - loa_observer 4y ago
    Just fixed this, thanks for feedback.

mighty_donkey 4y ago

Yessss this is really great! I have a bunch of time-series for climate and remote sensing data sources I could use for this