2 ms·

This used to be Google Refine at some point. It seems it still uses GWT (Google Web Toolkit) which was an amazing idea for its time. Rewrite it in Rust+SQLite+

by wg0 3y ago

This used to be Google Refine at some point. It seems it still uses GWT (Google Web Toolkit) which was an amazing idea for its time. Rewrite it in Rust+SQLite+Tauri+Typescript+Svelte?

jillesvangurp 3y ago
You could rewrite it but it doesn't really solve a problem this thing has. Web assembly of course is an opportunity to bring in a lot of existing data processing frameworks from e.g. the python, julia, r, etc. worlds and run them in a browser. Refine did a lot of it's processing with a Java based server approach. The goal should be to reuse, not to reinvent if you take on a project like this. Chat gpt integration is a no brainer for this stuff these days. It excels at cleaning things up and figuring out unstructured/half structured data. The spread sheet ui is super useful and something that non technical people are much more comfortable dealing with. I've used Google sheets as an interface to business people over the years. Whether it is categorizations, place descriptions, addresses, etc. just put it in a spreadsheet. Instead of building complicated UIs and tools, you just build a csv/tsv importer and let people do their thing in a spreadsheet, export, validate, import. Once you get it in people's heads that the column names are off limits for editing, they kind of get it. The nice thing about this stuff is that it is low tech, easy, and effective. And easy to explain to an intern, product owner, or other person that needs to sit down and do the monkey work. Refine takes this to the next level. You can take any old data in tabular format and cluster it phonetically, minor spelling differences, or by other criteria, bulk edit some rows, and export it. It's also easy to enrich things via some rest API or run some simple scripts. But even just the bulk editing and grouping is super useful. We used it when it was still Google Refine more than 12 years ago to clean up tens of thousands of POIs. Typically we'd be grouping things on e.g. the city name and find that there would be a few spelling variations of things like München, Munchen, Muenchen, Munich, etc. Toss in a few utf-8 encoding issues where the ü got garbled and it's a perfect tool for cleaning that up. Tens of thousands of records is potentially a lot of work but still tiny data. We had a machine learning team that used machine learning as the hammer for the proverbial nail. Google Refine achieved more in 1 afternoon than that team did trying to machine learn their way out of that mess in half a year.
5ersi 3y ago
It seems to be pure JS with jQuery: https://github.com/OpenRefine/OpenRefine/blob/master/main/webapp/package.json https://github.com/OpenRefine/OpenRefine/blob/master/main/we... It is SSR with Velocity templates.

by wg0 3y ago

This used to be Google Refine at some point. It seems it still uses GWT (Google Web Toolkit) which was an amazing idea for its time. Rewrite it in Rust+SQLite+Tauri+Typescript+Svelte?

jillesvangurp 3y ago

You could rewrite it but it doesn't really solve a problem this thing has. Web assembly of course is an opportunity to bring in a lot of existing data processing frameworks from e.g. the python, julia, r, etc. worlds and run them in a browser. Refine did a lot of it's processing with a Java based server approach. The goal should be to reuse, not to reinvent if you take on a project like this. Chat gpt integration is a no brainer for this stuff these days. It excels at cleaning things up and figuring out unstructured/half structured data. The spread sheet ui is super useful and something that non technical people are much more comfortable dealing with. I've used Google sheets as an interface to business people over the years. Whether it is categorizations, place descriptions, addresses, etc. just put it in a spreadsheet. Instead of building complicated UIs and tools, you just build a csv/tsv importer and let people do their thing in a spreadsheet, export, validate, import. Once you get it in people's heads that the column names are off limits for editing, they kind of get it. The nice thing about this stuff is that it is low tech, easy, and effective. And easy to explain to an intern, product owner, or other person that needs to sit down and do the monkey work. Refine takes this to the next level. You can take any old data in tabular format and cluster it phonetically, minor spelling differences, or by other criteria, bulk edit some rows, and export it. It's also easy to enrich things via some rest API or run some simple scripts. But even just the bulk editing and grouping is super useful. We used it when it was still Google Refine more than 12 years ago to clean up tens of thousands of POIs. Typically we'd be grouping things on e.g. the city name and find that there would be a few spelling variations of things like München, Munchen, Muenchen, Munich, etc. Toss in a few utf-8 encoding issues where the ü got garbled and it's a perfect tool for cleaning that up. Tens of thousands of records is potentially a lot of work but still tiny data. We had a machine learning team that used machine learning as the hammer for the proverbial nail. Google Refine achieved more in 1 afternoon than that team did trying to machine learn their way out of that mess in half a year.

5ersi 3y ago

It seems to be pure JS with jQuery: https://github.com/OpenRefine/OpenRefine/blob/master/main/webapp/package.json https://github.com/OpenRefine/OpenRefine/blob/master/main/we... It is SSR with Velocity templates.