3 ms·
As a tool that touches the big problem indirectly, I'd want to have a good tool for data cleanup ("data wrangling") that felt more WYSIWYG. Programmers have th
by TuringTest 9y ago
As a tool that touches the big problem indirectly, I'd want to have a good tool for data cleanup ("data wrangling") that felt more WYSIWYG.
Programmers have their sed and their grep and their regular expressions APIs in Python, but that doesn't help someone who doesn't know programming languages or how to build a precise expression to catch data, and prefer point-and-click and copy&paste.
I'm aware of OpenRefine, but it's somewhat hard to feed it data and create stable cleanup workflows.
- chubot 9y agoYes I think that is a good use case. It has the benefit of being concrete and useful, yet there is less notion of "time", which is a difficult part of programming. There are a bunch of academic papers about "example-based data cleaning", but I don't know much about them. I think the idea is that you provide examples and the system tries to deduce a data cleaning function. Of course, if you do this for the user, then they might learn fewer programming concepts, but be able to get the job done more quickly. So it's a tradeoff.
- TuringTest 9y agoI like the approach of the (experimental) Lapis editor, which allows you to select several examples and it infers a selection rule written in an English-like language. And it works on plain text, not just tables. You can tweak the rule, and then simultaneously edit all the records selected with it.