5 ms·

I've been looking for an AI/GPT/deep learning tool that would help me perform some sanitation and normalization of a large data set that's quite personal to me-

I've been looking for an AI/GPT/deep learning tool that would help me perform some sanitation and normalization of a large data set that's quite personal to me- my last.fm data, time-stamped logs of (nearly) every song I've listened to for almost twenty years now. The data has all kinds of issues- for example, yesterday I realized that I had two sets of logs for one album. One version of the album used U+2026 (…) and one used three periods (...). There are problems like that, stuff more akin to typos, styling stuff (& vs and), or even garbage-in garbage-out stuff (YouTube Music changing the tags on the same album over time making it look like I actually listened to different albums, or not actually having all of the tags they're supposed to have). I've got .NET code that hits the last.fm api and dumps the info to a LiteDB database, so I can export to CSV pretty easily if this tool would be useful to me, unless anyone has any better directions to point me in. Appreciate any thoughts you folks have.

nerdponx 4y ago
In the case of Unicode at least, the Unicode consortium maintains a database of "confusable" characters and a tool to detect them: https://util.unicode.org/UnicodeJsps/confusables.jsp?a=%E2%80%A6&r=None https://util.unicode.org/UnicodeJsps/confusables.jsp?a=%E2%8... You can download the database for use in your own programs, and there is at least one Python package built around it: https://pypi.org/project/confusables/ https://pypi.org/project/confusables/
- IanCal 4y ago
  If you've got borked encodings around as well, the python package ftfy is wonderful: https://pypi.org/project/ftfy/ https://pypi.org/project/ftfy/ Undoes whatever on earth it is excel does, helps clean up bits of html/etc.
- carterschonwald 4y ago
  Is there a schema document explaining the format of the dataset? Edit Found it in the associated doc It’s a cute approach ! Data File Format Each line in the data file has the following format: Field 1 is the source, Field 2 is the target, and Field 3 is obsolete, always containing the letters “MA” for backwards compatibility. For example: 0441 ; 0063 ; MA # ( с → c ) CYRILLIC SMALL LETTER ES → LATIN SMALL LETTER C # 2CA5 ; 0063 ; MA # ( ⲥ → c ) COPTIC SMALL LETTER SIMA → LATIN SMALL LETTER C # →ϲ→ Everything after the # is a comment and is purely informative. A asterisk after the comment indicates that the character is not an XID character [UAX31]. The comments provide the character names. Implementations that use the confusable data do not have to recursively apply the mappings, because the transforms are idempotent. That is, skeleton(skeleton(X)) = skeleton(X)
0live 4y ago
I would suggest https://github.com/OpenRefine/OpenRefine https://github.com/OpenRefine/OpenRefine to clean your data.
- yosai 4y ago
  @0live Cleaning data is only a module.YoBulk helps you to automate complete CSV import workflow.please read our blog https://www.yobulk.dev/blog/Building%20an%20In-house%20CSV%20Importer https://www.yobulk.dev/blog/Building%20an%20In-house%20CSV%2... to understand the CSV workflow problem.Happy to answer you queries.
  - aarondia 4y ago
    From the blog: > In a typical scenario, the customer success team who is in charge of this activity has to work back and forth with the customer. The customer has to resolve manual (unintended) errors. + 100. This is even the case when analysts are working with datasets that are created by their colleagues at the same company. Since most companies don't have clear standards for column header labelling, etc. getting a new dataset and incorporating it into an existing workflow requires collaboration with others from inside your company.
    - yosai 4y ago
      Thanks for resonating the problem statement.Yes internal teams also face the issue.You Reply on the other team to clean the data.YoBulk is automating this workflow where both data donor and receiver solve the data errors in a much collaborative way.
- anoonmoose 4y ago
  Love this suggestion, excited to check it out!
yosai 4y ago
@anoonmoose we have an internal pipeline which streams the monogoDB data to any CSV or any webhook url path.it's a export pipeline which streams the processed data to CSVs.We will expose an API in coming days which will fit your usecase.
breck 4y ago
It's still early, but TreeBase might be worth a look. (https://jtree.treenotation.org/treeBase/index.html https://jtree.treenotation.org/treeBase/index.html) It's the public domain software that powers PLDB.com and CancerDB.com. You store your data in Tree Notation in plain text files, and use the Grammar Language (a Tree Language) for schemas, which also enforces correctness. You use Git for version control. You then can query the data using TQL (also a Tree Language). You can display your data using Scroll (also a Tree Language) So your data, your query language, your schemas, your display language are all in the same simple plain-text notation: Tree Notation. Of course, there's also a lot of Javascript glue. Very little documentation at the moment, and it's brand new, but it simple was not possible before the M1s, which came out in December 2020, and the growth rate is very good. It's all signal, no noise, so it's a timeless solution, and you won't regret putting your data in there.
a_subsystem 4y ago
We're using PowerBI for this kind of thing. It's certainly not open source, but you put in wonky tables, give it a couple/few examples of how you would like it to be, and it uses AI to spit out clean tables for export. I'm not a fan of proprietary working files, but if that ever becomes a problem, at least we've still got the data.
aarondia 4y ago
Its not an AI based approach, but it is a step up from writing code by hand -- you could try using open source Mito -> https://www.trymito.io https://www.trymito.io -> full disclosure I built it -> to do some of this messy data wrangling. Mito lets you view and manipulate your data in a spreadsheet in Jupyter and it generates the equivalent Python code for each edit. For things like identifying that the data uses '&' and 'and', viewing your data in a spreadsheet is >> just writing code. Once you generate the code, you could copy it into your pipeline so that you pull the code from the last.fim api, preprocess it with the Python code that Mito generated, and then dump it into the LiteDB.

nerdponx 4y ago

In the case of Unicode at least, the Unicode consortium maintains a database of "confusable" characters and a tool to detect them: https://util.unicode.org/UnicodeJsps/confusables.jsp?a=%E2%80%A6&r=None https://util.unicode.org/UnicodeJsps/confusables.jsp?a=%E2%8... You can download the database for use in your own programs, and there is at least one Python package built around it: https://pypi.org/project/confusables/ https://pypi.org/project/confusables/

IanCal 4y ago
If you've got borked encodings around as well, the python package ftfy is wonderful: https://pypi.org/project/ftfy/ https://pypi.org/project/ftfy/ Undoes whatever on earth it is excel does, helps clean up bits of html/etc.
carterschonwald 4y ago
Is there a schema document explaining the format of the dataset? Edit Found it in the associated doc It’s a cute approach ! Data File Format Each line in the data file has the following format: Field 1 is the source, Field 2 is the target, and Field 3 is obsolete, always containing the letters “MA” for backwards compatibility. For example: 0441 ; 0063 ; MA # ( с → c ) CYRILLIC SMALL LETTER ES → LATIN SMALL LETTER C # 2CA5 ; 0063 ; MA # ( ⲥ → c ) COPTIC SMALL LETTER SIMA → LATIN SMALL LETTER C # →ϲ→ Everything after the # is a comment and is purely informative. A asterisk after the comment indicates that the character is not an XID character [UAX31]. The comments provide the character names. Implementations that use the confusable data do not have to recursively apply the mappings, because the transforms are idempotent. That is, skeleton(skeleton(X)) = skeleton(X)

0live 4y ago

I would suggest https://github.com/OpenRefine/OpenRefine https://github.com/OpenRefine/OpenRefine to clean your data.

yosai 4y ago
@0live Cleaning data is only a module.YoBulk helps you to automate complete CSV import workflow.please read our blog https://www.yobulk.dev/blog/Building%20an%20In-house%20CSV%20Importer https://www.yobulk.dev/blog/Building%20an%20In-house%20CSV%2... to understand the CSV workflow problem.Happy to answer you queries.
- aarondia 4y ago
  From the blog: > In a typical scenario, the customer success team who is in charge of this activity has to work back and forth with the customer. The customer has to resolve manual (unintended) errors. + 100. This is even the case when analysts are working with datasets that are created by their colleagues at the same company. Since most companies don't have clear standards for column header labelling, etc. getting a new dataset and incorporating it into an existing workflow requires collaboration with others from inside your company.
  - yosai 4y ago
    Thanks for resonating the problem statement.Yes internal teams also face the issue.You Reply on the other team to clean the data.YoBulk is automating this workflow where both data donor and receiver solve the data errors in a much collaborative way.
anoonmoose 4y ago
Love this suggestion, excited to check it out!

yosai 4y ago

@anoonmoose we have an internal pipeline which streams the monogoDB data to any CSV or any webhook url path.it's a export pipeline which streams the processed data to CSVs.We will expose an API in coming days which will fit your usecase.

breck 4y ago

It's still early, but TreeBase might be worth a look. (https://jtree.treenotation.org/treeBase/index.html https://jtree.treenotation.org/treeBase/index.html) It's the public domain software that powers PLDB.com and CancerDB.com. You store your data in Tree Notation in plain text files, and use the Grammar Language (a Tree Language) for schemas, which also enforces correctness. You use Git for version control. You then can query the data using TQL (also a Tree Language). You can display your data using Scroll (also a Tree Language) So your data, your query language, your schemas, your display language are all in the same simple plain-text notation: Tree Notation. Of course, there's also a lot of Javascript glue. Very little documentation at the moment, and it's brand new, but it simple was not possible before the M1s, which came out in December 2020, and the growth rate is very good. It's all signal, no noise, so it's a timeless solution, and you won't regret putting your data in there.

a_subsystem 4y ago

We're using PowerBI for this kind of thing. It's certainly not open source, but you put in wonky tables, give it a couple/few examples of how you would like it to be, and it uses AI to spit out clean tables for export. I'm not a fan of proprietary working files, but if that ever becomes a problem, at least we've still got the data.

aarondia 4y ago

Its not an AI based approach, but it is a step up from writing code by hand -- you could try using open source Mito -> https://www.trymito.io https://www.trymito.io -> full disclosure I built it -> to do some of this messy data wrangling. Mito lets you view and manipulate your data in a spreadsheet in Jupyter and it generates the equivalent Python code for each edit. For things like identifying that the data uses '&' and 'and', viewing your data in a spreadsheet is >> just writing code. Once you generate the code, you could copy it into your pipeline so that you pull the code from the last.fim api, preprocess it with the Python code that Mito generated, and then dump it into the LiteDB.