3 ms·
Having worked both in libraries and with large data sets I don't think a classification system like Dewey Decimal would work for data sets. For example, if yo
by rapjr9 3y ago
Having worked both in libraries and with large data sets I don't think a classification system like Dewey Decimal would work for data sets. For example, if you had a large data set of locations for people on a college campus, that data set could be used for sociological research (who meets up with whom?), or epidemiology (to see who potentially spread a virus), or building maintenance (which buildings have the most people using them?) So what do you classify the data set under? Maybe you could file it under location data (though it is also student+professor+manager data), but what if the basic data is WiFi AP associations? Then the data might also be of interest to wireless network managers and could be classified as wireless management data. So any classification system probably needs a tag system that can apply multiple tags to one data set. The main point of classification is being able to find what you need, so pigeon holing a data set into one category is not a good idea because data is much more versatile than a single book (although there are problems with Dewey Decimal and books in this regard also). There is also the problem of future categories, catagories that don't even exist now but might retroactively apply to current data sets. So an old data set that might have been tagged with "location" might in the future also need to be tagged with "privacy". Being able to add tags in the future is important. So a data set is not like a book on a shelf, it's more like a web page, and there should be multiple ways to link to it. Data sets are not physical things, and are not related to a single topic (though they may have been collected for only one purpose initially), so don't use a cataloging system for physical objects to catalog data sets. That said, the range of data sets is infinite. So understanding your own needs my help limit the number of tags/categories and understanding how your needs may change in the future would help. Machine Learning data sets are often repurposed. Business aims may expand. At the extreme you may want to do some research into the problem of ontologies. Specifically, how to build an ontology of your current and future intentions for these data sets. Look for something flexible because your needs are likely to change, so you may need to transition the "catalog" to a new form in the future. That seems very complicated and difficult, so reduce the problem as much as you can by imposing limits. What is the minimum range of categories that defines your interests?
You might take a completely different approach also, instead of categorizing data sets, just feed them (and all their metadata) to a machine learning algorithm that can search and summarize them for you. That puts the categorization into the search query+process, not associating it with the data. By its nature raw data is rather category-less, it can later be used for anything. Notes on butterfly migrations from the 1900's later becomes data on global warming.
- Mithriil 3y agoI appreciate your insights. Indeed, I can't see how a single template classification can work for every possible datasets. Best approaches I have in mind is either determining a set of mandatory and optional metadata shared by every dataset and, as you mentioned, defining an ontology of use cases, sources or characteristics to label datasets.