4 ms·

EDIT: I think I see the confusion. This project is about statistical classification, which generally assumes the data is in rows and columns. It is about statis

EDIT: I think I see the confusion. This project is about statistical classification, which generally assumes the data is in rows and columns. It is about statistical classification when data is not in rows and columns. Of course the data can be represented as a set of sequences. The challenge is how to build statistical classifiers for that, which is the scope of this project. > The title seems to imply that T-cell receptor data can't be represented with sets There are two datasets, one containing labelled sequences and the other containing labelled sets of sequences. The problem is that the number of features is irregular, resulting in some rows with more columns than others. Also, the information in each column does not line up. It's like you have patient age represented by a column, but because the number of features is irregular, suddenly that column switches to the patient's weight. > manipulated with relational algebra Can you be more specific? I'm not sure what you mean

Retric 7y ago
/Pedantic That’s a common issue for many datasets. You can map that just fine in a relational DB either using Nulls on some columns or breaking that column data into a separate table. EX: (CustomerID: 50, ColumnID : 6852, Value : “Mike”)
- jostmey 7y ago
  Hmm... I think I see the confusion with my title, which I had to shorten to meet the space requirements. It should read "Statistical Classification When Data Do Not Conform to Rows and Columns: … ". The problem is that statistical classification generally requires the data to be in tidy rows and columns, which is not the case for the T-cell data.
- tomnipotent 7y ago
  Except this isn't how statistical methods on vectors and matrices work. In an ML data set, the value "Mike" may actually be one-hot encoded to one of 500 columns (one column value is 1, everything else is 0) - because you have 500 different names in your dataset, so each VALUE gets its own column. It's a very different problem/solution than NULLs in databases.
  - Retric 7y ago
    I was referring to jacques_chester’s comment. You’re bringing up a related and still well explored topic as this situation is very common. https://link.springer.com/article/10.1007%2Fs00180-013-0468-8 https://link.springer.com/article/10.1007%2Fs00180-013-0468-...
jacques_chester 7y ago
> EDIT: I think I see the confusion. This project is about statistical classification, which generally assumes the data is in rows and columns. If I may come at this from the other direction, do you mean the rows and columns of a matrix? Coming from a relational background, I hear "data", "row" and "column" and I'm thinking of something that is essentially meant to be a set of relations. > Can you be more specific? I'm not sure what you mean If your data is structured in a normal form, you can project it back out in a number of different ways. This is not true of most data structures, which entangle the structure of the world with the pattern of data access. Relational algebra allows a database to distinguish between the model of the world and how data may be arranged and accessed. The flipside of this is that relational algebra is performed over classical sets and, depending on one's views about sets, they can be used to represent anything that can be represented.

Retric 7y ago

/Pedantic That’s a common issue for many datasets. You can map that just fine in a relational DB either using Nulls on some columns or breaking that column data into a separate table. EX: (CustomerID: 50, ColumnID : 6852, Value : “Mike”)

jostmey 7y ago
Hmm... I think I see the confusion with my title, which I had to shorten to meet the space requirements. It should read "Statistical Classification When Data Do Not Conform to Rows and Columns: … ". The problem is that statistical classification generally requires the data to be in tidy rows and columns, which is not the case for the T-cell data.
tomnipotent 7y ago
Except this isn't how statistical methods on vectors and matrices work. In an ML data set, the value "Mike" may actually be one-hot encoded to one of 500 columns (one column value is 1, everything else is 0) - because you have 500 different names in your dataset, so each VALUE gets its own column. It's a very different problem/solution than NULLs in databases.
- Retric 7y ago
  I was referring to jacques_chester’s comment. You’re bringing up a related and still well explored topic as this situation is very common. https://link.springer.com/article/10.1007%2Fs00180-013-0468-8 https://link.springer.com/article/10.1007%2Fs00180-013-0468-...

jacques_chester 7y ago

> EDIT: I think I see the confusion. This project is about statistical classification, which generally assumes the data is in rows and columns. If I may come at this from the other direction, do you mean the rows and columns of a matrix? Coming from a relational background, I hear "data", "row" and "column" and I'm thinking of something that is essentially meant to be a set of relations. > Can you be more specific? I'm not sure what you mean If your data is structured in a normal form, you can project it back out in a number of different ways. This is not true of most data structures, which entangle the structure of the world with the pattern of data access. Relational algebra allows a database to distinguish between the model of the world and how data may be arranged and accessed. The flipside of this is that relational algebra is performed over classical sets and, depending on one's views about sets, they can be used to represent anything that can be represented.