3 ms·
What would these tabular foundation models look like? LLMs work as foundation models because the input is fixed in format (a sequence of text). Would the model
by dweinus 3y ago
What would these tabular foundation models look like? LLMs work as foundation models because the input is fixed in format (a sequence of text). Would the model be for a specific fixed tabular format?
- scottyak 3y agoOne promising approach is to encode each feature key and feature value as embedding vectors, concatenate them into "feature tokens", then feed them into a Transformer (without positional encodings). This takes advantage of column-order invariance. See: https://arxiv.org/abs/2403.01841 https://arxiv.org/abs/2403.01841 (ICLR 2024 spotlight)