4 ms·
Since this is from 2022, I’m wondering how “tabular foundation models” could change this. The incredible success of DL we see at the moment comes partially from
by doubtfuluser 3y ago
Since this is from 2022, I’m wondering how “tabular foundation models” could change this. The incredible success of DL we see at the moment comes partially from foundation models learning on a lot of “semi-related” data an “understanding” of the behavior. Something similar has been explored in tabular data as well iirc.
So I would be curious to see latest DL results.
On the other hand it is also the case that in most cases where DL based on foundation models is used, specific heavily tuned models outperform the generalistic models. And for tabular data there is a lot of experience how to make it great with tree based models.
- dweinus 3y agoWhat would these tabular foundation models look like? LLMs work as foundation models because the input is fixed in format (a sequence of text). Would the model be for a specific fixed tabular format?
- scottyak 3y agoOne promising approach is to encode each feature key and feature value as embedding vectors, concatenate them into "feature tokens", then feed them into a Transformer (without positional encodings). This takes advantage of column-order invariance. See: https://arxiv.org/abs/2403.01841 https://arxiv.org/abs/2403.01841 (ICLR 2024 spotlight)