3 ms·
TabFM: A zero-shot foundation model for tabular data
- kingjimmy 3mo agointeresting to see this from Google after the SAP acquisition of Prior Labs.
- hodgehog11 3mo agoOn the one hand, this is impressive. TabPFN was already state of the art and is seriously shaking up Bayesian prediction for tabular data (which is almost everything). On the other hand, perhaps it is just me, but I do not feel that this is an acceptable form of benchmark reporting in this domain. TabArena actually has multiple metrics, since ELO does not properly quantify the degree of improvement. The fact that these are not displayed here should give pause. Also the results section in the GitHub is a dumpster fire.
- Eridrus 3mo agoGitHub Repo: Please see the results folder Results folder: Here's some undocumented parquet files Definitely feels like they're hiding the ball lol. If they had good benchmarks they'd talk about them. Not comparing to tuned xgboost is also a warning sign.
- nok22kon 3mo agowouldn't xgboost be covered under autogluon? not perfect, but not missing either
- Eridrus 3mo agoHonestly, I don't really know AutoGluon, if this does xgboost tuning that's good. I do still think ELO scores are still a way to obscure results though. For all we know this is like 0.1% better than a "normal" approach on like 70% of tasks and a tire fire on others.
- nok22kon 3mo agoAutoGluon does their own benchmarking, the default is Elo, but you can switch to other metrics: https://huggingface.co/spaces/TabArena/leaderboard https://huggingface.co/spaces/TabArena/leaderboard xgboost is listed, they say "tuned" but who knows what that means. and its below CatBoost and LightGBM as an aggregator of models (trees, neural networks, clustering, ...) AutoGluon doesn't really have a dog in this fight
- Eridrus 3mo agoThanks for the pointer, though I can't seem to find TabFM and AutoGluon at that link?
- actusual 3mo ago150,000 rows of data, where will I store it all?!
- nojito 3mo agoThe biggest misconception that people have when modeling using tabular data is that more data = better model.
- mvcalder 3mo agoThere's a story sharing the front page right now about scaling laws, so it's not an unreasonable assumption. https://news.ycombinator.com/item?id=48689744 https://news.ycombinator.com/item?id=48689744
- estetlinus 3mo agoBig Data is terrifying and only solvable with opaque deep learning
- piokoch 3mo agoYou don't need to. Just sample 1% of data, let the model do the feature engineering, ask model model to replicate everything in Pandas (or R, or whatever else), so you can run it for a full dataset.
- woadwarrior01 3mo agoIn hindsight, the Prior Labs exit to SAP couldn't have been timed better.
- noahho 3mo agoThey don't show the strongest TabPFN variants in the plot unfortunately, i.e. thinking and ensembled. Not really apples to apples.
- deleted 3mo ago[deleted]