Is there a way to directly train transformer models to output embeddings that could help tree based models downstream? For tabular data tree based models seems

Is there a way to directly train transformer models to output embeddings that could help tree based models downstream? For tabular data tree based models seems to be the best but I feel like foundational models could help them in some way