4 ms·
In general I like to send structured data (see the input format here: https://github.com/minimaxir/mtg-embeddings https://github.com/minimaxir/mtg-embeddings),
by minimaxir 2y ago
In general I like to send structured data (see the input format here: https://github.com/minimaxir/mtg-embeddings https://github.com/minimaxir/mtg-embeddings), but the ModernBERT base for the embedding model used here specifically has better benefits implicitly for structured data compared to previous models. That's worth another blog post explaining why.
- notpublic 2y agoplease do explain why
- minimaxir 2y agotl;dr the base ModernBERT was trained with code in mind unlike most encoder-only models (therefore assuming it was also trained on JSON/YAML objects) and also includes a custom tokenizer to support that, which is why I mention that indentation is important since different levels of indentation have different single tokens. This is mostly theoetical and does require a deeper dive to confirm.