5 ms·

I always love these posts. Great job to the interns. Building a tokenizer with a handcrafted automaton instead of a regex sounds like a cool project. And I foun

I always love these posts. Great job to the interns. Building a tokenizer with a handcrafted automaton instead of a regex sounds like a cool project. And I found it interesting that Jane Street has a frontend for interacting with different AI services and LLMs. I'm curious how that's used. But to be honest, what surprised me most about that was that Jane Street cares about counting the token usage in its API calls. I can see why they would care if they were interacting with AI models at very large scales, but are they? They have billions of dollars, so it struck me as odd to see such concern for efficiency in what must be a very small cost center. I wonder if the motivation is more about providing feedback to the users of the frontend tool, as they write their prompts, so they can gain an intuition for what makes an efficiently tokenized prompt? (So that when the day comes that they are running these prompts at scale, the token sizes really matter). Or maybe I'm imagining "prompts" wrong, and these are not just a few paragraphs of text, but actually 20,000+ tokens at a time, including data like e.g. pasted CSVs of trading history for the LLM to analyze?

gtoubassi 3y ago
Token counting is importing when you are injecting fetched data into the prompt to make sure you don't overflow the prompt size (e.g. in retrieval augmented generation). You want to give the LLM as many facts as will fit in the prompt to improve the quality of its response. So even with billions of dollars... token counting is a thing.
techbro92 3y ago
They might be analyzing sentiment with LLMs somehow. Which could be a larger scale task.
Barrin92 3y ago
>They have billions of dollars, so it struck me as odd to see such concern for efficiency You have it the wrong way around, it's because of that concern that they have billions of dollars. Obsessively making things a tiny bit more efficient is their business. Works both for software and for markets
- charlie0 3y ago
  Reminds me of the millionaire next door.
mtillman 3y ago
API costs are margin in trading and margin is everything. If you don’t have a way of managing that margin, you’re not a real trading firm. Pairing this type of execution with HFT would be their long term goal and that requires lots of feedback done in microseconds.
- caddemon 3y ago
  Jane Street isn't an especially big player in HFT, though even with a longer timescale your overall point still stands.
- user_named 3y ago
  Different type of margin.
nvm0n2 3y ago
I thought the more interesting thing was that they're spending time writing custom tokenizers, regex automatons and logging frameworks at all, just so it can be in OCaml. For the tokenizer they had three more obvious options: 1. Use a JavaScript lib and call it via FFI. Tokenization isn't a hard task. 2. Compile tiktoken to WASM and call it directly. 3. Stick with their Python server. Writing a custom regex automaton in OCaml and then running it in the browser is surely the worst option. How does this make sense, outside of just arbitrarily writing more OCaml for the sake of it? The justification for not using their previous scheme is only that it's "grungy", without further explanation. Every other project also seems to be like this. The justification for it boils down to nobody else using OCaml, so they need an intern army to reinvent all the wheels they're missing. I mean, I'm sure the interns had fun and were paid well. But there's a post below by someone with a massive inferiority complex, stating that everyone is "objectively inferior" to the people at Jane Street. This post doesn't make me feel inferior, it makes me feel a sense of awe and wonder that Jane Street's traders are so profitable they can subsidise this kind of objectively excessive IT bloat. Even the author seems to recognize this, it says the post is by the guy who has "the dubious honor of having convinced the firm to start using OCaml".
- caddemon 3y ago
  The primary purpose of an internship at most major companies is hiring - you can screen candidates much better before extending a full time offer and the candidates can also get a better read on how much they'd enjoy working there. Even the second order effect of drumming up hype for others to want to work there is probably more important to many internship programs than whatever the interns might tangibly produce. At least I can say for sure that people at Google aren't shy about how it takes a few months to get up to speed with internal tooling and not much is actually expected from a summer intern. But they built themselves into a brand on campuses, which became so mainstream there was a stupid movie made about how fun it is supposed to be to intern at Google. And as far as finance goes, trading interns aren't even legally allowed to trade, so I wouldn't be surprised if the culture surrounding internship at Jane Street just leans more heavily into this for SWEs as well.

gtoubassi 3y ago

Token counting is importing when you are injecting fetched data into the prompt to make sure you don't overflow the prompt size (e.g. in retrieval augmented generation). You want to give the LLM as many facts as will fit in the prompt to improve the quality of its response. So even with billions of dollars... token counting is a thing.

techbro92 3y ago

They might be analyzing sentiment with LLMs somehow. Which could be a larger scale task.

Barrin92 3y ago

>They have billions of dollars, so it struck me as odd to see such concern for efficiency You have it the wrong way around, it's because of that concern that they have billions of dollars. Obsessively making things a tiny bit more efficient is their business. Works both for software and for markets

charlie0 3y ago
Reminds me of the millionaire next door.

mtillman 3y ago

API costs are margin in trading and margin is everything. If you don’t have a way of managing that margin, you’re not a real trading firm. Pairing this type of execution with HFT would be their long term goal and that requires lots of feedback done in microseconds.

caddemon 3y ago
Jane Street isn't an especially big player in HFT, though even with a longer timescale your overall point still stands.
user_named 3y ago
Different type of margin.

nvm0n2 3y ago

I thought the more interesting thing was that they're spending time writing custom tokenizers, regex automatons and logging frameworks at all, just so it can be in OCaml. For the tokenizer they had three more obvious options: 1. Use a JavaScript lib and call it via FFI. Tokenization isn't a hard task. 2. Compile tiktoken to WASM and call it directly. 3. Stick with their Python server. Writing a custom regex automaton in OCaml and then running it in the browser is surely the worst option. How does this make sense, outside of just arbitrarily writing more OCaml for the sake of it? The justification for not using their previous scheme is only that it's "grungy", without further explanation. Every other project also seems to be like this. The justification for it boils down to nobody else using OCaml, so they need an intern army to reinvent all the wheels they're missing. I mean, I'm sure the interns had fun and were paid well. But there's a post below by someone with a massive inferiority complex, stating that everyone is "objectively inferior" to the people at Jane Street. This post doesn't make me feel inferior, it makes me feel a sense of awe and wonder that Jane Street's traders are so profitable they can subsidise this kind of objectively excessive IT bloat. Even the author seems to recognize this, it says the post is by the guy who has "the dubious honor of having convinced the firm to start using OCaml".

caddemon 3y ago
The primary purpose of an internship at most major companies is hiring - you can screen candidates much better before extending a full time offer and the candidates can also get a better read on how much they'd enjoy working there. Even the second order effect of drumming up hype for others to want to work there is probably more important to many internship programs than whatever the interns might tangibly produce. At least I can say for sure that people at Google aren't shy about how it takes a few months to get up to speed with internal tooling and not much is actually expected from a summer intern. But they built themselves into a brand on campuses, which became so mainstream there was a stupid movie made about how fun it is supposed to be to intern at Google. And as far as finance goes, trading interns aren't even legally allowed to trade, so I wouldn't be surprised if the culture surrounding internship at Jane Street just leans more heavily into this for SWEs as well.