6 ms·

An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a j

• bluechair 3mo ago
  Willing to be corrected but I believe this type of automated resume filtering is illegal. Not saying it never happens but my understanding is it is not typical.
  • small_scombrus 3mo ago
    They don't need to actually filter/blackhole to have have the same virtual effect. Show someone a list of resumes with an "applicant score*" and they'll naturally ignore the ones with a low ranking *scores are generated with AI, mistakes may be made, use only as a guide and verify results
  • thayne 3mo ago
    I would expect that to depend on jurisdiction. I don't know for sure, but I would be surprised if it was illegal in my particular US state. You might be able to argue the AI has inherent biases that introduce illegal discrimination in the hiring process, but my understanding is winning I case like that would be very difficult, especially since most employers are very cagey about their hiring process and why they mades a decision.
  • ivan_gammel 3mo ago
    In situations when you get hundreds of applications for one open position (real market now), whatever reduces your pool to the size a human can handle, works. You can preserve some diversity metrics in the process. This particular filtering is rather primitive, but LLM as a first filter can definitely do the job. You may burn less tokens than the hourly rate of your HR and it will be fairer than just dumping 50% of unread CVs in trash.
    • 369548684892826 3mo ago
      Great until someone realises you’ve filtered out minority groups from the application process (most developers are men so maybe the LLM decided they’re the best fit, but you’ll never know exactly why it screwed your over) and you suddenly have an expensive lawsuit
      • TeMPOraL 3mo ago
        LLMs are DEI-aware, as over past few years, their vendors all had various high profile news stories with their models and their default biases, so it's more likely they'll heavily discriminate in favor of minority candidates, not against them. Still, in both cases it would indicate whoever is operating the system is doing a really, really lazy job. It's really not hard to test and supervise LLMs on tasks where they give you mere 2-10x leverage, and prompt adherence today is much better than it was 3 years ago.
        • BigTTYGothGF 3mo ago
          Just last month: https://hai.stanford.edu/news/ai-hiring-tools-can-yield-racial-bias-and-systemic-rejection https://hai.stanford.edu/news/ai-hiring-tools-can-yield-raci...
        • FireBeyond 3mo ago
          Workday is facing a class action over claims their system is ageist. And another very large company dropped LLM use from their filtering, after finding it strongly preferred men over women, even when the females were objectively stronger resume candidates.
      • cyanydeez 3mo ago
        this happened a decade ago when a US courted tried to make sentencing decisions via ML. it was easialy demonstrated that the training data was flawed because the justice system was flawed so the data it was trained on was weighted against minorities because it oversampled because you know, police routinely oversample and poverty for es oversampling nonetheless, people will defend history as perfect and say those samples, like nepo babies, are "perfect".
  • dgellow 3mo ago
    Illegal where?
  • elric 3mo ago
    Under GDPR, you have the right to request manual processing whenever personal data is processed automatically to make a decision about you that has "significant impact". Not being hired seems like it would qualify.
• aesthesia 3mo ago
  A distribution with all probability mass on one outcome is deterministic, so in principle, setting temperature to 0 _should_ result in deterministic outputs. There are a few reasons it might not, but I don't think any of these apply when running a local model like the author did.
  • valzam 3mo ago
    I mean the easiest explanation would be that the model harness doesn't always take the most likely token but does top-k sampling or similar. temperatur just means that probabilities get more and more equalized, boosting the chance that an unlikely token gets picked. but even with temp 0 you could have 0.8 T1, 0.19 T2, ... and sometimes sample T2
    • aesthesia 3mo ago
      No, this can't happen at temperature 0. The formula defining temperature-adjusted softmax isn't strictly defined at 0, but taking the limit (in the case where all logits are distinct) results in probability 1 being placed on the largest logit. Samplers will typically special case temperature 0 and pick the most likely token at each step.
      • dvt 3mo ago
        This is a very authoritative answer that should be more nuanced and caveated as implementation-dependent. In some cases, repetition penalties take precedence over sampling; top_k and top_p can also be handled before or after the temperature step. In other cases, `0` is turned into like 1e-10 or some super tiny float value (which can drift if you do any arithmetic with it). Routing, quantization, etc. can also have an effect on sampling. And yes, in some cases, setting temperature to 0 can mean "pure greedy decoding" which makes the decoder about as deterministic as it can get.
  • easygenes 3mo ago
    There are. If the kernels are nondeterministic (e.g. timing issues) there are minor changes between runs, on a single system, even with eager decode enabled (typically what temperature=0 achieves).
• make3 3mo ago
  A more spikey distribution exactly makes the distribution closer to deterministic. That's not the point though. Even in greedy (deterministic) decoding, it is still a black box though that reacts in ways ways that are unpredictable to the inputs. Switching one word around might lead to different scores for example.
  • fluoridation 3mo ago
    Yeah, this is the forest that the people arguing about math trees are missing. It doesn't matter that the algorithm is deterministic if the algorithm passes the input through a cryptographic hash function to make a yes/no decision. The result may be perfectly reproducible and still non-sensical in its distribution with respect to its input domain.
• spwa4 3mo ago
  [flagged]
  • mnky9800n 3mo ago
    Test retest reliability is a thing in psychometrics.
    • spwa4 3mo ago
      [flagged]
      • mnky9800n 3mo ago
        There is evidence that children will oscillate between understanding and not understanding while learning topics. Philip Sadler at Harvard published about this but i can't find the paper im thinking of on his google scholar. too many papers! but moreover, to verify a test item you need to make sure that peopel will select the same answers under teh same conditions at different times. people generally forget the specific questions they were asked if you ask them the same questions a month later so being able to get them to answer the same way each time is important. it is assumed the people have some static knowledge of a topic in this scenario. If you want to consider a statistical examination of how people answer tests and how we assess knowledge and other things in people through surveying you can read about item response theory and rasch analysis.
  • smusamashah 3mo ago
    We expect computers to be consistent on the other hand. A calculator will always give you the same answer unless some chip gets struck by a particle. LLMs are on computers and should be fairly consistent too.
    • vidarh 3mo ago
      And this lies at the heart of the problem. We expect computers to be consistent despite running programs that are not designed to be consistent. This despite the fact that we have lots of experience of programs running on computers that produces wildly inconsistent outputs. But for some reason some people choose to assume LLMs should act like a calculator instead of any of those programs.
• miki123211 3mo ago
  In theory, temperature 0 does make the LLM deterministic. Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample (using the actual formula that works for non-zero values would cause a zero division). However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs.
  • sigmoid10 3mo ago
    >in theory theory, temperature 0 doesn't really exist. It does exist very much, even if you go to pure math. Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. I.e. in a discrete setting (like for LLMs with a finite set of output tokens), probability P becomes one for argmax and 0 for everything else. Only in coding practice it is easer to implement T=0 as a simple if check that directly chooses argmax instead of calculating the limit of some function that includes 1/T quotients. But setting T to zero is in both, theory and practice, turning the usual probability function into greedy sampling.
    • 317070 3mo ago
      > Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. In pure math, it does not always do that. It becomes a dirac-delta comb with equal weight on every maximum. There can be more than 1 maximum. Setting the temperature to zero turns into greedy sampling, but greedy sampling is not necessarily deterministic as you can have multiple equally optimal options.
      • sigmoid10 3mo ago
        That is not a problem for LLMs, because in practice floating point inaccuracies (in particular after exponentiation) prevent values from being exactly equal. That's why greedy sampling generally produces deterministic output for LLMs. The real gotchas are elsewhere (like with batch inference as we've seen with earlier GPTs). But unlike what the earlier comment says, this is a non-issue mathematically.
• bhanu786 3mo ago
  Agree
• deleted 3mo ago
  [deleted]
• nok22kon 3mo ago
  its a bad idea in general to use non-1.0 temperature. there is a reason labs are strongly recommending using 1.0. using low temperature is more deterministic, but the cost is the model becomes "dumber"
  • tipsytoad 3mo ago
    1.0 is actually pretty arbitrary and way too high as a general rule. Something like 0.3 is a more sensible default
    • embedding-shape 3mo ago
      Heavily depends on the model architecture and the implementation though, I don't think you can say what values are better than others without first specifying those, otherwise it's straight up guessing, ironically.
    • zipy124 3mo ago
      It really depends on the application does it not? I'm not an LLM guy, but for creative tasks like storytelling wouldn't you want a higher temperature usually? Happy to gain insight from anyone with experience here :)
    • 317070 3mo ago
      If RL was used to train the model, the model will have been trained on its own sequences. Those will have been generated with a temperature of 1.0. They must be, otherwise you would get a premature collapse or explosion of your entropy if the temperature was respectively lower or higher. After that RL step, you want to stick to the RL distribution, and so keep a temperature of 1.0. Other temperatures will drive the model out-of-distribution. That is why the sampling step for agents or thinking LLMs are usually kept at a temperature of 1.0.
    • nullc 3mo ago
      If you use a model in a configuration far from where it was RLed you get no warranty. (you also get no warranty the other way, however)
    • 3mo ago
• vessenes 3mo ago
  To be clear, temperature 0 is deterministic and will produce the same output for exact duplicate inputs, across all seed choices. Provided: * If it’s MoE we are talking about, that the duplicate inputs are for the whole batch (yes, your batch neighbours can impact your choice of experts. Blergh.) * Your kernels are deterministic * There’s no system wide effort switch that responds to, e.g. work load across the cluster (for a thinking model) Upshot: Temperature 0 is not deterministic in probably any existing cloud infra, but it could be for edge inference pretty reliably. To your quibble on 0.1 being more deterministic - I think it’s a pretty fair summary - we’re going to sample much more from the ‘temp 0’ answer at 0.1 than we would at temp 0.9, no?
  • Dylan16807 3mo ago
    Even then it's deterministic in the way a hash function is deterministic. Change one letter and you can get a completely different output. What people actually want is something continuous.
    • guhcampos 3mo ago
      This is it. People mistake deterministic for precise/exact/correct. It's not.
    • vessenes 3mo ago
      Agreed on the desire for continuous behavior. That said, in a modern LLM, is this hash analogy accurate? I would be surprised if a single letter changed most zero temp force ranked outputs. E.g: “Where is the Eiffel Tower Located? One word only.” “Where is the Effel Tower located? One word only.” “Where is the Eiffel Tower located? One wor only.” I’d be very surprised if those got different answers from even a small local model at temp 0.
      • knome 3mo ago
        For a single word response, perhaps. But for anything else I wouldn't. The entire chain will be affected from the different tokenization on down. Even if it lands in roughly the same semantic area, it doesn't mean it will land there with anything like the same syntactic selections. Anywhere there were multiple near-tokens could easily select a different route based on even minor fluctuations in the starting conditions. It's chaotic.
• lelanthran 3mo ago
  > temperature is not some kind of "deterministic" switch, but rather it affects the sampling distribution (which becomes more "spiky"—but is still very much a distribution). You're correct. The confusion arises because we use the word "non-deterministic" when we mean "probabilistic". I tried to explain it better: https://www.lelanthran.com/chap15/content.html https://www.lelanthran.com/chap15/content.html
• mtharrison 3mo ago
  Small refinement: the underlying model isn’t stochastic at all. The forward pass is a deterministic function of the weights and input, it just produces a probability distribution over the next token. The stochasticity is an optional sampling step layered on top, not something inherent to LLMs. Greedy/argmax decoding (or temperature 0) makes the whole thing deterministic. So “purely stochastic” overstates it a bit: the distribution is computed deterministically, and you choose whether to sample from it or not.
  • simiones 3mo ago
    There are more layers to this problem, if we want to get into the details. The LLM is defined in terms of floating point operations, and those are not actually fully deterministic, on most hardware and in most performant implementations. IEEE 754 only specifies precision requirements for certain operations, not precise bit patterns (e.g. for exponentials). So, at least in principle, the same hardware performing the same operation could produce different results at different times, as long as they are close enough to the theoretical answer. I'm not sure if any hardware actually works like this. IEEE 754 also specifies that many of the basic arithmetic operations are not associative - so any reordering (which is common when batching multiple queries at the same time) will introduce indeterminacy from the perspective of your own query (that is the result for your query will change depending on what other query happens to be processed at the same time, which is not under your control). Finally, even if we take the case when a query is processed alone, and even if one particular hardware is completely deterministic, the result will be different on different hardware - which can again look like non-determinism if you're sending your query to a load balancer. So, the math for LLMs is deterministic in theory, but implemented with non-deterministic approximations & optimizations in practice, and their results are then normally used only as a probability distribution to be sampled from.
• margalabargala 3mo ago
  > I'm happy to see in-depth pieces like this It's somewhat ironic that this "in depth" piece was written by an LLM as well.
• PaulHoule 3mo ago
  The whole problem of text understanding is a problem of reasoning under uncertainty, that is, you can't really be sure which witch people are talking about all the time. A person you might hire might be successful or unsuccessful at the role, no matter what hiring process you use. Two people might look at the same resume and come to the same conclusions. Two patients with the same symptoms and clinical presentation might have different diseases, etc. I don't buy the story that the old AI died primarily due to the cost of knowledge base maintenance [1], but rather the lack of a universal system of reasoning over uncertainty. For me it's a running gag that Spock was always saying things like "Captain, we have a 21% probability of surviving this mission" when Bayes teaches us your probability distribution has a probability distribution, "we have a β(5,1) chance of surviving this mission" is more like it. To that end it wouldn't be too crazy to run a resume through that machine 100 times and look at the probability distribution of the score. [1] then again I am the kind of maniac who will sort images on a tablet lying in bed until my visual system malfunctions
• mywittyname 3mo ago
  > This is not correct Several of my claimed AI-expert colleagues repeat this as though it's gospel. I've heard "set the temperature to 0 so we get consistent results" more times that I can count.
  • Terr_ 3mo ago
    I imagine it's much like game-developers saying: "Set a fixed seed so the player gets consistent results." Yeah, it can work, but it is subject to so many potential pitfalls that you can't casually assume it will. It's a property you have to actively design-for and rigorously test to be sure the system can deliver it for some particular scenario.
• thesuitonym 3mo ago
  > resumes are just dumped in some LLM black hole and no one really knows how it works. Not that I'm defending AI, but HR departments rarely knew how their ATS ranked and sorted applicants before they were AI powered.
• Nimitz14 3mo ago
  He said it nudges it to be more deterministic. Your comment is not correct.
• fatnoah 3mo ago
  My favorite recent example was submitting a resume for a job that was almost a word-for-word description of my current title and job at a similarly sized company. Within 24 hours, I got the rejection, and several days later, a recruiter reached out to let me know that my profile looked like a great match for the role and wanted to schedule an intro call.
  • FireBeyond 3mo ago
    I remember applying for one position that was around building care and management systems for pregnant mothers - EHR, practice management, claims benefits, etc., all of which I had over a decade of experience in. "Name something that might stand you out from the crowd." "In addition to all this I've also delivered 12 babies as a paramedic". Twenty minutes later "we are looking for candidates whose experiences and skill sets are more closely aligned with the role we are looking to fill".