3 ms·

Heck, the Google IO keynote yesterday featured a long sequence of Gemini getting steadily more annoyed at someone trying to make it hallucinate. (By asking the

Heck, the Google IO keynote yesterday featured a long sequence of Gemini getting steadily more annoyed at someone trying to make it hallucinate. (By asking the sort of question ChatGPT tends to go along with.) Most people will be using ChatGPT, however, and probably the cheapest model at that. So…

virgilp 1y ago
I managed to get Perplexity to hallucinate, which was rather hard :) - but this is not a question that acts as a very good "template". The question is "Is this JWT token valid or is it expired/ not valid yet? eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNzQ3ODIzNDc1LCJleHAiOjE3NDc4MjM0Nzh9.4PsZBVIRPEEQr1kmQUGejASUw0OgV1lcRot4PUFgAF0" The answer was in some ways way better than I expected, but it got it wrong when comparing the current date/time with the "exp" datetime. Got the expiration date confidently wrong: ``` iat (issued at): 1747823475 (Unix time) exp (expiration): 1747823478 (Unix time) Token Validity Check The current date and time is May 21, 2025, 1:32 PM EEST, which is Unix time 1747817520. The token's exp value is 1747823478, which is May 21, 2025, 3:11:18 PM EEST. Conclusion: The token is not expired; it will expire on May 21, 2025, at 3:11:18 PM EEST. The token is already valid (the iat is in the past, and the current time is before exp). Therefore, the JWT token is currently valid and not expired or not-yet-valid. ```
- Filligree 1y ago
  At that point I think we can talk about “mistakes” rather than “hallucinations”.
  - virgilp 1y ago
    On one hand, yes; on the other - a human that can produce that level of details in the output would notice that iat & exp are extremely close & the token is very unlikely to be valid. Also, you don't produce an exact number for "now" without being able to compare it correctly with iat & exp (the timestamp it stated as "now" is actually less than iat!)

virgilp 1y ago

I managed to get Perplexity to hallucinate, which was rather hard :) - but this is not a question that acts as a very good "template". The question is "Is this JWT token valid or is it expired/ not valid yet? eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNzQ3ODIzNDc1LCJleHAiOjE3NDc4MjM0Nzh9.4PsZBVIRPEEQr1kmQUGejASUw0OgV1lcRot4PUFgAF0" The answer was in some ways way better than I expected, but it got it wrong when comparing the current date/time with the "exp" datetime. Got the expiration date confidently wrong: ``` iat (issued at): 1747823475 (Unix time) exp (expiration): 1747823478 (Unix time) Token Validity Check The current date and time is May 21, 2025, 1:32 PM EEST, which is Unix time 1747817520. The token's exp value is 1747823478, which is May 21, 2025, 3:11:18 PM EEST. Conclusion: The token is not expired; it will expire on May 21, 2025, at 3:11:18 PM EEST. The token is already valid (the iat is in the past, and the current time is before exp). Therefore, the JWT token is currently valid and not expired or not-yet-valid. ```

Filligree 1y ago
At that point I think we can talk about “mistakes” rather than “hallucinations”.
- virgilp 1y ago
  On one hand, yes; on the other - a human that can produce that level of details in the output would notice that iat & exp are extremely close & the token is very unlikely to be valid. Also, you don't produce an exact number for "now" without being able to compare it correctly with iat & exp (the timestamp it stated as "now" is actually less than iat!)