6 ms·

There is an ongoing bug[1] with OpenAI's API, where it stops streaming responses after exactly 5 minutes. When I first came across the issue, I debugged it by w

• smcleod 3y ago
  Yeah I’ve noticed similar, you can even explicitly say, do not truncate your output, or do not take any shortcuts. Make sure you haven’t missed anything ensure there are no placeholders etc…. And it ignores all of them.
• saliagato 3y ago
  I can get 8k + tokens response from OpenAI API without problems (with gpt-3.5-turbo-16k) but I encounter errors from Azure OpenAI when using the same prompt and parameters. Obviously it sucks because I can't get long response from gpt-4-32k on Azure
• gberger 3y ago
  That looks like a timeout configured on the web servebr
  • jw1224 3y ago
    Yes, it seems their servers are terminating responses early. Unfortunately it's undocumented, was introduced unexpectedly one day, and prevents full use of larger context-window models. It also charges in full for the entire response, which seems to continue running in the background. Definitely a bug at their end.
    • pbhjpbhj 3y ago
      Seems like a means to avoid DoS?
      • jw1224 3y ago
        I think it's just some networking misconfiguration they haven't noticed. I can't see them intentionally trying to prevent DoS this way, that's what API key rate limits are for (it would be a very expensive and slow DoS otherwise!).
        • charcircuit 3y ago
          >that's what API key rate limits are for You don't need an API key to make a connection to the API.
    • yossi_peti 3y ago
      Since it's 5 minutes I'm wondering if it's just "that's how the web works by default". I remember a while ago I was doing a long-running calculation on an POST request and was trying to debug why the connection always broke after 5 minutes, and found the hard way that there's a 5 minute timeout.
  • donpark 3y ago
    Very likely. Typical serverless hosts like Vercel times out at 5 minutes.
• retube 3y ago
  How does chat-gpt actually get this right? This would appear to require some degree of reasoning, but as far as i understand its output is purely probabilistic, based on existing corpus of text.
  • 1270018080 3y ago
    You answered your own question.
    • 15457345234 3y ago
      Can you elaborate?
      • jw1224 3y ago
        Your question: > How does chat-gpt actually get this right? Your answer: > its output is purely probabilistic, based on existing corpus of text Because GPT was trained on existing text, some of which included numbers and counting, it's learnt the natural ordering of most common/everyday numbers. For larger or more complex numbers, it's learnt the patterns behind how they're constructed linguistically, which allows it to output a written count in sequence. This same pattern recognition doesn't work anywhere near as well for actual numerals (e.g. "47600", instead of "forty seven thousand six hundred"), as the tokenizer tends to break long numerals apart (e.g. into ["476", "00"]).
        • pbhjpbhj 3y ago
          In text, we don't often count in series, and it seems likely that we often choose a non-counting sequence: like 'I chose options 1, 2, 7' or 'my code was 0 1 2 5', whatever. Unless training included line-level skips, rather than just next-word skips (like word2vec) or concept-level associations? At the line level, or paragraph level, ordered numerical sequences are obviously very common in formal texts or in code. I've seen sentence based training, I suppose for code (which it seems GPT4 excells at) line-level training would be essential. Anyone recommend a mid-level read on this covering different modes of training and such; I'm happy with a bit of code and undergrad level maths. Thanks.
• dontupvoteme 3y ago
  Wow, was this with the regular model (gpt-N) or a purported unchanging datestamped model like gpt-N-0314/0613?
  • jw1224 3y ago
    This was using gpt-4-0314, fixed and datestamped. The counting shortcut still affects newer models too, but the old version undoubtedly changed behaviour without warning.
    • dontupvoteme 3y ago
      That's honestly beyond the pale. Thanks for the heads-up.
• Stagnant 3y ago
  Did you take in to account that the default GPT4 model changed in June? That would explain the response difference. You can still use the original model released in march with the GPT-4-0314 identifier.
  • jw1224 3y ago
    I first noticed the counting shortcut back in April. It still persists with both the previous and current models.
• wouldbecouldbe 3y ago
  It's actually supper annoying. I keep asking it to write out entire pieces of code, instead of only things it changed, can be annoying to merge it otherwise. But nowadays it acknowledges my request "I will write complete code" and proceed to still comment out parts.
  • moonchrome 3y ago
    I think they probably finetuned that because it used to be the other way around and it was pretty bad UX to wait 1 min to apply a function change to a class or stuff like that.
    • wouldbecouldbe 3y ago
      Yeah sometimes you want it but sometimes you dont. I think they are enforcing it to save resources
      • jw1224 3y ago
        Are you using the API, or just the ChatGPT UI? Do custom instructions/system messages help at all? Saving resources like this seems okay(ish) for general-purpose ChatGPT, as they'd know there's a human who can interpret the abridged output. But for API-driven requests this would be pretty terrible, and makes it far harder to develop with.
        • wouldbecouldbe 3y ago
          no the UI, where I ask it to "Please write out entire code with skipping parts."
          • nomel 3y ago
            ChatGPT 3 or 4? Try the playground. I’ve had much better luck with system prompts.
  • pbhjpbhj 3y ago
• aragonite 3y ago
  I wonder if the streaming problems are related to the problem I experienced a lot lately to do with constant network connection error when using ChatGPT over mobile tethering (via Google Fi). The mobile connection itself is very stable and fast, but for a while every single ChatGPT response stops streaming halfway through with network connection error.
• visarga 3y ago
  I have this too, but on information extraction tasks, if the input document is too long or complex it will skip sections saying something along the lines of < !-- List of products. Truncated for brevity -- > (instead of the list) < !-- Shipper details go here -- > (instead of the details) Very annoying when you want to extract everything and is hard to fix. Maybe they had some examples solved by abbreviation in the fine-tuning set and it damaged the model. It doesn't make sense to train this behaviour into GPT-4.
  • antonvs 3y ago
    Using GPT models like that is unreliable anyway. I was talking to someone who was excited that he was able to use ChatGPT to convert a short tabular text listing of about 70 rows to a comma-separated list of the ids on each row, without having to e.g. import it into a spreadsheet or database. I pointed out that this couldn't be relied on, but he was skeptical. So I checked the output he pasted - there were only 69 ids. It had left out an id from the middle of the list, seemingly at random.
    • phatskat 3y ago
      This boggles my mind. I know doing it by hand, or using a tool takes more time and effort, but if you’re going to automate it away _anyway_ you should be doing it in a reliable and retractable way. I had a vim session saved that had a command history with a handful of regex patterns to format copy/pasted bank statement pdfs converted to docx (don’t ask) that would format the data into a CSV. Load the session, paste the ugly data, and cycle through the command history. It was ugly but reliable, and after the initial time sink of writing it once, it became quite efficient and wasn’t made more so (say a VimScript function) because it worked well enough.
      • antonvs 3y ago
        Yeah, I know. The person in question is CTO of a well-funded Series B startup, and is highly technical. But he's very much a cowboy when it comes to coding practices etc. I think LLMs appeal to him because they hold out the promise of not having to deal with his own developers when he's trying to do something himself that he doesn't already know how to do.
• SeanAnderson 3y ago
  This doesn't seem like an unintentional behavior? It seems like they don't want people querying their API for an indefinite amount of time. What amount of time do you expect a query to be able to run for before timing out?
  • jw1224 3y ago
    I lost countless hours debugging this, finally I came to the conclusion it must be unintentional because: 1. It's undocumented. None of the regular rate limit responses are returned. 2. You're charged for the full generation length. So if the output takes 10 minutes to generate, that's what you'll pay for (despite only getting half back). 3. It defeats the point of the larger context limit models. Why offer a 32K model if it fails after ~6K tokens? 4. The server response doesn't include any error codes or message, it simply terminates unexpectedly. Hit any of the actual rate limits, and you get told about it. I'd expect to be able to generate output until the model reaches its context limit, or a stop sequence is detected, or I hit an actual documented rate limit. We're paying for these requests in full. We should get the full response back!
    • meandmycode 3y ago
      I wouldn't be surprised if this was an Azure related issue, given some of the similar madness I've experienced on Azure
    • SeanAnderson 3y ago
      You have fair concerns. Thanks for elaborating. Sorry you experienced all that trouble.
    • hnfong 3y ago
      Given that OpenAI is probably heavy on ML scientist types and lean on SRE types, I suspect there's just some nginx setting at the load balancer/edge servers that sets the timeout to 5 minutes, and it wasn't communicated to whoever was designing the API/writing the docs.
• weird-eye-issue 3y ago
  The issue you linked to is related to an SDK, not the API itself. Have you tried just directly streaming the response back without an SDK? For example we use fetch directly without any SDK for OpenAI in production with no issue
  • jw1224 3y ago
    I've tried making manual requests without an SDK, exactly the same thing happens. Even when you don't use streaming (wait for the full response), it still occurs. Whilst the bug is API-specific, there's no other way to report this, and that issue is the only place I've found anyone discussing this. I contacted OpenAI support about it, waited several weeks, and just got a generic useless autoreply back.
• manojlds 3y ago
  That looks like a library issue than an API issue no?
  • jw1224 3y ago
    Definitely an API fault, but that library issue is the only place anyone's discussing it so far. OpenAI support were no help, I waited several weeks and they just sent back an autoreply linking to their docs.
• layer8 3y ago
  The inability to resolve issues like the mentioned 5-minute timeout with OpenAI support even after months should be a red flag to not build any products based on their services.
  • ilaksh 3y ago
    Can you give me a use case where generating a response for more than five minutes is important or even desirable?
    • layer8 3y ago
      It’s obviously a problem for the parent (https://news.ycombinator.com/user?id=jw1224 https://news.ycombinator.com/user?id=jw1224), but the issue is not primarily whether such a limit is reasonable or not, it’s that it’s undocumented and seemingly unintentional (relative to the limits they do document), and most importantly it’s that the lack of response from OpenAI support indicates that you can’t count on them providing any reliable and timely level of support. This is just unacceptable for a service you’d want to build a business on.
  • nomel 3y ago
    There’s no real alternatives for a product that would need their service to begin with.
    • layer8 3y ago
      That still means it’s a no-go to build a business on such a product, in my book.
      • appplication 3y ago
        I agree 100%. It’s irresponsible and frankly a stupid idea for anyone to bet on building a business that relies on ChatGPTs API (or really any private API in 2023). We’ve seen time and time again (and more and more recently) companies weaponizing API dependence and fees to drive sucker-ons out of business. If you want something to build a business around, build with an OSS model. Your competitive moat is going to be trash, but that’s going to be true for anyone building an AI startup that’s backed by any LLM technology. It’s a fool’s gold rush right now no matter how you slice it.
• TheHumanist 3y ago
  I was 'arguing' with it just yesterday about this with some coding work! I am newer to some things I am currently working on so it's been like a free tutor/mentor. Really nice at times. But lately, it will go into an explanation with code examples and part way through the code examples it adds things like '# The rest is very similar. Not including for brevity' Not exact wording of what it says but it is essentially saying that. It is incredibly annoying because I have started telling it not to do that, to include the full code of the example, and it STILL DOES THIS. No, I'm not getting it to code for me. The examples aren't even things I would use in my code but for teaching me certain concepts along with them walking through them with me which, in the past, worked really well and never had this repetitive issue. GPT-4 is definitely operating differently.
  • nielsole 3y ago
    Oh yeah, I had to write a simple but long Prometheus query. > [Correctly lists the first fifteen Elements] > # ... continue for each bucket > This would be for individual buckets, and you'd have to add more lines for each additional bucket in your range.Given that you have a wide range of buckets, writing this manually could be quite tedious. This is why pre-computing these values with recording rules in Prometheus might be a more scalable solution, but the above should give you an idea of how you could do this in a more manual fashion. Arguing with the AI who of the two of us has to do the tedious work.
• wbhart 3y ago
  The tendency to begin summarising is very annoying. I'd assumed it was because of limited attention span of human raters who rated summarised or shorter outputs more highly. And I'd assumed this had been there from the beginning. I encountered it when doing some research into getting GPT-4 to reliably multiply n-digit numbers. Up to 8x8 multiplications it doesn't do this very much, but by 10x10 it is almost impossible to get it to stop doing it. When the multiplications become even larger, it seems to be literally impossible to prevent.
• Wowfunhappy 3y ago
  I can't replicate this. I tested your prompt verbatim in the OpenAI playground. Model set to gpt-4-0314, maximum length set to 6485 (nothing special about this number, I just dragged the slider to something high-ish), all other settings left at their default. I cancelled the request when it reached "one thousand one hundred twenty-one" (which took over five minutes), but up until that point it did not take any shortcuts. I haven't meticulously checked every number, but it printed exactly 1,121 lines, and a cmd+F for "..." doesn't return anything.
  • deleted 3y ago
    [deleted]
• flir 3y ago
  I've seen that "one, two, skip a few" behaviour converting structured data from Russian to English, and in the occasional programming task. Gotta really emphasise you want the whole document, complete, translated. But we're fighting an uphill battle against OpenAI's own training, and I really wish we weren't. I have to wonder if it's coincidence that this behaviour is less computationally intensive.