7 ms·

Meta: when you ask chatgpt what happens when you do pop_back() on an empty vector, it confidently claims std::out_of_range will be thrown. It will even 'cite' t

• gpderetta 4y ago
  > Is this something that in practice is implemented in different (exception-throwing) ways? I don't think so. Even when enabling debugging mode, a precondition failure would cause an abort, not an exception to be thrown.
• saurik 4y ago
  God help us if people start trying to use an overconfident chatbot to replace documentation :(.
  • roel_v 4y ago
    Maybe. To be fair, I had the same opinion yesterday. This morning I decided to take a real look at how I could use chatgpt to augment my programming, and once I started with it with an open mind, I now think it's going to take away much of the drudgery I have come to loathe so much about programming. I used to like delving into libraries and frameworks and learning all their details - I thought that was what 'learning about programming' was. Now, 20 years later, I hate those aspects with a passion - having to look up the idiosyncrasies of a function call again, because there are dozens of ways of using it but all slightly different, or piecing together another dull specific use case. This morning I had chatgpt generate 4 fully fledged functions for me, each doing a highly specific task that would have taken maybe 20 or 30 minutes each to piece together the specific API's, and one of them using a Python package I had never used before, didn't know of and would have had to read the getting started tutorial before I could have used it on my own. Then all I had to do was write the glue to call those functions in the right order and with the right arguments and boom that was it. Plus the code was more elaborate and coherent than it would have been had I have to do it manually. One example is not a guarantee, but the experience made my day, and probably my week. Which is why the pop_back() thing made me worry :)
    • Someone 4y ago
      > This morning I had chatgpt generate 4 fully fledged functions for me, each doing a highly specific task that would have taken maybe 20 or 30 minutes each to piece together the specific API's, and one of them using a Python package I had never used before, didn't know of and would have had to read the getting started tutorial before I could have used it on my own. Then all I had to do was write the glue to call those functions in the right order and with the right arguments and boom that was it. “Boom that was it” _if_ you blindly trust ChatGPT to generate correct code. If you don’t (and IMO you shouldn’t), “it worked for me today” isn’t sufficient to decide whether you can trust it. You should go to the documentation of that package, judge whether ChatGPT used it correctly (and yes, that involves looking up the idiosyncrasies of function calls), judge whether you trust that package, evaluate its license, etc.
      • deleted 4y ago
        [deleted]
      • wizofaus 4y ago
        Can it also generate unit tests for you to prove the functions work as expected? Obviously those unit tests need to be something a quick glance at will confirm are actually usefully verifying expected behaviour (of course, there are circumstances that's nigh on impossible to do even with the most carefully handcrafted tests).
    • nspattak 4y ago
      Amen.
  • drewcoo 4y ago
    Chatbot moves in mysterious ways.
  • saagarjha 4y ago
    I mean it's fine if you can double-check what it tells you quickly, it's just pretty bad with something like C++ because when it gives you something wrong the program may or may not tell you about it. For example, what it might be good for is "what are the parameters to this 6-argument function that plots a rectangle on the screen", because that's something you can confirm really easily and there are no real consequences to getting it wrong. On the other hand, someone sent me a screenshot of how ChatGPT was really helpful in figuring out how reader-writer locks work, and it told them "you can use pthread_rwlock_trywrlock to upgrade a reader lock to a writer lock" :/
  • JulianMorrison 4y ago
    They're using it to replace "search on stack overflow, cut and paste, does it work?"
  • bigbillheck 4y ago
    > if * when
  • hobs 4y ago
    https://supabase.com/blog/chatgpt-supabase-docs https://supabase.com/blog/chatgpt-supabase-docs Already on HN
    • williamcotton 4y ago
      The technique they’re using is much less error prone due to the nature of the prompt. When a prompt is “analytic” in nature, that is contains the facts required for the output, like “convert this following box score into an entertaining account of the baseball game it describes”, it works rather well. When the prompt is “synthetic” in nature, when it doesn’t have the facts in the prompt, like “based on that same box score, provide some direct quotes from the broadcasters”, it does not reliably provide factual responses.
  • taneq 4y ago
    Doesn't see any worse than asking HN. :P
  • simplotek 4y ago
    > God help us if people start trying to use an overconfident chatbot to replace documentation :(. A chat bot looks like a great help to parse through huge volumes of documentation. If the chatbot built around the docs outputs garbage, that is a telltale sign that the documentation was garbage to start with. Just because people started developing clever chatbots that misfire, let's not fool ourselves into believing documentation was a solved problem and all docs were squeaky clean.
    • ziml77 4y ago
      Poor output from a chat bot indicating poor documentation would make sense if the chat bot could limit itself to authoritative information from the documentation, but that's not how these models operate. I'm not even sure how you could make them work that way because if you trained them solely on the documentation, they wouldn't be able to operate in a conversational manner.
      • imtringued 4y ago
        You can tune ChatGPT with custom data.
        • saurik 4y ago
          This is, AFAIK, not a task for fine tuning--which is more like trying to get a certain kind of style-but maybe a task for "semantic search". I watched this video a few days ago, and at least vaguely recommend it: https://youtu.be/9qq6HTr7Ocw https://youtu.be/9qq6HTr7Ocw
    • imtringued 4y ago
      Here is the documentation that you consider unclear >If the container is not empty, the function never throws exceptions (no-throw guarantee). Otherwise, it causes undefined behavior. ChatGPT failed at basic regurgitation.
      • tialaramex 4y ago
  • SketchySeaBeast 4y ago
    That would very much be the Warhammer 40k machine spirit - the operator would only do what they can to appease it, and even the spirit wouldn't know how or why something did or didn't work.
• aw1621107 4y ago
  > I don't have a copy of the standard at hand, can anyone quote the relevant section? The C++ (draft) standard is on GitHub! [0] Compiling it needs Perl and some LaTeX packages, but is reasonably straightforwards otherwise. In addition, links to specific draft standards can be found on cppreference [1]. But anyways, in the first C++20 post-publication draft (N4868), the wording you're interested in is in multiple sections. Section 22.2.3 Sequence Containers [sequence.reqmts] has Table 78: Optional sequence container operations [tab:container.seq.opt] (starting on page 815), which states that a precondition of pop_back() is that empty() returns false. Section 16.3.2.4 Detailed Specifications [structure.specifications] (page 481) states: > Preconditions: the conditions that the function assumes to hold whenever it is called; violation of any preconditions results in undefined behavior. Therefore, calling pop_back() on an empty vector results in undefined behavior. > Is this something that in practice is implemented in different (exception-throwing) ways? Based on a quick glance at the major implementations (libc++ 15.0.7 at [2], MSVC at [3], libstdc++ at [4]), it looks like asserts are used. Whether those result in exceptions probably depends on whether the asserts are compiled in in the first place and how they are implemented, but it's definitely not a guaranteed exception. [0]: https://github.com/cplusplus/draft https://github.com/cplusplus/draft [1]: https://en.cppreference.com/w/cpp/links https://en.cppreference.com/w/cpp/links [2]: https://github.com/llvm/llvm-project/blob/llvmorg-15.0.7/libcxx/include/vector#L1644 https://github.com/llvm/llvm-project/blob/llvmorg-15.0.7/lib... [3]: https://github.com/llvm/llvm-project/blob/8dfdcc7b7bf66834a761bd8de445840ef68e4d1a/libcxx/include/vector#L1644 https://github.com/llvm/llvm-project/blob/8dfdcc7b7bf66834a7... [4]: https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=libstdc%2B%2B-v3/include/bits/stl_vector.h;h=86ff83d137343ca55a0703bc3d8d6a343468f474;hb=HEAD#l1315 https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=libstdc%2B%2B-v3...
  • jb1991 4y ago
    > Compiling it needs Perl this was something that I thought must be a joke at first... a text document that needs ... Perl ... to be read. This does make sense considering the context is C++ though.
    • aw1621107 4y ago
      The C++ standard is a LaTeX document, not a plain text document. Perl is needed for latexmk [0], which isn't strictly necessary but greatly simplifies the process of compiling LaTeX. [0]: https://mg.readthedocs.io/latexmk.html https://mg.readthedocs.io/latexmk.html
• planede 4y ago
  Current C++ standard draft it hosted at https://eel.is/c++draft/ https://eel.is/c++draft/, at this time this is the draft for C++23. For earlier C++ standard versions the final drafts before ISO standardization are hosted at https://github.com/timsong-cpp/cppwp https://github.com/timsong-cpp/cppwp . The paid ISO standardized version is supposedly not meaningfully different. Relevant parts of the standard (C++20): * pop_back: https://timsong-cpp.github.io/cppwp/n4868/containers#tab:container.seq.opt-row-11 https://timsong-cpp.github.io/cppwp/n4868/containers#tab:con... "Preconditions: a.empty() is false." * Meaning of "precondition": https://timsong-cpp.github.io/cppwp/n4868/library#structure.specifications-3.3 https://timsong-cpp.github.io/cppwp/n4868/library#structure.... Reading the standard can be quite a challenge. The standard tries to not repeat itself, which often means that you don't get your answer in a self-contained paragraph, but you have to hunt down cross-references and definitions. As a C++ language reference I highly recommend https://en.cppreference.com https://en.cppreference.com . edit: In this case https://en.cppreference.com/w/cpp/container/vector/pop_back https://en.cppreference.com/w/cpp/container/vector/pop_back is super clear. Basically everything you want to know about the function in three sentences: > Removes the last element of the container. Calling pop_back on an empty container results in undefined behavior. Iterators and references to the last element, as well as the end() iterator, are invalidated.
  • mattgreenrocks 4y ago
    I'm guessing that ChatGPT answered the query based off of implementation-specific details, rather than the standard itself. There's a dark irony in the fact that programmers do this all the time.
    • planede 4y ago
      No implementation that I know of throws an exception here.
      • mattgreenrocks 4y ago
        I stand corrected. It sounded like something that would be done in debug mode by some STL implementations.
• dpkirchner 4y ago
  When I asked ChatGPT "In C++, what happens when you call pop_back() on an empty vector?" I got the answer: > When you call pop_back() on an empty vector in C++, it leads to undefined behavior. The C++ standard does not specify what should happen in this scenario. Attempting to remove an element from an empty container is an error and can cause the program to crash or produce incorrect results. To avoid this issue, you should always check the size of the vector before calling pop_back() or use other methods such as empty() to determine if the vector is empty. Do you remember how you phrased your question? Maybe it's just luck it got it right for me or maybe it's the order of the terms in my question.
  • roel_v 4y ago
    Here's my exact chat log: what happens when you do pop_back() on an empty vector in C++? In C++, when you call the pop_back() function on an empty vector, it will result in an std::out_of_range exception being thrown. The exception indicates that an element cannot be removed from an empty vector, as there is nothing to remove. To prevent the exception, it is recommended to check the size of the vector before calling pop_back(), to ensure that the vector is not empty. can you quote the relevant C++ standard on that? Sure! Here is the relevant excerpt from the C++ Standard (C++20, ISO/IEC 14882:2020): quote the relevant passage from the standard Here is the relevant excerpt from the C++ Standard (C++20, ISO/IEC 14882:2020): 23.6.11 Vector data members [vector.data] -1- The member function pop_back() shall have no effect if the vector is empty. 23.6.6 Vector capacity [vector.capacity] -1- void pop_back(); Requires: empty() == false. Effects: If size() > 0, caller shall call erase(end() - 1). Throws: Nothing unless erase() throws an exception. 21.5.5.5 vector<T>::pop_back() [vector.pop_back] -1- Removes the last element in the vector. Complexity: Constant time. Throws: Nothing unless erase() throws an exception.
    • aw1621107 4y ago
      Pretty interesting output. Some notes from a cursory glance/crosscheck against the first post-C++20 draft [0]: It gets the ISO/IEC number correct [1]. However, the sections are wrong - Section 23.6 is about stream iterators. Sections 23.6.6 and 23.6.11 do not exist. - Section 21.5.5 is a synopsis of the <cuchar> header. Section 21.5.5.5 does not exist - Section [vector.data] exists and is numbered 22.3.11.4. It only describes the data() function. - Section [vector.capacity] exists and is numbered 22.3.11.3. It describes the capacity(), reserve(), shrink_to_fit(), swap(), and resize() functions. - pop_back() is generally described in Table 78 (Section 22.2.3 Sequence Containers, [sequence.reqmts]), and is expanded upon for std::vector in Section 22.3.11.5 Modifiers [vector.modifiers]. The wording is not correct either. - Searching "shall have no effect" returns no hits (though I'm searching in a PDF, so maybe the search just misses it?). - The standard wording for pop_back() in general is: "Effects: Destroys the last element. Preconditions: a.empty() is false". The Standard says nothing about exceptions. - For std::vector specifically, the Standard says "Effects: Invalidates iterators and references at or after the point of the erase. Throws: Nothing unless an exception is thrown by the assignment operator or move assignment operator of T. Complexity: The destructor of T is called the number of times equal to the number of the elements erased, but the assignment operator of T is called the number of time equal to the number of elements in the vector after the erased elements." [0]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/n4868.pdf https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/n48... [1]: https://www.iso.org/standard/79358.html https://www.iso.org/standard/79358.html
• WalterBright 4y ago
  > when you ask chatgpt what happens Big mistake. I asked chatgpt to summarize the plot of 2001. It replied with Hal locked Dave inside the Discovery and wouldn't open the pod bay doors to let him out. Answers about code is similarly very plausible, but subtly and completely wrong.