12 ms·

OpenAI, at least in my day-day workflow for the last 9+ months has so superseded anything that google ever was to me that I'm having a difficult time comparing

• busyant 3y ago
  > OpenAI, at least in my day-day workflow for the last 9+ months has so superseded anything that google ever was to me that I'm having a difficult time comparing the two. Seconded. Let me tell you what I used to do. First, Imagine I have an error executing code or an error running some bit of 3rd party software. I go to Stack Overflow and search. I find posts related to my problem and I spend a great deal of time trying to shoehorn existing answers to my specific issue. Sometimes, my shoehorning works and I fix my problem. Other times, it doesn't work, and then I post on Stack Overflow myself. And I wait ... and wait ... for a response. Sometimes I get a response. Now, when I have this type of problem, I tell ChatGPT, "Hey, I'm trying to do <xyz> and I'm getting this error <abc>. Help me troubleshoot this. And it almost always helps me fix my problem. And it's ~10× faster than Stack Overflow. ============= Second, there are times where I have write code to do some relatively 'complex' data manipulation--nothing sophisticated, mind you, but stuff like, "I need these data columns rearranged based on complicated logic. And I need the text in columns A, X, AQ, and F merged, but only if <blah blah is true>. Otherwise, Just merge text in Columns A and AQ, except if the date in column ZZ is after January 1, 2019.". I can do this stuff on my own, but: a] it's cognitively draining, b] it takes time, c] I often make silly errors due to the complexity. ChatGPT is, again, an order of magnitude faster than I am. And it makes _fewer_ errors. It still makes errors. And I still have to know what to look for to catch those errors, but it decreases my cognitive load tremendously. edit: I haven't used Stack Overflow in 6 months. And "Googling" is Plan B. ============= Edit 2: I recently had to write a sympathy letter to someone whose husband died. I knew the general ideas behind what I wanted to say, but I knew I wasn't going to write anything great. I fired up ChatGPT and said, "write a sympathy letter to <x>. Tell her that I didn't know her husband well, but the few times I met him, I could tell that he cared deeply about you<x> and his daughter. I know his daughter well and I think she gets a lot of her great qualities from him and you. Tell her I don't know what to say in times like this. Keep it short-ish. Avoid schmaltz and sentimentality because <x> isn't that kind of person." It gave me about as perfect a letter as I could have asked for.
  • Rastonbury 3y ago
    As a non-professional coder, I'm honestly surprised there seems quite a few professional coders in this thread who don't do the first Stackoverflow first scenario. As someone learning code, shoehorning SO code when I don't understand 20% of the replies made the learning curve so steep, chatGPT has made this so much easier by changing it and telling me why will/won't work, even when watching out for or correcting hallucinations, overall its a time save
    • meiraleal 3y ago
      > I'm honestly surprised there seems quite a few professional coders in this thread who don't do the first Stackoverflow first scenario You just found your path to become a better developer than the majority then
    • smrq 3y ago
      As I've matured in my career I've gone to SO less and less, to the point where I mostly avoid it now. I use tools, languages, and libraries with which I am now highly familiar--perhaps more so than the average SO answerer, although I guess it would be arrogant to say for sure--and so the bulk of my questions are edge cases in my knowledge which are more easily answered by reading source code or specs. I suspect this is one of the differences between people that find GPT helpful and people that don't--it seems much more potentially helpful when you are in a less familiar environment. In my work, when I run into unexpected behavior, if I knew what question to ask then I would probably already know the answer too. Until the day I can feed in an entire codebase and ask "What's wrong with this? vaguely gestures at everything", I don't expect that to change significantly.
      • Rastonbury 3y ago
        That makes sense, and its funny because the other reply I got a joke was about how bad coders don't even bother going to SO
      • aprdm 3y ago
        Probably because you're deep specialized in something or have stagnated on a company for a long time ? I cannot imagine knowing everything about what I am working on, that has changed dramatically every 3-4 months even within the same company
• collyw 3y ago
  Have you checked the results it has given you? I tried it with some science related stuff. It literally made up a paper that didn't exist. I pointed this out so it linked to a real paper on an unrelated subject. For coding related stuff it is like a better stack overflow, but again it doesn't work well with niche libraries, or even some fairly big ones.
  • danielbln 3y ago
    Yea, LLMs hallucinate. They almost never hallucinate if you provide the right context, either via a search enabled LLM (ChatGPT with search, or perplexity.ai or phind.com etc.) or by providing it with more code/text/input to work with. Don't use LLMs as fact database lookup machines.
• dieselgate 3y ago
  This seems like an extreme case, I'm glad it works for you but remain skeptical and indifferent personally
• andrei_says_ 3y ago
  My issue with ChatGPT is that it sounds like an expert consultant but I know it’s likely it’s just making up sentences some of the time. And I’d need a real expert consultant to know when. So for an MBA major, 80% accuracy at 5% of the cost may be amazing but for me as an engineer and a person who cares, the inaccuracies are catastrophic. I’m open to suggestions on how to work with this.
  • maayank 3y ago
    > My issue with ChatGPT is that it sounds like an expert consultant but I know it’s likely it’s just making up sentences some of the time So, just like an expert consultant?
    • adamhp 3y ago
      I was just about to reply with this hahaha. This guy clearly does not know consultants.
      • andrei_says_ 3y ago
        I have worked with people who do say “I don’t know”
• upupupandaway 3y ago
  I have a different view. I have been trying to use GPT for some tasks like NP reductions, Linear Programming formulations, Graph problems, etc. and the results are maddeningly bad. Seriously, I have raged in front of my computer like I hadn't raged since the days of the NES Ghostbusters game. The amount of mistakes is incredible and the model seems to constantly feed into vicious circles of "I am sorry, you are correct, I made a mistake" then moving a few things around then being corrected again and ending up on the original mistake. Two interesting things it did for me lately: insisted that 8 + 6 = 11 and also proved that P = NP. I don't know which solutions it may be providing you, but it can't be anything too complex, or at least not too abstract. edit: typo
  • eugmill 3y ago
    I feel like every chatgpt thread needs every comment to be prepended with 3.5 or 4. It's incredibly difficult to tell when someone is having a bad time with 3.5, which has many known limitations, or 4 which tends to be more better at problem solving (though still makes dumb mistakes). I think the "GPT is amazing" vs "GPT is useless" debate is just going to get more confusing as more versions are released.
  • billythemaniam 3y ago
    I'm not surprised. LLMs are not good at those problems. There is a lot of hype, it not good at every problem, but it is quite good at some of them. If you have a classical NLP task, then it is good, particularly GPT4. If you have a generative problem where you don't care about mistakes, such as grocery list, marketing copy rough draft, etc, then it is good. LLMs are not good at search, math, encyclopedias, logic engines. Maybe some day they will be, but not yet.
  • lacker 3y ago
    ChatGPT (specifically GPT4) is very good at complicated but shallow questions. For example, "I'm setting up a new TypeScript + Svelte app, I made some changes to the config that I thought were good, but when I try to run the dev server, I get this error." And then paste in your incomprehensible ten line error. Any time you are using some software tool that you're not very familiar with, and you get an error message that you don't understand, try asking GPT4 to explain it to you. It isn't so great at deep, theoretical, algorithmic questions. "NP reductions" are probably not a great fit. Think of it as a research assistant that has a broad understanding of every technology in the world, but isn't as smart as you are about your specific area of expertise.
• xnx 3y ago
  > they are easily a $100B+ company once they nail down how to monetize That would be amazing if Google, Microsoft, Amazon, or Meta sat on their hands while OpenAI got that big.
• seydor 3y ago
  i havent found much use for it otoh outside programming code. I tried my best to make it help me review some papers, but it just comes back with generic replies and many errors. It isnt even good at finding typos. One thing it did, it deanonymized a paper because it was uploaded to biorxiv (but i could have done that googling). I am struggling to find uses for it beyond text beautification. OTOH, I have found a use for stablediffusion that actually resulted to some income.
• eachro 3y ago
  How do you control for hallucinations?
  • malux85 3y ago
    You use your critical faculties, very important things must be verified from other sources, unit, integration, numerical tests can evaluate code and results. Sometimes you need to go back and forward a bit, I tell ChatGPT it’s wrong and the error message and then it spits out the correct result, sometimes I need an algorithm tweaked because it has assumed a wrong constraint on the problem, again just explain it clearly and unambiguously and it will make corrections, there’s only been maybe 1/40 problems I couldn’t get a correct answer on after (sometimes a lot!) of back and forwards. I am not looking for a perfect oracle, I am looking for something to write 80% of the code and then I’ll fix it up. It’s still way faster this way, especially in domains I don’t know, E.g. I just learned CUDA with ChatGPTs help. It’s not perfect, and neither am I, but it doesn’t have to be perfect to be useful, you can get to millions in revenue through 80% solutions
  • quickthrower2 3y ago
    The same way you control for hallucinations with say Medium articles or Stackoverflow answers. You verify the answer. Being given some code and reviewing it is a lot quicker than writing that code. Copilot is great. Half the time it spits out the wrong answer but you can see what it was “thinking”. Phind giving you an answer and references lets you quickly double check. Sometimes it hallucinates but the answer and references combo is much better than a Google search which in turn is better than nothing. Of course I recommend people not use AI for everything. I will go straight to MDN for any WebAPI question and use —help as my first port of call on the command line. This is like your L2 cache as a developer. Using AI for everything is like swapping to disk for everything.
  • 2OEH8eoCRo0 3y ago
    Ask for sources. Ask if what it has already told me makes sense. Knowing that hallucinations happen and not to blindly trust. Learn the limits of your tools.
  • anonzzzies 3y ago
• theossuary 3y ago
  This interesting, I've had the opposite experience so far. I tried to do that because if it worked it'd be game changing; but it seems every question I ask it, I get half truths and lies back. Its impossible to figure out what is and isn't true either, because the LLM says everything with such a certainty. I end up having to research everything it says again to figure out what to trust. Sometimes it's good to get a foothold in a subject, but that's all I've been successful with so far.
  • alchemist1e9 3y ago
    You have such an experience with technical problems? It’s written at least 20 python scripts with me and almost all of them have been close to perfect almost on first draft.
    • goatlover 3y ago
      Really? That hasn't been my experience. Out of 20 python scripts, at least 50% have some error, some of which look right until you run them. A few involve hallucinations, like calling non-existent methods.
      • alchemist1e9 3y ago
        Actually yes that’s true, but within a few iterations it quickly repairs those flaws when given the errors and changes. I’m always running the draft script immediately and iterating on it. Sometimes in same context but often I’ll take it’s script, change it a bit, then start a new context with that version to keep adding features.
  • aprdm 3y ago
    Are you using 3.5 or 4? And what kind of things are you trying to do ? It's not like asking it, copy + pasting results + paste errors loop takes a long time to validate if it works or not. Even when it doesn't work it saves me time vs google-ing and also not solving my problem. Or gives me insights.
• bmitc 3y ago
  Are you able to give a specific example of a problem it helped you solve? Especially one that you were at a complete blocking point, it provided some solution, and then you were able to continue to expand upon that solution. I keep reading responses like yours, but I haven't seen any specific examples of problems being solved, so it all sounds very abstract. In my interactions with ChatGPT, it felt like just interacting with a search engine. There were zero continuations between questions and responses, and nearly 100% of the responses contained incorrect information. Edit 1: As an example, I just asked ChatGPT to implement the Warren Abstract Machine for me. It gave me two different implementations, both with caveats that they are simple examples and not the whole thing, and neither implementation even type checked. It feels just like reading someone's homework where they copied off of someone else's work and had no idea what's going on. I don't see the point in this if it's just going to give me some high-level idea, which I already have, and an implementation that isn't remotely complete, much less one that doesn't even run. "Additionally, you'd need to ... handle backtracking". You don't say, ChatGPT? Edit 2: I've kept on asking it to implement things I already know how to implement, even things that it probably has my code (being it was on GitHub) in its training data, and it keeps giving me code that doesn't even typecheck. ChatGPT is just doing what I've always imagined it doing: it's just doing a statistical merge. It has zero concept of anything I'm asking or it's saying. Edit 3: I asked it something regarding WebGPU. It gave the typical, oh that's complex and "Here's a simplified example using GLFW and WebGPU through the F# WebGPU bindings provided by FableGL". But FableGL isn't a thing that exists, and even if it did, it wouldn't have anything to do with WebGPU which has nothing to do with OpenGL. And it imported Fable.Import.WebGPU and Fable.Import.GLFW, neither of which exist. I mean, this is literally all smoke and mirrors. It boggles my mind when I hear people say they successfully use it every day. I haven't ever got it to tell me anything remotely correct.
  • usgroup 3y ago
    My experience is much the same, far from "I've essentially got 15-20 high-priced world-class consultants in every field that I chose to pull from, working at my beck and call..." that OP mentions. I suspect it is because I tend to have off piste questions which are not recipes or high-level ideas. It is where it'd be most useful to me, but it is at the same time where least training data is going to be. It is also where actual legitimate experts are most useful: the special sauce, not the meat and potatoes.
• bloppe 3y ago
  Mind if I ask what kind of projects you work on? And any specific examples where GPT really shined and delivered a lot of value?
  • ghshephard 3y ago
    All SRE type work. Where it specifically shines is whenever you need to dig deep into linux kernel specifics, networking specifics, code in different languages to handle issues (Rust (which I don't know), Go, Python, Bash, etc...), All sorts of error messages it knows so often what precisely needs to be done to resolve, using and configuring graphing, database, TSDB type queries, formulating a clever tcpdump query for a particular scenario, etc... Google Used to be able to handle a lot of the non-procedural questions - but somewhere in circa 2022/2023 something started happening to it's results and I started getting back mostly SEO churn, to the point I was going back to using manuals and having to dig in and learn the fundamentals on a lot of things - which is unsustainable if you are touching 30-50 different technology stacks. Chat changed all that - I can now go 3-5 levels deep in some stack, ask some incredibly nuanced question, get a reasonable answer that points me in the right direction, close off the issue and then move onto the next one.
    • sgarland 3y ago
      > having to dig in and learn the fundamentals on a lot of things - which is unsustainable if you are touching 30-50 different technology stacks. IMO, this is the crux of the problem, and it’s caused by allowing dev teams to use whatever stack they want, without also requiring them to support it. More cynically, it will always be the case if teams are allowed to use whatever they want, regardless of “service ownership” – because when there’s a SEV-1 and the owning teams have no idea what’s wrong, the SREs are going to be told to fix it.
      • meiraleal 3y ago
        > More cynically, it will always be the case if teams are allowed to use whatever they want Are you a manager? My cynical take is that this happens when managers choose tech as they know nothing and push for LinkedIn driven development
• soasdfg 3y ago
  I've also gotten good results that have added value to whatever I was working on at the time, but I also get a lot of partially correct responses, or responses where ChatGPT will forget what I told it seconds ago and repeat the same incorrect response from a few prompts ago. It's very effective if you know enough about what you ask it to disregard incorrect information quickly. It can be very helpful to guide you in a direction maybe you didn't consider looking into to begin with, but I don't fully trust it. There was a fun example I had a few months ago where I wanted to see how well it would do with being asked to solve a problem iteratively instead of recursively, something like: "generate all valid, distinct permutations of a provided string given that you have a dictionary to check valid words against" and it got most of the problem correct, but when asked to fix anything it would go right back to a recursive solution with the same issue appearing, or in some cases a new issue. It got me most of the way there with some edge cases I needed to handle myself, but it definitely seemed like that was as far as it was going to be able to go
• mywacaday 3y ago
  I'm bullish in the short to medium term. I also pay the $20 a month as it's worth it for the quick in co text answers. The company I work for (60k+ employees) has also rolled out an internal version Of ChatGPT and will be integrating live data with our ticketing system which will be a game changer from the demos I've seen. We are also doing a 9M trial of office copilot which I think that will be as big as office 365 as it's $30 per user per month which is costing 32 per user per month. Also bullish on Microsoft as a result. Hard to see how MS will be out manuvered in the enterprise.
  • layer8 3y ago
    I wonder if this will disadvantage smaller companies who can’t afford an on-premise instance but also can’t use the cloud version due to NDAs and data protection.
• enumjorge 3y ago
  I'd pay money to watch someone like you work. Like other responders, I've tried using chatgpt multiple times, and very often when I've used it for tasks where I was familiar with the subject matter, I was disappointed in the results. It's possible I haven't learned the proper way to get what I need from it and would love to learn how others are using it to solve real problems. I mean it. I'd pay money to watch streams of someone using chatgpt to solve non-trivial problems.
  • fyokdrigd 3y ago
    [flagged]
  • wenc 3y ago
    You can start by paying $20 for ChatGPT 4 and try it on tasks where you're familiar with the subject matter. I've tried it and I've been amazed.
    • blharr 3y ago
      Please, please, reference the tasks or chats that were especially impressive. For example, trying to get chat gpt to do something very simple for my work, like implementing a convolution it took me in circles and circles. It gets the general idea right, sure. But it actually makes significant minor errors that ended up being more confusing than helpful.
      • adamhp 3y ago
        Yesterday I asked it to generate i18n translations in a dozen languages for a few words in my app, in a specific syntax for React Native localization support. I have a pretty simple app I'm building with a few UIs, so only needed to translate a few words, but I basically just made my app shippable to a dozen other languages with a simple ask to ChatGPT. I'm not really an evangelist. It suits specific use cases well, and does require some learning and some engineering. I think people are expecting to say "Solve this very complex problem for me by yourself" and giving it a terrible prompt and then blaming it. It requires the same level of creativity that programming does. You have to think about what to ask it, and it what way. Once you get the hang of that, it's pretty powerful. In my example I said "I'm trying to translate my React Native app into multiple languages. I need an object that looks like this: <example JS code>. Can you generate this object for French, German, <insert more languages>, and for the following words: <words I needed>." Luckily I know basics in a few languages and the words were simple enough that I could easily verify it wasn't a hallucination. I also know enough JavaScript to know that it was syntactically sound, but my IDE would've told me if it wasn't.
• elforce002 3y ago
  I used it to create a credit score system to start a project and it gave me the basics. I don't expect anything other than that. The real issue will be that majority of users will take the answer face value without even knowing if that answer is a good one or just nonsense. "The AI said..." will create lots of issues. Time will tell.
• rossant 3y ago
  I use ChatGPT dozens of times a day and it saves me hours of work. I rarely experienced such a technical breakthrough in my lifetime that affects my workflow so profoundly.
  • warner25 3y ago
    The only breakthrough (technical or not) to which I can draw a comparison is getting always-on, cable Internet access for the first time in 2000.
• replwoacause 3y ago
  This has been my experience too