27 ms·

I was on a team developing a critical public safety system on a tight deadline a few years ago, and i had to translate some wireframes for the admin back-end in

• miiiiiike 3y ago
  I’d need to see it. I can’t get ChatGPT to outperform a novice. And now I’m having candidates argue that they don’t need to learn the fundamentals because LLMs can do it for them.. Good luck HTML/CSS expert who couldn’t produce a valid HTML5 skeleton. Reminds me of the pre-LLM guy who said he was having trouble because usually uses React.. So I told him he could use React. I don’t mean to rag on novices but these guys really seemed to think the question was beneath them. If you want to get back into front-end read “CSS: The Definitive Guide”. Great book, gives you a complete understanding of CSS by the end.
  • ern 3y ago
    Just out of curiosity: is the code generated by ChatGPT not what you expected or is it failing to produce the result that you wanted. I suspect you mean the latter, but just wanted to confirm.
    • miiiiiike 3y ago
      The statements are factually inaccurate and the code doesn’t do what it claims it should.
      • meiraleal 3y ago
        Right. That's an experience completely different from the majority here that have been able to produce code that integrates seamlessly into their projects. Do you have any idea why? I guess we should start by what version of ChatGPT you are using.
        • BlueTemplar 3y ago
          ChatGPT might as well not exist - I'm not touching anything GAFAM-related. Any worthy examples of open source neural networks, ideally not from companies based in rogue states like the US ?
          • meiraleal 3y ago
            Falcon is developed by an UAE tech arm, not sure if you would consider it a rogue state or not: https://falconllm.tii.ae/ https://falconllm.tii.ae/
            • BlueTemplar 3y ago
              What is «tech» supposed to mean here ? Infocoms ? The United Arab Emirates ? Well, lol, of course I do, that's way worse than the US.
        • deleted 3y ago
          [deleted]
  • vidarh 3y ago
    Requirements vary. It certainly can't produce really complex visual designs, or code a designer would be very happy with, but I have a hobby project work in progress where gpt4 has produced all of the CSS and templates. I have no doubt that the only reason that worked well is that it's a simple design of a type there is about a billion of in its training set and that it'd fall apart quickly if I started deviating much from that. But if t produced both clean CSS and something nicer looking than I suspect I would have myself. A designer would probably still beat it - this doesn't compete with someone well paid to work on heavily custom designs. But at this point it does compete with places like Fiverr for me for things I can't or don't want to do myself. It'll take several iterations for it to eat it's way up the value chain, but it probably will. But also, I suspect a lot of the lower end of the value chain, or at least part of them, will pull themselves up and start to compete with the lower end of the middle by figuring out how to use LLMs to take on bigger, more complex projects.
    • miiiiiike 3y ago
      This meshes pretty well with my experience.
  • flir 3y ago
    I'm always asking it to stitch together ad hoc bash command lines for me, eg "find all the files called *.foo in directories called bar and search them for baz". (`find / -type d -name 'bar' -exec find {} -type f -name '*.foo' \; | xargs grep 'baz'` apparently.) I would have done that differently, but it's close enough for government work.
    • BlueTemplar 3y ago
      Except you should at least try to write code for someone else (and probably of lower level of competence - this also helps for your own debugging later) - obscure one-liners like these should be rejected.
      • Zababa 3y ago
        Assuming there is a comment just above the one-liner saying "find all directories named 'bar', find all files named '*.foo' in those directories, search those files for 'baz'", this code is perfectly clear. Even without the comment, it's not hard to understand.
      • flir 3y ago
        I wouldn't call it obscure, just bog standard command line stuff. How would you have done it?
        • deleted 3y ago
          [deleted]
      • radiator 3y ago
        To me, the only obscure thing about this is that it is a one-liner. If you write it in three lines, it is fine. Although, I guess the second find and the grep could be shortened, combined into one command.
        • BlueTemplar 3y ago
          I agree - but my point still stands.
  • Cthulhu_ 3y ago
    I haven't practiced or needed to use the fundamentals in literal years; I'm sure I'd fumble some of these tests, and I've got err, 15 years of experience. It's good to know the fundamentals and be able to find them IF you find a situation where you need them (e.g. performance tuning), but in my anecdotal and limited experience, you're fine staying higher level.
    • dragonelite 3y ago
      Kind of same sort of situation, but i do like to refresh some of the fundamentals every 3~4 years or so. Usually when i do a job hop. Its kind of like asking an olympic sprinter how to walk fast.
    • soco 3y ago
      I had a chilling experience of late when, out of curiosity, I tried the actual online practice exam for driving school. Boy did I fail it. I realized that there are quite some road signs I never saw in my life, and more important, that my current solution to all their right of way questions is "slow down and see what the others do" - not even that wrong if I think about but won't get you points in the exam.
      • collyw 3y ago
        And I suspect you would be a lot less likely to be involved in a crash than someone who had just passed the test.
    • worksonmine 3y ago
      There are levels of fundamentals though, since parent mentioned HTML/CSS/React I guess they're referring to being able create a layout by hand vs using a CSS framework/library. You don't need to know how a CPU works to fix a CSS issue, but if all you know is combining the classes available you'll have trouble with even the simplest web development. Everyone should know enough fundamentals to be able to write simple implementations of the frameworks they depend on.
  • j45 3y ago
    Sometimes it requires expert guidance to get something meaningful out.
    • Art9681 3y ago
      This is the correct answer. I have 23 years of experience in datacenter ops and it has been a game changer for me. Just like any tool on one's arsenal, it's utility increases with practice and learning to use it correctly. ChatGPT is no different. You get out of it what you put in to it. This is the way of the world. I used to be puzzled as to why my peers are so dismissive of this tech. Same folks who would say "We don't need to learn no Kubernetes! We don't need to code! We don't need ChatGPT". They don't! And it's fine. If their idea of a career is working in same small co. doing the same basic Linux sysadmin tasks for a third of the salary I make then more power to them. The folks dismissive of the AI/ML tech are effectively capping their salary and future prospects in this industry. This is good for us! More demand for experts and less supply. You ever hire someone that uses punch cards to code? Neither have I.
      • varelse 3y ago
        [dead]
      • kevindamm 3y ago
        I think it's more akin to using compilers in the early days of BCPL or C. You could expect it to produce working assembly for most code but sometimes it would be slower than a hand-tuned version and sometimes a compiler bug would surface, but it would work well enough most of the time. For decades there were still people who coded directly in assembly, and with good reason. And eventually the compiler bugs would be encountered less frequently (and the programmer would get a better understanding of undefined behavior in that language). Similar to how dropping into inline assembly for speeding up execution time can still have its place sometimes, I think using GPT for small blocks of code to speed up developer time may make some sense (or tabbing through CoPilot), but just as with the early days of higher level programming languages, expect to come across cases where it doesn't speed up DX or introduces a bug. These bugs can be quite costly, I've seen GPT spit out encryption code and completely leave out critical parts like missing arguments to a library or generating the same nonce or salt value every execution. With code like this, if you're not well versed in the domain it is very easy to overlook, and unit tests would likely still pass. I think the same lesson told to young programmers should be used here -- don't copy/paste any code that you do not sufficiently understand. Also maybe avoid using this tool for critical pieces like security and reliability.
  • itsoktocry 3y ago
    >If you want to get back into front-end read “CSS: The Definitive Guide”. Great book, gives you a complete understanding of CSS by the end. Do you realize for how many technologies you can say the same thing? I don't want to read a 600 page tome on CSS. The language is a drop in the bucket of useful things to know. How valuable is a "complete understanding" of CSS? I just want something on my site to look a specific way.
    • miiiiiike 3y ago
      > I don't want to read a 600 page tome on CSS. The latest edition is closer to 1,100 pages. It’s worth it. It’s usually worth it for any long-standing technology you’re going to spending a significant amount of time using. Over the years you’ll save time because you’ll be able to get to the right answer in less time, debug faster, and won’t always be pulling out your hair Googling.
• sublinear 3y ago
  [flagged]
  • throwbadubadu 3y ago
    Calling bullshit is maybe too harsh. There may be requirements matching the available training data and the right mood the LLM has been tuned for where it delivers acceptable, then considered flawless, results (extreme example: just try "create me hello world in language x" will mostly deliver flawless)... and by that amateurs (not judging, just mean less exposed to variety of problems and challenges) may end up with the feeling that LLMs could do it all. But yes, any "serious" programmer working on harder problems can quickly derail an LLM and prove otherwise, with dozens of his simple problems each day (tried it *). It doesn't even need that, one can e.g. prove ChatGPT (also 4) quickly wrong and going in wrong circles on C++ language questions :D, though C++ is still hard, one can also do the same with questions on the not-ultracommon Python libs. It confidently outputs bullshit quick. (*): Still can be helpful for templating, ideas, or getting into the direction or alternatives, no doubts on that!
• lucideer 3y ago
  > I recently asked ChatGPT to produce similar CSS and it did so flawlessly. I use ChatGPT every day for many tasks in my work and find it very helpful, but I simply do not believe this. > The article points this out: middling generalists can now compete with specialists. I'd say it might allow novices to compete with middling generalists, but even that is a stretch. On the contrary, ChatGPT is actually best suited to use by a specialist who has enough contextual knowledge to construct targeted prompts & can then verify & edit the responses into something optimal.
  • tlarkworthy 3y ago
    I wrote just the tool to optimize AI in the hand of a coding expert https://observablehq.com/@tomlarkworthy/robocoop https://observablehq.com/@tomlarkworthy/robocoop
  • jorams 3y ago
    I agree with this. There are cases where it produces good results, but there are also cases where it produces bs, and it's not always obvious. I find it to work fine for cases where I know what I want but could use a starting point, but it often invents or misunderstands all kinds of things. The most frustrating situations are those where it invents a function that would miraculously do what's necessary, I tell it that function does not exist, it apologizes, shuffles the code around a bit and invents a different function, etc. It's the most annoying kind of debugging there is.
    • lucideer 3y ago
      Another very obvious thing it does when it comes to code is take the most common misconceptions & anti-patterns used within the programming community & repeats them in an environment where there's no-one to comment. People have critiqued Stack Overflow for having so many "wrong" answers with green checkmarks, but at least those threads have surrounding context & discussion. A case in point: I asked ChatGPT to give me some code for password complexity validation. It gave me perfectly working code that took a password and validated it against X metrics. Obviously the metrics are garbage, but the code works, and what inexperienced developer would be any the wiser. The only way to get ChatGPT to generate something "correct" there would be to tell it algorithmically what you want (e.g. "give me a function measuring information entropy of inputs", etc.) - you could ask it 50 times for a password validator: every one may execute successfully & produce a desired UI output for a web designer, but be effectively nonsense.
• rixed 3y ago
  Out of curiosity, how did you pass the wireframe to chatGPT ?
  • Cthulhu_ 3y ago
    Doesn't ChatGPT support image uploads these days?
    • frabcus 3y ago
      Yes, but the paid-for Plus version only. Ignore the free version, pretend it doesn't exist.
  • ern 3y ago
    I described what I wanted. It was earlier this year..not sure if chatgpt can understand wireframes now, but it couldn’t at the time.
    • rixed 3y ago
      you described it with pixel accuracy ?
• Cthulhu_ 3y ago
  > middling generalists can now compete with specialists. I want to say that this has been the state of a lot of software development for a while now, but then, the problems that need to be solved don't require specialism, they require people to add a field to a database or to write a new SQL query to hook up to a REST API. It's not specialist work anymore, but it requires attention and meticulousness.
• janosdebugs 3y ago
  > middling generalists can now compete with specialists. They can maybe compete in areas where there has been a lot of public discussion about a topic, but even that is debatable as there are other tasks than simply producing code (e.g. debugging existing stuff). In areas where there's close to no public discourse, ChatGPT and other coding assistance tools fail miserably.
  • red-iron-pine 3y ago
    this be the answer. GPT is as good as the dataset it's trained off of, and if you're going by the combined wisdom of StackOverflow then you're going to have a middling time.
• pjc50 3y ago
  I would really like to see the prompts for some of these. Mostly because I'm an old-school desktop developer who is very unfamiliar with modern frontend.
• gumballindie 3y ago
  But if you are a middling programmer when it comes to CSS how do you know the output was “flawless” and close to the quality that css “masters” produce?
  • ern 3y ago
    It looked correct visually and it matched the techniques in the actual CSS that the team lead and I produced when we paired to get my layout to the standard he expected.
• crabbone 3y ago
  It probably depends on the area. CSS is very popular on one hand and limited to a very small set of problems on the other. I did try asking ChatGPT about system-related stuff several times and had given up since then. The answers are worthless if not wrong, unless the questions are trivial. ChatGPT works if it needs to answer a question that was already answered before. If you are facing a genuinely new problem, then it's just a waste of time.
• barrkel 3y ago
  ChatGPT goes from zero to maybe 65th percentile? There or thereabouts. It's excellent if you know nothing. It's mediocre and super buggy if you're an expert. A big difference is that the expert asks different questions, off in the tails of the distribution, and that's where these LLMs are no good. If you want a canonical example of something, the median pattern, it's great. As the ask heads out of the input data distribution the generalization ability is weak. Generative AI is good at interpolation and translation, it is not good with novelty. (Expert and know-nothing context dependent here.) One example: I use ChatGPT frequently to create Ruby scripts for this and that in personal projects. Frequently they need to call out other tools. ChatGPT 4 consistently fails to properly (and safely!) quote arguments. It loves the single-argument version of system which uses the shell. When you ask it to consider quoting arguments, it starts inserting escaped quotes, which is still unsafe (what if the interpolated variable contains a quote in its name). If you keep pushing, it might pull out Shell.escape or whatever it is. I assume it reproduces the basic bugs that the median example code on the internet does. And 99% of everything being crap, that stuff is pretty low quality, only to be used as an inspiration or a clue as to how to approach something.
  • FrustratedMonky 3y ago
    I encountered this with particular problem in python. Seemed like GPT wanted to always answer with something that had a lot of examples on the web, even if most answers were not correct. So garbage in, garbage out problem. I'm bit worried that the LLM's will continue to degrade as the web has increasing amount of LLM generated content. Seems to already be occurring.
    • fkyoureadthedoc 3y ago
      Why do people that hand wave away the entire concept of LLMs because of one instance of it doing one thing poorly that they could do better, and yet always seem to fail to just show us their concrete example?
      • barrkel 3y ago
        FWIW I didn't wave away the entire concept. LLMs definitely have uses.
• YeGoblynQueenne 3y ago
  >> The article points this out: middling generalists can now compete with specialists. They can't, and aren't even trying to. It's OpenAI that's competing with the specialists. If the specialists go out of business, the middling generalists obviously aren't going to survive either so in the long term it is not in the interest of the "middling generalists" to use ChatGPT for code generation. What is in their interest is to become expert specialists and write better code both than ChatGPT currently can, and than "middling generalists". That's how you compete with specialists, by becoming a specialist yourself. Speaking as a specialist occupying a very, very er special niche, at that.
  • elif 3y ago
    It REALLY depends on the task. For instance, if you provide GPT with a schema, it can produce a complex and efficient SQL query in <1% of the time an expert could. I would also argue that not only are the models improving, we have less than a year practically interfacing with LLM's. OUR ability to communicate with them is in infancy, and a generation that is raised speaking with them will be more fluent and able to navigate some of the clear pitfalls better than we can.
    • emporas 3y ago
      There is not much of a need for humans to get closer to the machine long term, when with new datasets for training the machine will get closer to humans. Magic keywords like "step by step" won't be as necessary to know. One obstacle for interfacing with LLM's is the magic cryptic commands it executes internally, but that need not be the case in the future.
• kypro 3y ago
  You may think it did a good job because of your limited CSS ability. I'd be amazed if ChatGPT can create pixel-perfect animations and transitions along with reusable clean CSS code which supports all of the browser requirements at your org. I've seen the similar claims made on Twitter by people with zero programming ability claiming they've used ChatGPT to build an app. Although 99% of the time what they've actually created is some basic boilerplate react app. > middling generalists can now compete with specialists. Middling generalists can now compete with individuals with a basic understanding assuming they don't need to verify anything that they've produced.
  • rpmisms 3y ago
    It does great at boilerplate, so I think it's safe to say it will disrupt Java. I've been using tabnine for years now, and I use chatGPT the same way; write my boilerplate, let me think about logic.
  • nisegami 3y ago
    >I'd be amazed if ChatGPT can create pixel-perfect animations and transitions along with reusable clean CSS code which supports all of the browser requirements at your org. Personally, I'd be more amazed if a person could do that than if a LLM could do it.
    • kypro 3y ago
      Google, "UI developer".
  • jameshart 3y ago
    Here’s the thing though: If a new version of the app can be generated on the fly in minutes, why would we need to worry about reusability? GPT generated software can be disposable. Why even check the source code in to git - the original source artifact is the prompt after all.
    • philipwhiuk 3y ago
      Let me know how you get on with a disposable financial system, safety system or electoral voting system.
• scythe 3y ago
  I suspect that the "depth" of most CSS code is significantly shallower than what gets written in general purpose programming languages. In CSS you often align this box, then align that box, and so forth. A lot of the complexity in extant CSS comes from human beings attempting to avoid excessive repetition and typing. And this is particularly true when we consider the simple and generic CSS tasks that many people in this thread have touted GPT for performing. There are exceptions where someone builds something really unique in CSS, but that isn't what most people are asking from GPT. But the good news is that "simple generic CSS" is the kind of thing that most good programmers consider to be essentially busywork, and they won't miss doing it.
• calvinmorrison 3y ago
  > being pixel perfect was a source of pride. Then use LaTex and PDF. CSS is not for designing pixel perfect documents.
• ryandvm 3y ago
  > middling generalists can now compete with specialists Great point. That's been my experience as well. I'm a generalist and ChatGPT can bring me up to speed on the idiomatic way to use almost any framework - provided it's been talked about online. I use it to spit out simple scripts and code all day, but at this point it's not creating entire back-end services without weird mistakes or lots of hand holding. That said, the state of the art is absolutely amazing when you consider that a year ago the best AIs on the market were Google or Siri telling me "I'm sorry I don't have any information about that" on 50% of my voice queries.
• philipwhiuk 3y ago
  Would you give the critical public safety system bit to ChatGPT? This scenario reminds me of: If a job's worth doing, do it yourself. If it's not worth doing, give it to Rimmer. Except now it's "give it to ChatGPT"
• city41 3y ago
  I'm a developer but also have an art degree and an art background. I'm very mediocre at art and design. But lately I've been using AI to help plug that gap a bit. I really think it will be possible for me to make an entire game where I do the code, and AI plus my mediocre art skills get the art side across the line. I think at least in the short term, this is where AI's power will lie. Augmentation, not replacement.
• hanzmanner 3y ago
  > It had zero business value, but such was the state of our team…being pixel perfect was a source of pride UX and UI are not some secondary concerns that engineers should dismiss as an annoying "state of our team" nuance. If you can't produce a high quality outcome you either don't have the skills or don't have the right mindset for the job.
• sonicanatidae 3y ago
  AI is a tool. Like all tools, it can be useful, when applied the right way, to the right circumstances. I use it to write powershell scripts, then just clean them up, and voila. That being said, humans watch too much tv/movies. ;)
• number6 3y ago
  I might be a bit out of the loop: how did you do it? I thought ChatGPT is text based?
• itsoktocry 3y ago
  >The article points this out: middling generalists can now compete with specialists. This is why you're going to get a ton of gatekeepers asking you to leetcode a bunch of obscure stuff with zero value to business, all to prove you're a "real coder". Like the OP.