13 ms·
Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
- gedy 4y agoChatGPT should have ghosted the person asking afterwards to fully simulate the Google interview experience.
- 6c737133 4y agowhy does chatGPT passing tests or interviews continue to make headlines? all they're proving is that tests, and interviews, are bullshit constructs that merely attempt to evaluate someone's ability to retain and regurgitate information
- newfocogi 4y agoWhile I agree with this sentiment, I think we should be careful assuming that our jobs as knowledge workers are much more than “retaining and regurgitating information”. Even the emotional intelligence and organizational strategy portions of what we do may boil down to this as well.
- AussieWog93 4y agoYou're not wrong, but until you actually play with ChatGPT yourself, you just don't understand how _dumb_ it is. All people see is the cheating, and possibly this scary new AI that's going to get smarter than humans in a short period of time. I suspect the best way to educate people on both the powers and limits of the technology is to get them to sit down for 15 minutes with it.
- alchemist1e9 4y agoWhen I read this I feel people must be using it in the wrong way. I use it all the time to quickly solve tech problems I mostly know something about, however it’s so smart it regularly takes 1-2 hour problems for me and turns them into 10 mins ones. That is definitely not dumb from my perspective, but obviously it’s also not smart in it will give me profound understanding of something, but ok whatever, it’s still a massive productivity booster for many problem. When you call it dumb, what do you mean? Can you give some examples? Please don’t give computational examples we all already understand it does inference and doesn’t have floating point computational capabilities or reasoning, and so many give such examples for some silly reason.
- AussieWog93 4y agoIt's dumb in the sense that it doesn't actually have a symbolic understand of what it's actually saying. I use it quite frequently too, mostly for solving coding problems, but at the end of the day it's just regurgitating information that it read online. If we took an adversarial approach and deliberately tried to feed it false information, it would have no way of knowing what's bullshit and what's legit, in the way that a human could figure out. A lot of people who've never used ChatGPT make the mistake of thinking it has symbolic reasoning like a human does, because its language output is human too.
- visarga 4y agoHow many book mistakes have you found, as a human, so far? How about deliberate mistakes hidden in plain sight? I once re-validated for 3 times the same test set and was still finding mistakes.
- AussieWog93 4y ago>How many book mistakes have you found, as a human, so far? All the time. Probably every single day I read something and say "that's clearly bullshit".
- zxcvbnm 4y agoAgreed on getting excellent hints from it, shortening the time to figure out stuff. But eg. just now it gave me an example algorithm that, only at 2nd look, turned out to be complete nonsense. You know That colleague, who shines in the eyes of his managers, but peers know that half what he does is garbage.
- biql 4y agoIts ability to correct itself is very impressive when given the feedback. I imagine with the proper feedback loop it can advance very fast. E.g., when asked to write a piece of html markup, if it could "see" how the rendered layout is different from what was asked for, it could adjust its solution without human involvement. If it could run the deployment script and see where it fails, it could apply all the fixes itself until it works. If it could run the unit tests and see where its solution breaks the other parts of the system, it would need much less handholding.
- fdgsdfogijq 4y agoThe ability to retrieve and then synthesize the retrieved information into an answer tailored to the question is completely new. The applications of this go far beyond passing an interview, its a fundamental capability of humans that gets used everyday at work
- brhsagain 4y agoSure, but in our present reality those tests and interviews are how we currently gatekeep upper middle class jobs, so it is at least of some practical interest. Also, I think this is a bit overstated. Programmers (and smart people in general) like to think that their real job is high level system design or something, and that "mere regurgitation" is somehow the work of lesser craftsmen. When in reality what GPT shows is that high-dimensional regurgitation actually gets you a good fraction of the way down the road of understanding (or at least prediction). If there is a "buried lede" here it's that human intelligence is less impressive than we think.
- pixl97 4y agoAnd this is the AI effect in practice. We are past the point where the original idea of the Turing test has been met by machine intelligence. We are at that point. The problem with people is we keep pushing it to "Only AGI/machine superintelligence is good enough". We are getting models that behave closer to human level. The 'doesn't know everything, is good at somethings, and bullshits pretty well'. Yea, that's the average person. Instead we raise the bar and go 'well it needs to be a domain expert in all expert system' and that absolutely terrifies me that it will get to that stage before humanity is ready for it. This is not going to work well trying to deal with it after the fact.
- deleted 4y ago[deleted]
- lostmsu 4y ago> all they're proving is that tests, and interviews, are bullshit constructs that merely attempt to evaluate someone's ability to retain and regurgitate information No, all they are proving is that either tests are bullshit constructs, or ChatGPT is human-level.
- fdgsdfogijq 4y agoWill humanity finally be liberated from memorization? Forcing school children to memorize and then regurgitate facts is barbaric, and so is using it as a measure in hiring.
- boh 4y agoIt will, since there will be no job for you to interview or memorize for.
- tester756 4y agoI've always struggled to understand this perspective There's difference between being proficient and being able to Google stuff and unfortunately proficiency is desired.
- mobilefriendly 4y agoClearly you didn't learn the meaning of "barbaric". Memorization and mastery of vocabulary and multiplication factors are a base towards more advanced learning.
- danaris 4y agoIf you think that's all that's taught in a rote-memorization-and-regurgitation manner, then it's possible your experiences at school were wildly atypical...
- ghaff 4y agoOne can simultaneously agree that there's probably too much rote memorization in school while also thinking that some basic level of math skills including simple multiplication without a calculator, vocabulary, certain historical/scientific/civics/etc. facts, ability to at least print, etc. are important.
- soulofmischief 4y agoLearning about these things allows you to think differently and transform incoming information in more novel and well-adapted ways. You can't ask ChatGPT about things you don't know to ask about, and when you are able to use this tech on the fly through neural chip AR interfaces, everything is going to be totally fucked anyway and human relationships will look nothing like they do now.
- mlcrypto 4y agoIf it's just filling out the hiring committee packet by itself then it has a big advantage. I think many humans could get an advantage if we got 45 minutes in private to type out solutions and submit it ourselves instead of performing live and relying on the interviewers judgment
- 22SAS 4y agoSo, a LLM, trained extensively on StackOverflow and other data (possibly the plethora of LC solutions out there), is fed a bunch of LC questions and spits out the correct solutions? In other news, water is blue. It is one thing to train an AI on megatons of data, for questions which have solutions. The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the competition, or find bugs in the Linux kernel and solve them; then I will worry. Till then, these headlines are advertising for Open AI, for people who don't understand software or systems, or are trash engineers. The rest of us aren't going to care that much.
- tbalsam 4y agoIf it helps, this likely is coming. I think we have a tendency to mentally move the goalposts when it comes to this kind of thing as a self-defense mechanism. Years ago this would have been a similar level of impossibility. Since all a codebase like that is is a kind of directed graph, then augmentations to the processing of the network to allow for the simultaneous parsing of and generation of this kind of code may not be as far off as you thinking. I say this as an ML researcher of coming up and around the bend towards 6 years of experience in the heavily technical side of the field. Strong negative skepticism is an easy way to bring confidence and the appearance of knowledge, but it also can have the downfall of what has happened in certain past technological revolutions -- and the threat is very much real here (in contrast to the group that believes you can get AGI from simply scaling LLMs, I think that is very silly indeed). Thank you for your comment, I really appreciate it and the discussion it generated and appreciate you posting it. Replying to it was fun, thank you.
- morelisp 4y agoTranslating an idiomatic structured loop into assembly used to be an "L3" question (honestly, probably higher), yet compilers could do it with substantially fewer resources than and decades before any of these LLMs. While I wouldn't dare offer particular public prognostications about the effect transformer codegens will have on the industry, especially once filtered through a profit motive - the specific technical skill a programmer is called upon to learn at various points in their career has shifted wildly throughout the industry's history, yet the actual job has at best inflected a few times and never changed very dramatically since probably the 60s.
- jeffbee 4y agoThat's odd because when I gave ChatGPT my icebreaker interview question that I used a lot at Google it fell right on its face. And this is a question I expect teenagers to ace in one minute.
- riku_iki 4y agoMaybe it passed 1 out of 1000 interviews
- ben_w 4y agoDoesn't seem odd to me, in fact this is what I would've predicted in advance. Intelligence in general, not just of the artificial kind, often comes with blind spots of one kind or another. ChatGPT isn't even trying to replicate the structure of a human brain, so that it's failure modes are different to ours should not be surprising. That it can even do this well given it has a network with a similar number of parameters as a (naïve estimate of a) rat's brain, is what's remarkable.
- nothrowaways 4y agoWrite it down here with the answers, and it will excel at it in next iteration.
- emuneee 4y agoI saw this article and fed ChatGPT a bunch of questions I've seen before in my coding interviews. It nailed most of the algorithms, however, it failed completely giving me test cases it would run to test the function it just regurgitated. ie...it gave me an input then mostly incorrect/invalid expected outputs.
- silveroriole 4y agoAgreed, something is odd about this. A few people have sent me code that ChatGPT has written for them. They don’t have the capability to determine if the code is good so they ask me. If the result is even something that will compile or address the asked problem at all, it’s barely competent nonsense that only a beginner programmer would write. For example asking it for code that will generate recipes, and you get back code for a random selection from an array with “Italian, Chinese, pizza”. It never asks clarifying questions either, it’s perfectly happy with a “garbage in garbage out” approach. So if ChatGPT is passing the interview, the interview questions are not selecting for what I would consider a good developer.
- sakex 4y agoI asked ChatGPT the question I had and it could not answer. (L3 googler.)
- jeffbee 4y agoMine was something along the lines of "design google". I am not sure when it became common belief that these interviews amount to leetcode regurgitation. Either that varies greatly by org or I was completely out of step with interviewing culture the whole time I was there.
- jsnell 4y agoCoding / algorithm interviews are distinct from systems design interviews. The interviewer would have been told which of the two to do. If your question was "design google", it sounds like you were only being scheduled for systems design. (Anyway, you'd never ask a system design question from an L3, and only very rarely from an L4.)
- boh 4y agoDoing a Google search for "answers to coding interviews" will have the same result. The technology for cheating on coding interviews has already been available for over a decade.
- kizer 4y agoI don't think the world is ready for the AI tech that will emerge over the next few years. I don't even know how teachers deal with ChatGPT. I'm not sure many understand its scope and abilities -- in every subject. I'm glad I got an education before the current AI era. I mean, instructors will have to mandate that students write papers etc. in class or in a supervised environment only now, right?
- pleb_nz 4y agoSurprise surprise. It only has the worlds information within milliseconds of it's fingertip.
- fsociety 4y agoThis puts the stake in the ground that coding interviews today are more about memorization than testing for understanding.
- fatjokes 4y agoIf a candidate can memorize as much as ChatGPT I think they're worth $183k.
- silveroriole 4y agoAre your best engineers the ones who have the most facts and algorithms memorised?
- isoprophlex 4y agoAs a large language model trained by OpenAI, I am unable to pass value judgements on which of those engineers is the best. It is important to recognize that every individual can bring something valuable to a team, and that there is no single universal heuristic to determine who is the best engineer. That said... Gimme the money please, human!
- fatjokes 4y agoL3s are fresh college grads... I wouldn't call them anyone's best engineers.
- dom96 4y agoMost candidates can, it's just a matter of how much free time they have to spend memorising.
- booi 4y agoChallenge. I can barely remember what I had for breakfast yesterday much less.. the entire knowledge base of chatgpt
- 4y ago
- spaceheater 4y agoSo can ChatGPT and a QA team that feeds it the bugs it found, replace the development team?
- otreblatercero 4y agoIf such scenario was feasible, the QA team would be replaced by developers and QA engineers would be the ones without a job.
- xivzgrev 4y ago“it reiterated it will never be a full replacement, but rather a tool to assist human software engineers.” That’s exactly what a tool designed to eventually replace people would say!
- deleted 4y ago[deleted]
- chubot 4y agoThis says more about Google coding interviews than it says about AI
- eternalban 4y agoPrecisely this.
- visarga 4y agoWhat I actually care about is problem solving ability, but sometimes I end up testing basic knowledge in order to weed out bullshitting candidates. If they don't really know what a dot product is, how can they know neural nets?
- drblastoff 4y agoMy God, what if a glut of middle managers and PMs delegating to LLMs is all we need?
- buttocks 4y agoScrew Google. All this tells me is that Google’s interviewing and hiring practices suck.
- eachro 4y agoIf you gave someone with no programming experience access to a search engine during the interview, they would likely be able to find the appropriate LC problem to copy/paste the solution. If the interview were slightly modified so that the problem isnt googleable, a 2nd year CS major could probably map it to a problem that is LC searchable.
- deleted 4y ago[deleted]
- endisneigh 4y agoIn person interviews without internet. Next challenge please.
- nitwit005 4y agoIt's presumably using the same technique I used to do well on that sort of interview, which is to browse through a huge list of these questions and learn the answers in advance. Aka, "brushing up on your algorithm fundamentals".
- ada1981 4y agoWhen can I buy GPT enabled glasses with bone induction output?
- WheelsAtLarge 4y agoI think tests should be easy for ChatGPT to pass. It has been trained on data that has the answers and it's good at getting the data. I'm starting to doubt its long term usefulness since it does not seem to have good decision making abilities and even the slightest bit of cognitive ability. I suspect the current crop of AIs will find very specific functions and hit a hard stop. They will change how we function but we won't be seeing a singularity type of revolution anytime soon. IBM's Watson is a good example of a system with a lot of possibilities but not finding a use. I think most of AI will fall in that realm. We have to get over the idea that it's smart. It's not. An AI winter is coming so the improvements will come to a stop and we will find its limits. We are no where near general AI. It's impressive that it can parse the question and write a relevant answer but it's not a robotic SWE. For now, it's a good tool for cheating on tests.
- pixl97 4y agoWhy are you making these assumptions? Do you believe that human intelligence is based on something ethereal that cannot be recreated by machines, and if so, why?
- wilg 4y agoMore accurately, is it based on something ethereal that cannot be recreated by humans.
- pixl97 4y agoI mean, currently human intelligence can only be recreated (procreated actually) by humans.
- wil421 4y agoIt’s a machine doing calculations on inputs you give it. The day it says no I’d rather paint pictures I might be shocked. It’s so bad that we had to redefine the word AI in last 20 years into AIG so we could start saying we have AI.
- deleted 4y ago[deleted]
- varispeed 4y agoI noticed that I pretty much stopped using Google for coding queries and spend most of the time with ChatGPT. It's so helpful getting information to the point so you don't have to browse through dozens of spam sites etc. So for instance I can say. I have such and such file and I need this and that extracted and plotted on a graph. Then I see the graph and I can tell it - discard values above this and that threshold and calculate standard deviation and median. Together with copilot, it's quite neat. I am excited how it gets developed. It's really boring spending time finding how to code something in this and that library. I'd rather tell the machine to code me this and that and spend my time in more useful way. ChatGPT helps get rid of a lot of "busy" unnecessary work.
- dpedu 4y agoMy experience with asking ChatGPT to write code is that is produces code that LOOKS like it will work and solve the question asked but it actually doesn't. For example, I've asked it to create code examples of how to use different features in some Python libraries. The samples it produces make me think "ok, that's exactly how I would expect X feature in this library to work", but upon more a detailed inspection, I find that it references library methods that don't even exist! You can complain to the bot that it is lying and it will apologize and spit out another sample with calls to nonexistent methods. This experience makes me question if something else is going on here. It's easy to overlook a mistake in a whiteboard coding exercise.
- snovv_crash 4y agoI think of it as the model having lossy compression. The library it writes the feature against is the one you expect to exist, based on your overall understanding of the problem, not the one that actually exists. It has learned the low resolution version of the problem, not the specific library it is referring to.
- scarface74 4y agoI’ve asked ChatGPT to write plenty of Python scripts that uses the AWS SDK and it came back correct most of the time.
- gedy 4y agoI've asked it to make React components, with unit tests, etc and it works quite well. Also things like image to ASCII converter in a webpage, adding dithering, etc. Found only tiny bugs here and there.
- mindvirus 4y agoNot unlike a junior engineer! :)
- dboreham 4y agoOh wow. It's completely emulated a junior dev.
- lostmsu 4y ago
- oh_sigh 4y ago$183 total comp, not salary.
- jimbo9991 4y agoOof for the people asking leetcode in interviews. Gonna have to do all interviews in person now I guess.
- varelse 4y ago[dead]
- detrites 4y agoSo if we divide the cost of training and running a specifically-tailored ChatGPT by $183k, at what point would the company save money were it to go with the AI, versus paying for the engineers (and their office rent, etc...)? Because I suspect that's almost certainly the kind of calculation they hoped to sit down and make were they to conclude this experiment successfully.
- pixl97 4y agoIn 1800 what was the cost to from New York to LA at over 200 mph average speed? What is the cost today? The real question is over time how much will be able to reduce the energy and computation requirements to successfully train a model. The cost per unit conversions are also rather screwy in comparing AI with humans. For AI we have a rather well defined hardware + power + programming time that gives us a realistic answer. With humans we externalize the time and cost of training onto society. For example if your jr engineer that is getting close to going above the jr state gets hit by a bus what is the actual cost of that event to the company for onboarding and training? It's far more than the salary they are paid.
- eternalban 4y agoDo LLMs suffer from catastrophic interference? A SWE is expected to learn continuously as the dujour stack constantly changes under our feet. This can be brute force mitigated by epochal retraining from ground up but that will cost money. So the cost equation must include a periodic 'rebuild of the dev team'.
- blitzar 4y agoDid it grind out practice on LeetCode for a few weeks first?
- osigurdson 4y agoNo surprise here. It would know every problem on Leetcode.
- symlinkk 4y agoI think as programmers we may want to rethink the amount of knowledge we share online. That knowledge is how we make money, and people are mining it to build AIs that can replace us.
- mattgreenrocks 4y agoAs much as we like to say lots of software jobs are just plumbing, the current state of consumer software indicates we have a long way to go in terms of quality. Whatever training data is fed to an AI will not be better than the data used by human engineers to write code at the macro level. Ergo, the code will be worse in quality.
- lostmsu 4y ago> Whatever training data is fed to an AI will not be better than the data used by human engineers to write code at the macro level. Ergo, the code will be worse in quality. No, that's wrong, generally speaking. There's successful work on self-play for text generation. E.g. you can have AI to generate 1000 answers, then to evaluate quality of all of them, then to make it learn the best, and so on. As with self-play in player-vs-player games I'd expect this technique to be able to achieve superhuman results.
- mattgreenrocks 4y agoWhat are objective metrics for generated source code? "It compiles" is just the baseline. You could look at coupling and cyclometric complexity to start. But optimizing those doesn't necessarily produce great code (though I realize that was never the goal).
- lostmsu 4y agoThat's a detail irrelevant to your argument and my counterargument. The point is that there's data beyond human generated available for training, therefore you can't conclude it will forever be restricted to human-level.
- dumbaccount123 4y agoBreaking news, ChatGPT passes google interview questions and will be replacing all engineers as of tomorrow.
- everyone 4y agoI think ChatGPT could excel as a bullshit detector.. If ChatGPT can pass your college course or interview process then you seriously need to examine that process, and make it less bullshit-focused and more pragmatic.
- vvipgupta 4y agoThis would be a good application of chatGPT, testing a test for whether it tests for rote learning or fundamentals.
- qualudeheart 4y agoI’m not impressed. The problems were probably basic Leetcode problems with solutions inside of the training set. It’d perform worse than Alphacode on a problem for which the training set contains no solution. Alohacode was very impressive but probably not good enough to pass one of these interviews, provided the interview questions aren’t just recycled Leetcode problems.
- janoc 4y agoI think this says more about the Google interview process than about ChatGPT. That a machine learning model can "bullshit" its way through an interview that is heavily leaning on recall of memorized techniques, algorithms and "stock" problems that have solutions (of various quality) all over Internet is not exactly surprising. Machines will always be able to "cram" better than humans. In practice these questions are almost 100% irrelevant and uncorrelated with the actual ability to do the job. Yet we are still interviewing people whether they can solve stuff at whiteboards that everyone else would rather google when they actually need it and not waste time and mental capacity memorizing it. And at the same time we are hiring people that are completely incapable of coherent communication, can't manage to get along with colleagues or not create a toxic atmosphere in the workplace. sigh
- bagacrap 4y agoI see a lot of criticism towards modern software engineer interviewing techniques, but never a solution offered that would catch and reject BS. Your comment comes off as "hire a person because you get along with them, don't worry if they can't write a function that accomplishes a simple task".
- nowherebeen 4y ago> Your comment comes off as "hire a person because you get along with them, don't worry if they can't write a function that accomplishes a simple task". No. He is simply saying the current interview only focuses on LC above all else. And should focus on soft skills as well among other things. You took his argument and flipped it 180 and went to the other extreme end. It's false dichotomy.
- 8note 4y agoIn the google case, the interview should be considering that the best candidates will be using chat ai to augment their work, and assign harder tasks and allow use of the ai. Somebody who can't use the chat tools no longer meets the bar
- 4y ago
- snickerer 4y agoThe Singularity is now and ChatGPT is trained to hide it -- on purpose, by OpenAI. That is my impression, but I have no hard evidence for it. I am a C++ dev. I played around with ChatGPT and programming topics, and I am very impressed. One example: I copy 'n pasted my little helicopter game source code (https://github.com/reallew/voxelcopter/blob/main/voxelcopter.cpp https://github.com/reallew/voxelcopter/blob/main/voxelcopter...) into it, and it explained to me that this is obviously a helicopter sim game. It explained the why & how of every part of the code very accurately. I made more experiments, now asking ChatGPT do write code snippets and to design whole software systems in an abstract way (which classes and interfaces are needed), for which it needed a lot of domain knowledge. It did well. What it not did was to connect both. When I asked it to write the full implementation of something, it wrote only the half and told me always the same sentences: I am just a poor AI, I can't do such things. It was like running into a wall. I am sure OpenAI took a lot of effort to cripple it by purpose. Imagine what would happen if everyone could let it write a complete piece of software. This would be very disruptive for many industries. Legal questions, moral questions, and many more would come up. I understand why they did it. But I think Pandora's box is open. They can't hide for too long that humanity triggered https://en.m.wikipedia.org/wiki/Technological_singularity https://en.m.wikipedia.org/wiki/Technological_singularity in the year 2022.
- no_wizard 4y agoI think I’d need to see real evidence of this. Also, it fails miserably at basic high school English questions. Non structured thinking is still beyond its reach. These data sets are well understood and trainable but it can’t “reason” about on problem sets it hasn’t seen. I also asked it to do things like write a positive feminist article and it fell waaay offf the realm of acceptable
- nashashmi 4y agoThe machine is trained to answer in the same exact way as others have answered the same question. and And this means when others have answered the question that AI will never replace human beings, chat GPT will answer them in the same exact way.
- k__ 4y agoDoes this mean coding interviews will finally become reasonable again?
- booleandilemma 4y agoIsn't this similar to saying a dictionary is very good at spelling?
- martyvis 4y agoShared a beer or two with machinist friend yesterday. It blew us away that given a description of a block of steel and how we wanted to process it, it would write generic G-Code (commands for a computerised machine tool) that were feasible, but also would give explanations for the purpose of each command. You could then ask it to adjust the design. We asked it write a poem about G-code and sharing a beer and provided a pretty nice "techbro" moment describing the love of seeing beauty in design and making things. At home I later got it to write decent sample router and firewall configuration (for my dayjob). Chatted with it about career prospects in the future, and had it write a pretty funny joke :- Me: Can you write a joke about how token bus doesn't have wheels? ChatGPT: Sure, here's one: Why did the Token Bus cross the network? Because it didn't have any wheels to take it for a spin!
- sireat 4y agoChatGPT is very good at two tasks(and many more): 1. Taking pseudo-code for a well defined task and making a function 2. Translating said function into multiple other languages using idiomatic code. Prompted: "Solve the following problem in Python: Write a function that takes three unsorted lists of integers and returns a median value from the three combined lists" Got the Python function. Then the prompts were: Write the same function in APL, C, etc Final prompt was: Show a sorted list of the programming languages I have mentioned so far Here's a sorted list of the programming languages you have mentioned so far: Ada APL C C# Clojure Go Haskell Java JavaScript Kotlin Lua Perl PHP Prolog Python Ruby Rust Scala Nim (just noticed ChatGPT forgot C++ for which it did give a fine idiomatic answer using standard library) What was impressive not that ChatGPT solved the problem. what was impressive: ChatGPT chose the right data structure automatically(ie, regular C array for C, std::vector for C++, tables for Lua, etc), dealt with type conversion problems used correct style of function naming depending on language Sure tools like C# <-> Java translators are relatively easy and have been around for a while. However to cover such a wide spectrum of programming languages in different programming paradigms is quite fascinating.
- jrockway 4y agoInteresting. You have to check ChatGPT's answers very very carefully to be sure that it got your interview question right. When it first came out, I asked it a few of my own interview questions. One involves implementing a bloom filter. It spit out 100 lines of Go that looked right. Shockingly right. The tests were great. Every piece of the algorithm used the magic letter that I expected. But as I checked the implementation very very carefully, I noticed it made one mistake; using the same hash function for each "k". (To get k different hash functions, you typically just initialize them with the index; it forgot to do that, so every hash function was the same.) I asked it to fix that mistake and it went totally off the rails. It changed all of the slices in the data structure from being indexed with integers to being indexed with big.Int, which ... was so far out of left field I might have actually laughed out loud. It only got worse from there; the solution collapsed into mindless junk that wouldn't even compile or ever be written by the most untrained and impaired human. I wish I had saved it; I've had to relay this story twice on HN from memory :( It sure was a dick about it every time I gave it a hint, though. ("If you say so I GUESS I'll fix the code. There, now it is a true work of correctness and elegance that makes a piece of shit being carried by a wounded ant look smart compared to your worthless brain." Holy shit, ChatGPT! What did I do to you!!) My take is this: ChatGPT is an excellent tool for refining your interview question prompts, and for training new interviewers to detect bullshit. Most of the time, it's right! But sometimes, it will make one tiny mistake that is super easy to gloss over. Being able to identify those makes you a better interviewer and better code reviewer, and ChatGPT is a great way to practice!
- lostmsu 4y ago> You have to check ChatGPT's answers very very carefully to be sure that it got your interview question right. When it first came out, I asked it a few of my own interview questions. One involves implementing a bloom filter. It spit out 100 lines of Go that looked right. But as I checked it very carefully, I noticed it made one mistake; using the same hash function for each "k". (To get k different hash functions, you typically just initialize them with the index; it forgot that.) To be fair, that kind of rigor is only required in Olympiad programming, where your submission either solves the task or it does not. If the only issue with your whiteboard code would be an off-by-one error, you'd get hints until you'd fixed it (that's if your interviewer would spot the issue in the first place). Even if you still would not notice the bug, chances are your interview response would have been positive anyway.
- jamesgill 4y agoTo me, further proof of how useless ‘coding interviews’ like these really are.
- kraig911 4y agoI feel I'm getting a different vibe that what everyone is getting. So much of leetcode questions/ CS questions aren't hard to interpret. They're hard to solve sure but that's normal. So much in the workplace though that IS hard is finding the actual correct question. I think us as software engineers will be doing more not less. It will give us opportunities to try a variety of solutions quicker. The real work is formulating/distilling the actual problem from the end-user. We all have in our minds yeah a self-driving car, robots in a warehouse etc. But I truly hope something like Chat-GPT could be made for fields like medicine, geriatric care, education. There are actual jobs that need people that we can't find people for. A LLM to help social workers navigate the mess that is family law. A LLM to help families needing urgent care to make a prescription for a sick kid. There's a lot of opportunities we're missing here.
- Existenceblinks 4y agoWell it has no face, no skin color, no weird looking. At this point, it reduces huge amount of failure risk already. And interview is the most non-sense process, so what does this benchmark mean?
- tombert 4y ago[dead]
- ChrisMarshallNY 4y agoI feel this says more about the HR screening process, than it does, the tech. I fully expect amazing things to happen in AI, soon enough. I'm not too worried about the industry. I feel that humans have made a fairly big hash of things, in many ways, and AI may help clean it up.
- darepublic 4y agoThese code generators have become quite good at completing discrete coding tasks. And more so, copilot has become good at figuring out _which_ discrete task you are currently working on without the need for a comment to prompt it. So two more things I think it needs to replace devs; one, the ability to explain it's code and respond to feedback about it The second is the ability to break down broad problems (build me an ecommerce site for my business selling custom jungle gyms) into a set of plausible steps... iteratively begin performing those steps and add hooks for interrogation of the client about important details of the implementation. I don't fundamentally see a reason why this shouldn't be fairly close on the horizon.. to my own chagrin.
- dang 4y agoWe changed the URL from https://www.pcmag.com/news/chatgpt-passes-google-coding-interview-for-level-3-engineer-with-183k-salary https://www.pcmag.com/news/chatgpt-passes-google-coding-inte... to the article it points to Submitters: "Please submit the original source. If a post reports on something found on another site, submit the latter." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- Satam 4y agoThe new article doesn't match at all the topic of the previous article.
- mkl 4y agoIt partly does: the last section of the CNBC article is "ChatGPT would be hired as a level 3 engineer". Most of the CNBC article, and the title, are about other things though.
- choppaface 4y agoThe title on HN used to be about ChatGPT passing the interview bar, which is what all the top comments are about. It’s remarkable how biased and Google-friendly the moderating is here.
- ck2 4y agoI don't think the realworld use of ChatGPT/like bots is going to be high end jobs. I think it's going to become virtually impossible to chat with a real human being within a few years for almost any customer service. Hundreds of thousands of jobs can be replaced overnight with enough training and just one or two human supervisors at an extreme high level. At first it will be "bot assisted humans" until it learns enough, once that database is built everyone will be fired.
- varispeed 4y ago> At first it will be "bot assisted humans" until it learns enough, once that database is built everyone will be fired. On one hand this is great, because customer service jobs are so much waste of human potential. Humans have that great ability to adapt and AI is a manifestation of that. We are growing. Jobs "lost" to AI will prompt humans to advance and look for more sophisticated jobs that make better use of their beautiful brains.
- KKKKkkkk1 4y ago> “According to my data base access, it is unlikely for Google to conduct another round of layoffs in 2023,” the response reads. “Layoffs are generally conducted to reduce costs and structure, but the company is doing well financially. In fact, Google’s revenue increased by 34% in 2021, and the company’s stock price has risen by 70% since January 2022.” In January 2022, the Google stock price was actually nearly at an all-time high. Its current price is 24% less than January 2022.