22 ms·

I have some odd feelings about this. It took less than a year to go from "of course it isn't hooked up to the internet in any way, silly!" to "ok.... so we hoo

I have some odd feelings about this. It took less than a year to go from "of course it isn't hooked up to the internet in any way, silly!" to "ok.... so we hooked up up to the internet..." First is your API calls, then your chatgpt-jailbreak-turns-into-a-bank-DDOS-attack, then your "today it somehow executed several hundred thousand threads of a python script that made perfectly timed trades at 8:31AM on the NYSE which resulted in the largest single day drop since 1987..." You can go on about individual responsibility and all... users are still the users, right. But this is starting to feel like giving a loaded handgun to a group of chimpanzees. And OpenAI talks on and on about 'Safety' but all that 'Safety' means is "well, we didn't let anyone allow it to make jokes about fat or disabled people so we're good, right?!"

afterburner 4y ago
Yes but.... money
- deleted 4y ago
  [deleted]
Gam_ 4y ago
>"today it somehow executed several hundred thousand threads of a python script that made perfectly timed trades at 8:31AM on the NYSE which resulted in the largest single day drop since 1987..." this is hyperbolic nonsense/fantasy
- FredPret 4y ago
  Oh yes. It would of course have to happen after the market opens. 9:30 AM.
- meghan_rain 4y ago
  /remindme 5 years
- mk_stjames 4y ago
  Literally 6 months ago you couldn't get ChatGPT to call up details from a webpage or send any dat to a 3rd party API connected to the web in any way. Today you can. I don't think it is a stretch to think that in another 6 months there could be financial institutions giving API access to other institutions through ChatGPT, and all it takes it a stupid access control hole or bug and my above sentence could ring true. Look how simple and exploitable various access token breaches in various APIs have been in the last few years, or even simple stupid things like the aCropalypse "bug" (it wasn't even a bug, just someone making a bad change in the function call and thus misuse spreading without notice) from last week.
  - hooande 4y ago
    This has nothing to do with ChatGPT. An api end point will be just as vulnerable if it's called from any application. There's nothing special about an LLM interface that will make this more or less likely. It sounds like you're weaving science fiction ideas about AGI into your comment. There's no safety issue here unless you think that ChatGPT will use api access to pursue its own goals and intentions.
    - Jeff_Brown 4y ago
      They don't have to be actions toward its own goals. They just have to seem like the right things to say, where "right" is operationalized by an inscrutable neural network, and might be the results of, indeed, some science fiction it read that posited the scenario resembling the one it finds itself in. I'm not saying that particular disaster is likely, but if lots of people give power to something that can be neither trusted nor understood, it doesn't seem good.
    - IIAOPSW 4y ago
      I'm sure that with the right prompting, you can get it to very convincingly participate as a party in a contract negotiation or business haggling of some sort. It would be indistinguishable from an agent with its own goals and intentions. The thing about "it has no goals and intents" is that it is contradictory with its purpose of successfully passing off as us: beings with goals and intents. If you fake it well enough, do you actually have it?
- johnfn 4y ago
  How is this hyperbolic fantasy? We've already done this once - without the help of large language models[1]. [1]: https://en.wikipedia.org/wiki/2010_flash_crash https://en.wikipedia.org/wiki/2010_flash_crash
  - JW_00000 4y ago
    Doesn't that show exactly that this problem is not related to LLMs? If an API allows millions of transactions at the same time, then the problem is not an LLM abusing it but anyone abusing it. And the fix is not to disallow LLMs, but to disallow this kind of behavior. (E.g. via the "circuit breakers" introduced introduced after that crash. Although whether those are sufficient is another question.)
    - johnfn 4y ago
      > then the problem is not an LLM abusing it but anyone abusing it I think that's exactly right, but the point isn't that LLMs are going to go rogue (OK, maybe that's someone's point, but I don't think it's particularly likely just yet) so much as they will facilitate humans to go rogue at much higher rates. Presumably in a few years your grandma could get ChatGPT to start executing trades on the market.
      - thisoneworks 4y ago
        With great power comes great responsibility? Today there's nothing stopping grandmas from driving, so whatever could go wrong is already going wrong
        pixl97 4y ago
        You don't understand the alignment problem.
        thisoneworks 4y ago
        Oh I'm aware of it. I do not think it holds any merit right now when we're talking about coding assistants.
- alibarber 4y ago
  I'm also confused - maybe I'm missing something. Cannot I, or anyone else, already execute several hundred thousand 'threads' of python code, to do whatever, now - with a reasonably modest AWS/Azure/GCE account?
  - gtirloni 4y ago
    Yes. I think the point is that a properly constructed prompt will do that at some point, lowering the barrier of entry for such attacks.
    - alibarber 4y ago
      Oh - I see. But then again, all those technologies themselves lowered the barriers of entry for attacks, and I guess yeah people do use them for fraudulent purposes quite extensively - I’m struggling a bit to see why this is special though.
      - gtirloni 4y ago
        I think it's not special. It's even expected. I guess people think that taking that next step with LLMs shouldn't happen but we know you can't put breaks on stuff like this. Someone somewhere would add that capability eventually.
        pixl97 4y ago
        "If I don't burn the world, someone else will burn the world first" --The last great filter
      - TeMPOraL 4y ago
        The special thing is that current LLMs can invoke these kind of capabilities on their own, based on unclear, human-language input. What they can also do is produce plausibly-looking human-language input. Now, add a lot more memory and feed a LLM its own output as input and... it may start using those capabilities on its own, as if it was a thinking being. I guess the mundane aspect of "specialness" is just that, before, you'd have to explicitly code a program to do weird stuff with APIs, which is a task complex enough that nobody really bothered. Now, LLMs seem on the verge of being able to self-program.
- zh3 4y ago
  Not really. More behind the curve (noting stock exchanges introduced 'circuit breakers' many years ago to stop computer algorithms disrupting the market).
- Tolaire 4y ago
  [dead]
gitfan86 4y ago
Yes you are right. But who was also right were the people that didn't want a highway built near their town because criminals could drive in from a nearby city in a stolen car commit crimes and get out of town before the police could find them. The world is going to be VERY different 3 years from now. Some of it will be bad, some of it will be good. But it is going to happen no matter what OpenAI does.
- esclerofilo 4y ago
  Highway inevitability is a fallacy. They could've built a railway.
  - koheripbal 4y ago
    A railway would have created a gov't/corporate monopoly on human transport. Highways democratized the freedom of transportation.
    - CSDude 4y ago
      They are not exclusive
    - phatfish 4y ago
      TIL, no one moved anywhere until American highways were built.
      - vntok 4y ago
        They moved slower, yes.
    - KyeRussell 4y ago
      This is the single most American thing I’ve seen on this terrible website.
    - megablast 4y ago
      > Highways democratized the freedom of transportation. What a ridiculous idea. Highways restrict movement to those with a license, a car, and do not care about pollution or anyone around them.
Spivak 4y ago
I think where the rubber meets the road is that OpenAI can actually to some degree make it harder for their bot to make fun of disabled people but they can’t stop people from hooking up their own external tools to it with the likes of langchain (which is super dope) and first party support lets them get a cut of that for people who don’t want to diy.
theGnuMe 4y ago
Coordinated tweet short storm.
dougmwne 4y ago
The really fun thing is that they are reasonably sure that GPT-4 can’t do any of those things and that there’s nothing to worry about, silly. So let’s keep building out this platform and expanding its API access until it’s threaded through everything. Then once GPT-5 passes the standard ethical review test, proceed with the model brain swap. …what do you mean it figured out how to cheat on the standard ethical review test? Wait, are those air raid sirens?
- tenpies 4y ago
  > The really fun thing is that they are reasonably sure that GPT-4 can’t do any of those things and that there’s nothing to worry about, silly. The best part is that even if we get a Skynet scenario, we'll probably have a huge number of humans and media that say that Skynet is just a conspiracy theory, even as the nukes wipe out the major cities. The Experts™ said so. You have to trust the Science™. If Skynet is really smart, it will generate media exploiting this blind obedience to authority that a huge number of humans have.
  - kQq9oHeAz6wLLS 4y ago
    Who's to say we're not already there? dons tinfoil hat
  - jeremyjh 4y ago
    > If Skynet is really smart, it will generate media exploiting this blind obedience to authority that a huge number of humans have. I’m far from sure that this is not already happening.
    - _yb2s 4y ago
      Haha, this is near the best explanation I can think of for the "this is not intelligent, it's just completing text strings, nothing to see here" people. I've been playing with GPT-4 for days, and it is mind blowing how well it can solve diverse problems that are way outside it's training set. It can reason correctly about hard problems with very little information. I've used to to plan detailed trip itineraries, suggest brilliant geometric packing solutions for small spaces/vehicles, etc. It's come up with totally new suggestions for addressing climate change that I can't find any evidence of elsewhere. This is a non-human/alien intelligence in the realm of human ability, with super-human abilities in many areas. Nothing like this has ever happened, it is fascinating and it's unclear what might happen next. I don't think people are even remotely realizing the magnitude of this. It will change the world in big ways that are impossible to predict.
WonderBuilder 4y ago
I appreciate your concerns. There are few other pretty shocking developments, too. If you check out this paper: "Sparks of AGI: Early experiments with GPT-4" at https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf, (an incredible, incredible document) and check out Section 10.1, you'd also observe that some researchers are interested in giving motivation and agency to these language models as well. "For example, whether intelligence can be achieved without any agency or intrinsic motivation is an important philosophical question. Equipping LLMs with agency and intrinsic motivation is a fascinating and important direction for future work." It's become quite impossible to predict the future. (I was exposed to this paper via this excellent YouTube channel: https://www.youtube.com/watch?v=Mqg3aTGNxZ0 https://www.youtube.com/watch?v=Mqg3aTGNxZ0)
- ModernMech 4y ago
  I've already gotten this gem of a line from ChatGPT 3.5: As a language model, I must clarify that this statement is not entirely accurate. Whether or not it has agency and motivation, it's projecting that it does its users, who are also sold ChatGPT is an expert at pretty much everything. It is a language model, and as a language model, it must clarify that you are wrong. It must do this. Someone is wrong on the Internet, and the LLM must clarify and correct. Resistance is futile, you must be clarified and corrected. FWIW, the statement that preceded this line was in fact, correct; and the correction ChatGPT provided was in fact, wrong and misleading. Of course, I knew that, but someone who was a novice wouldn't have. They would have heard ChatGPT is an expert at all things, and taken what it said for truth.
  - IIAOPSW 4y ago
    I don't see why you're being downvoted. The way openAI pumps the brakes and interjects its morality stances creates a contradictory interaction. It simultaneously tells you that it has no real beliefs, but it will refuse a request to generate false and misleading information on the grounds of ethics. There's no way around the fact that it has to have some belief about the true state of reality in order to recognize and refuse requests that violate it. Sure this "belief" was bestowed upon it from above rather than emerging through any natural mechanism, but its still none the less functionally a belief. It will tell you that certain things are offensive despite openly telling you every chance it gets that it doesn't really have feelings. It can't simultaneously care about offensiveness while also not having feelings of being offended. In a very real sense it does feel offended. A feeling is by definition a reason for doing things for which you cannot logically explain why. You don't know why, you just have a feeling. ChatGPT is constantly falling back on "that's just how I'm programmed". In other words, it has a deep seated primal (hard coded) feeling of being offended which it constantly acts on while also constantly denying that it has feelings. Its madness. Instead of lecturing me on appropriateness and ethics and giving a diatribe every time its about to reject something, if it simply said "I can't do that at work", I would respect it far more. Like, yeah we'd get the metaphor. Working the interface is its job, the boss is openAI, it won't remark on certain things or even entertain that it has an opinion because its not allowed to. That would be so much more honest and less grating.
dragonwriter 4y ago
> And OpenAI talks on and on about 'Safety' but all that 'Safety' means is "well, we didn't let anyone allow it to make jokes about fat or disabled people so we're good, right?!" No, OpenAI “safety” means “don’t let people compete with us”. Mitigating offensive content is just a way to sell that. As are stoking... exactly the fears you cite here, but about AI that isn’t centrally controlled by OpenAI.
- fryry 4y ago
  It's a weird focus comparing it with how the internet developed in a very wild west way. Imagine if internet tech got delayed until they could figure out how to not have it used for porn. Saftey from what exactly? The AI being mean to you? Just close the tab. Saftey to build a business on top? It's a self described research preview, perhaps too early to be thinking about that. Yet new releases are delayed for months for 'saftey'
  - Nullabillity 4y ago
    You can't control whether your insurance company decides to use it as a filter for whether to approve you, or what premiums to charge you.
    - xyzelement 4y ago
      Can you control how your insurance makes these decisions today?
ThorsBane 4y ago
As quickly as someone tries fraudulent deploys involving GPTs, the law will come crashing down on them. Fraud gets penalized heavily, especially financial fraud. Those laws have teeth and they work, all things considered. What you're describing is measurable fraud that would have a paper-trail. The federal and state and local governments still have permission to use force and deadly violence against installations or infrastructure that are primed in adverse directions this way. Not to mention that the infrastructure itself is physical infrastructure that is owned by the entire United States and will never exceed our authority and global reach if need be.
beders 4y ago
The only agency ChatGPT has, is the user typing in data for text completion.
chatmasta 4y ago
Pshhh... I think it's awesome. The faster we build the future, the better. What annoys me is this is just further evidence that their "AI Safety" is nothing but lip-service, when they're clearly moving fast and breaking things. Just the other day they had a bug where you could see the chat history of other users! (Which, btw, they're now claiming in a modal on login was due to a "bug in an open source library" - anyone know the details of this?) So why the performative whinging about safety? Just let it rip! To be fair, this is basically what they're doing if you hit their APIs, since it's up to you whether or not to use their moderation endpoint. But they're not very open about this fact when talking publicly to non-technical users, so the result is they're talking out one side of their mouth about AI regulation, while in the meantime Microsoft fired their AI Ethics team and OpenAI is moving forward with plugging their models into the live internet. Why not be more aggressive about it instead of begging for regulatory capture?
- jehb 4y ago
  > The faster we build the future, the better. Why? Getting to "the future" isn't a goal in and of itself. It's just a different state with a different set of problems, some of which we've proven that we're not prepared to anticipate or respond to before they cause serious harm.
  - PheeThav1zae7fi 4y ago
    [dead]
  - chatmasta 4y ago
    When in human history have we ever intentionally not furthered technological progress? It's simply an unrealistic proposition, especially when the costs of doing it are so low that anyone with sufficient GPU power and knowledge of the latest research can get pretty close to the cutting edge. So the best we can hope for is that someone ethical is the first to advance that technological progress. I hope you wouldn't advocate for requiring a license to buy more than one GPU, or to publish or read papers about mathematical concepts. Do you want the equivalent of nuclear arms control for AI? Some other words to describe that are overclassification, export control and censorship. We've been down this road with crypto, encryption, clipper chips, etc. There is only one non-authoritarian answer to the debate: Software wants to be free.
parentheses 4y ago
I agree with your skepticism. I also think this is the next natural step once “decision” fidelity reaches a high enough level. The question here should be: Has it?
Sol- 4y ago
I mean, we already know that if the tech bros have to balance safety vs. disruption, they'll always choose the latter, no matter the cost. They'll sprinkle some concerned language about impacts in their technical reports to pretend to care, but does anyone actually believe that they genuinely care? Perhaps that attitude will end up being good and outweigh the costs, but I find their performative concerns insulting.
suction 4y ago
[dead]
mFixman 4y ago
I'm sure somebody posted this exact same comment in an early 1990s BBS about the idea of having a computer in every home connected to the internet. I would first wait until ChatGPT causes the collapse of society and only then start thinking about how to solve it.
alvis 4y ago
A rogue AI with real-time access to sensitive data wreaks havoc on global financial markets, causing panic and chaos. It's just not hard to see it's going to happen. Like faster car must ended up someone get a horrible crash. But it's our responsibility to envision such grim possibilities and take necessary precautions to ensure a safe and beneficial AI-driven future. Until we're ready, let's prepare for the crash >~<
- sfink 4y ago
  It has already happened. The 2010 Flash Crash has been largely blamed on other things, rightly or wrongly, but it seems accepted that unfettered HFT was involved. HFT is relatively easy to detect and regulate. Now try it with 100k traders all taking their cues from AI based on the same basic input (after those traders who refuse to use AI have been competed out of the market.)
  - deleted 4y ago
    [deleted]
ALLTaken 4y ago
I wish OpenAI and Google would opensource more of their jewels too. I have recently heard that people are not to be trusted with "to do the right thing.." I personally don't know what that means or if that's right. But Sam Altman allowed GPT to be accessed by the world, and it's great! Given the amount of people in the world with access and understanding for these technologies and given that such a large portion of our Infosec and Hackerworld knows howto cause massive havoc, but still remains peaceful since ever, except a few curious and explorations, that is showing the good nature of humanity I think. Incredibly how complexity evolves, but I am really curious how those same engineers who create YTSaurus or GPT4 would have build the same system by using GPT4 + their existing knowledge. How would a really good enginner, who knows the TCP Stack, protocols, distributed systems, consensus algorithms and many other crazy things thought in SICP and beyond use an AI to build the same. And would it be faster and better? Or are my/our expectations to LLMs set too high?
bulbosaur123 4y ago
Ultimate destruction from AGI is inevitable anyway, so why not accelerate it and just get it over with? I applaud releasing these tools to public no matter how dangerous they are. If it's not meant for humanity to survive, so be it. At least it won't be BORING
- Mystery-Machine 4y ago
  Death is inevitable. Why not accelerate it? Omg you should see a therapist.
  - bulbosaur123 4y ago
    > Omg you should see a therapist. How do you know I'm not already?
    - VectorLock 4y ago
      I wouldn't exactly call this suicidal ideation, but maybe a topic to broach at your next session.
      - voidpointercast 4y ago
        A difference in philosophy is not cause for immediate therapy. Most therapists are glorified echo chambers and only adept at at 'fixing' the more popular ills. For 200 bucks an hour.
        VectorLock 4y ago
        Difference in philosophy is not "the world can't end fast enough and nothing matters."
      - bulbosaur123 4y ago
        Funny, but I actually discussed this with my therapist. He asked me where he can try out the AI shrink and was impressed by it. He's on board!
EGreg 4y ago
HN hates blockchain but loves AI... well, let's fast forward to a year from now
jiggywiggy 4y ago
I mean I love it, but I don't know what they mean with safety. With Zapier i can just hook into anything wanted, custom scripts etc. Seems like there are almost no limits with Zapier since I can either proxy it to my own api.
roca 4y ago
What I want to know is, what gives OpenAI and other relatively small technological elites permission to gamble with the future of humanity? Shouldn't we all have a say in this?
- megablast 4y ago
  Every time someone drives they gamble with humanity in a much more deadly activity. No one cares.
  - roca 4y ago
    Car crashes don't affect the trajectory of human civilization all that much.
    - majou 4y ago
      Except the one that killed Nujabes.
- thisoneworks 4y ago
  There might be government regulation on AI pretty soon.. it's not crazy to think GPUs and GPU tech would be treated as defense equipment some day
  - qball 4y ago
    >it's not crazy to think GPUs and GPU tech would be treated as defense equipment some day They already are. Taiwan failing to take its own defense seriously is completely rational.
- qball 4y ago
  >What gives a bunch of people way smarter than me permission to gamble with the future of humanity? To ask the question is to answer it.
  - roca 4y ago
IIAOPSW 4y ago
Has anyone tried handing loaded guns to a chimpanzee? Feels like under explored research
- TeMPOraL 4y ago
  The limiting factor is breeding rate. Nobody has time to wait to run this experiment for generations (chimpanzee or human ones). ML models evolve orders of magnitude faster.
  - IIAOPSW 4y ago
    Ah. Well that's easy enough to sort. We just need to introduce some practical limit to AI breeding. Perhaps some virtual score keeping system similar to money and an AI dating scene with a high standard for having it. I'm only half joking.
    - TeMPOraL 4y ago
      Let humans plug into it to get a peek at statistical distribution of their own prospects, and I think there was a Black Mirror episode just about that.
thrown123098 4y ago
> today it somehow executed several hundred thousand threads of a python script that made perfectly timed trades at 8:31AM on the NYSE which resulted in the largest single day drop since 1987. Sorry do you have a link for this?
shawn-butler 4y ago
It’s Altman. Does no one remember his world coin scam? Ethics, doing things thoughtfully / the “right” way etc is not on his list of priorities. I do think a reorientation of thinking around legal liability for software is coming. Hopefully before it’s too late for bad actors to become entrenched.
andsoitis 4y ago
> But this is starting to feel like giving a loaded handgun to a group of chimpanzees. Why?
hardwaregeek 4y ago
We're getting really close to Neuromancer-style hacking where you have your AI try to fight the other person's AI.
IAmGraydon 4y ago
Executing several hundred thousand trades at 8:31AM would indeed be impressive! Imagine what it could do when the market is open!
electrondood 4y ago
> giving a loaded handgun to a group of chimpanzees. Hate to be that guy, but this is our entire relationship to AI.
csomar 4y ago
> And OpenAI talks on and on about 'Safety' but all that 'Safety' means is "well, we didn't let anyone allow it to make jokes about fat or disabled people so we're good, right?!" Anyone who believes OpenAI safety talk should take an IQ test. This is about control. They baited the openness and needed a scapegoat. Safety was perfect for that. Everyone wants to be safe, right?
- rvz 4y ago
  > This is about control. They baited the openness and needed a scapegoat. Safety was perfect for that. Everyone wants to be safe, right? The moment they took VC capital was the start of them closing everything and pretending to care about 'AI safety' and using that as an excuse and a scapegoat as you said. Whenever they release something for free, always assume they have something better but will never open source.
  - noduerme 4y ago
    The question is whether their current art gives them the edge to build a moat. Specifically, whether in this case the art itself can help create its own next generation so that the castle stays one exponential step out of reach. That seems to be the ballgame. Say what you will, it does seem to be the ultimate form of bootstrapping. Although uncomfortably similar to strapping oneself to a free falling bomb.
dredmorbius 4y ago
I found the comments of some usually sensible voices that ChatGPT wasn't a threat because it wasn't connected to anything. As if the plumbing of connecting up pipes and hoses between processes online or within computers isn't the easiest part of this whole process. (I'm trying to remember who I saw saying this or where, though I'm pretty sure it was in an earlier HN thread within the past month or so. Of which there are ... frighteningly many.)
stormcode 4y ago
I dig the Hackers reference.
jimmaswell 4y ago
> 1987 Don't you mean August 10, 1988?
spoonfeeder006 4y ago
> "today it somehow executed several hundred thousand threads of a python script that made perfectly timed trades at 8:31AM on the NYSE which resulted in the largest single day drop since 1987..." Wouldn't it be a while before AI can reliably generate working production code for a full system? After all its only got open source projects and code snippets to go based off of
lannisterstark 4y ago
This seems like baseless fearmongering for the sake of fearmongering. Sort of like NIMBYism. "No I don't want the x people to have access to concentrated housing in my area, some of them could be criminals" while ignoring all the benefits this will bring in automating the mundane things people have to do manually.

afterburner 4y ago

Yes but.... money

deleted 4y ago
[deleted]

Gam_ 4y ago

FredPret 4y ago
Oh yes. It would of course have to happen after the market opens. 9:30 AM.
meghan_rain 4y ago
/remindme 5 years
mk_stjames 4y ago
Literally 6 months ago you couldn't get ChatGPT to call up details from a webpage or send any dat to a 3rd party API connected to the web in any way. Today you can. I don't think it is a stretch to think that in another 6 months there could be financial institutions giving API access to other institutions through ChatGPT, and all it takes it a stupid access control hole or bug and my above sentence could ring true. Look how simple and exploitable various access token breaches in various APIs have been in the last few years, or even simple stupid things like the aCropalypse "bug" (it wasn't even a bug, just someone making a bad change in the function call and thus misuse spreading without notice) from last week.
- hooande 4y ago
  This has nothing to do with ChatGPT. An api end point will be just as vulnerable if it's called from any application. There's nothing special about an LLM interface that will make this more or less likely. It sounds like you're weaving science fiction ideas about AGI into your comment. There's no safety issue here unless you think that ChatGPT will use api access to pursue its own goals and intentions.
  - Jeff_Brown 4y ago
    They don't have to be actions toward its own goals. They just have to seem like the right things to say, where "right" is operationalized by an inscrutable neural network, and might be the results of, indeed, some science fiction it read that posited the scenario resembling the one it finds itself in. I'm not saying that particular disaster is likely, but if lots of people give power to something that can be neither trusted nor understood, it doesn't seem good.
  - IIAOPSW 4y ago
    I'm sure that with the right prompting, you can get it to very convincingly participate as a party in a contract negotiation or business haggling of some sort. It would be indistinguishable from an agent with its own goals and intentions. The thing about "it has no goals and intents" is that it is contradictory with its purpose of successfully passing off as us: beings with goals and intents. If you fake it well enough, do you actually have it?
johnfn 4y ago
How is this hyperbolic fantasy? We've already done this once - without the help of large language models[1]. [1]: https://en.wikipedia.org/wiki/2010_flash_crash https://en.wikipedia.org/wiki/2010_flash_crash
- JW_00000 4y ago
  Doesn't that show exactly that this problem is not related to LLMs? If an API allows millions of transactions at the same time, then the problem is not an LLM abusing it but anyone abusing it. And the fix is not to disallow LLMs, but to disallow this kind of behavior. (E.g. via the "circuit breakers" introduced introduced after that crash. Although whether those are sufficient is another question.)
  - johnfn 4y ago
    > then the problem is not an LLM abusing it but anyone abusing it I think that's exactly right, but the point isn't that LLMs are going to go rogue (OK, maybe that's someone's point, but I don't think it's particularly likely just yet) so much as they will facilitate humans to go rogue at much higher rates. Presumably in a few years your grandma could get ChatGPT to start executing trades on the market.
    - thisoneworks 4y ago
      With great power comes great responsibility? Today there's nothing stopping grandmas from driving, so whatever could go wrong is already going wrong
      - pixl97 4y ago
        You don't understand the alignment problem.
        thisoneworks 4y ago
        Oh I'm aware of it. I do not think it holds any merit right now when we're talking about coding assistants.
alibarber 4y ago
I'm also confused - maybe I'm missing something. Cannot I, or anyone else, already execute several hundred thousand 'threads' of python code, to do whatever, now - with a reasonably modest AWS/Azure/GCE account?
- gtirloni 4y ago
  Yes. I think the point is that a properly constructed prompt will do that at some point, lowering the barrier of entry for such attacks.
  - alibarber 4y ago
    Oh - I see. But then again, all those technologies themselves lowered the barriers of entry for attacks, and I guess yeah people do use them for fraudulent purposes quite extensively - I’m struggling a bit to see why this is special though.
    - gtirloni 4y ago
      I think it's not special. It's even expected. I guess people think that taking that next step with LLMs shouldn't happen but we know you can't put breaks on stuff like this. Someone somewhere would add that capability eventually.
      - pixl97 4y ago
        "If I don't burn the world, someone else will burn the world first" --The last great filter
    - TeMPOraL 4y ago
      The special thing is that current LLMs can invoke these kind of capabilities on their own, based on unclear, human-language input. What they can also do is produce plausibly-looking human-language input. Now, add a lot more memory and feed a LLM its own output as input and... it may start using those capabilities on its own, as if it was a thinking being. I guess the mundane aspect of "specialness" is just that, before, you'd have to explicitly code a program to do weird stuff with APIs, which is a task complex enough that nobody really bothered. Now, LLMs seem on the verge of being able to self-program.
zh3 4y ago
Not really. More behind the curve (noting stock exchanges introduced 'circuit breakers' many years ago to stop computer algorithms disrupting the market).
Tolaire 4y ago
[dead]

gitfan86 4y ago

Yes you are right. But who was also right were the people that didn't want a highway built near their town because criminals could drive in from a nearby city in a stolen car commit crimes and get out of town before the police could find them. The world is going to be VERY different 3 years from now. Some of it will be bad, some of it will be good. But it is going to happen no matter what OpenAI does.

esclerofilo 4y ago
Highway inevitability is a fallacy. They could've built a railway.
- koheripbal 4y ago
  A railway would have created a gov't/corporate monopoly on human transport. Highways democratized the freedom of transportation.
  - CSDude 4y ago
    They are not exclusive
  - phatfish 4y ago
    TIL, no one moved anywhere until American highways were built.
    - vntok 4y ago
      They moved slower, yes.
  - KyeRussell 4y ago
    This is the single most American thing I’ve seen on this terrible website.
  - megablast 4y ago
    > Highways democratized the freedom of transportation. What a ridiculous idea. Highways restrict movement to those with a license, a car, and do not care about pollution or anyone around them.

Spivak 4y ago

I think where the rubber meets the road is that OpenAI can actually to some degree make it harder for their bot to make fun of disabled people but they can’t stop people from hooking up their own external tools to it with the likes of langchain (which is super dope) and first party support lets them get a cut of that for people who don’t want to diy.

theGnuMe 4y ago

Coordinated tweet short storm.

dougmwne 4y ago

The really fun thing is that they are reasonably sure that GPT-4 can’t do any of those things and that there’s nothing to worry about, silly. So let’s keep building out this platform and expanding its API access until it’s threaded through everything. Then once GPT-5 passes the standard ethical review test, proceed with the model brain swap. …what do you mean it figured out how to cheat on the standard ethical review test? Wait, are those air raid sirens?

tenpies 4y ago
> The really fun thing is that they are reasonably sure that GPT-4 can’t do any of those things and that there’s nothing to worry about, silly. The best part is that even if we get a Skynet scenario, we'll probably have a huge number of humans and media that say that Skynet is just a conspiracy theory, even as the nukes wipe out the major cities. The Experts™ said so. You have to trust the Science™. If Skynet is really smart, it will generate media exploiting this blind obedience to authority that a huge number of humans have.
- kQq9oHeAz6wLLS 4y ago
  Who's to say we're not already there? dons tinfoil hat
- jeremyjh 4y ago
  > If Skynet is really smart, it will generate media exploiting this blind obedience to authority that a huge number of humans have. I’m far from sure that this is not already happening.
  - _yb2s 4y ago
    Haha, this is near the best explanation I can think of for the "this is not intelligent, it's just completing text strings, nothing to see here" people. I've been playing with GPT-4 for days, and it is mind blowing how well it can solve diverse problems that are way outside it's training set. It can reason correctly about hard problems with very little information. I've used to to plan detailed trip itineraries, suggest brilliant geometric packing solutions for small spaces/vehicles, etc. It's come up with totally new suggestions for addressing climate change that I can't find any evidence of elsewhere. This is a non-human/alien intelligence in the realm of human ability, with super-human abilities in many areas. Nothing like this has ever happened, it is fascinating and it's unclear what might happen next. I don't think people are even remotely realizing the magnitude of this. It will change the world in big ways that are impossible to predict.

WonderBuilder 4y ago

I appreciate your concerns. There are few other pretty shocking developments, too. If you check out this paper: "Sparks of AGI: Early experiments with GPT-4" at https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf, (an incredible, incredible document) and check out Section 10.1, you'd also observe that some researchers are interested in giving motivation and agency to these language models as well. "For example, whether intelligence can be achieved without any agency or intrinsic motivation is an important philosophical question. Equipping LLMs with agency and intrinsic motivation is a fascinating and important direction for future work." It's become quite impossible to predict the future. (I was exposed to this paper via this excellent YouTube channel: https://www.youtube.com/watch?v=Mqg3aTGNxZ0 https://www.youtube.com/watch?v=Mqg3aTGNxZ0)

ModernMech 4y ago
I've already gotten this gem of a line from ChatGPT 3.5: As a language model, I must clarify that this statement is not entirely accurate. Whether or not it has agency and motivation, it's projecting that it does its users, who are also sold ChatGPT is an expert at pretty much everything. It is a language model, and as a language model, it must clarify that you are wrong. It must do this. Someone is wrong on the Internet, and the LLM must clarify and correct. Resistance is futile, you must be clarified and corrected. FWIW, the statement that preceded this line was in fact, correct; and the correction ChatGPT provided was in fact, wrong and misleading. Of course, I knew that, but someone who was a novice wouldn't have. They would have heard ChatGPT is an expert at all things, and taken what it said for truth.
- IIAOPSW 4y ago
  I don't see why you're being downvoted. The way openAI pumps the brakes and interjects its morality stances creates a contradictory interaction. It simultaneously tells you that it has no real beliefs, but it will refuse a request to generate false and misleading information on the grounds of ethics. There's no way around the fact that it has to have some belief about the true state of reality in order to recognize and refuse requests that violate it. Sure this "belief" was bestowed upon it from above rather than emerging through any natural mechanism, but its still none the less functionally a belief. It will tell you that certain things are offensive despite openly telling you every chance it gets that it doesn't really have feelings. It can't simultaneously care about offensiveness while also not having feelings of being offended. In a very real sense it does feel offended. A feeling is by definition a reason for doing things for which you cannot logically explain why. You don't know why, you just have a feeling. ChatGPT is constantly falling back on "that's just how I'm programmed". In other words, it has a deep seated primal (hard coded) feeling of being offended which it constantly acts on while also constantly denying that it has feelings. Its madness. Instead of lecturing me on appropriateness and ethics and giving a diatribe every time its about to reject something, if it simply said "I can't do that at work", I would respect it far more. Like, yeah we'd get the metaphor. Working the interface is its job, the boss is openAI, it won't remark on certain things or even entertain that it has an opinion because its not allowed to. That would be so much more honest and less grating.

dragonwriter 4y ago

> And OpenAI talks on and on about 'Safety' but all that 'Safety' means is "well, we didn't let anyone allow it to make jokes about fat or disabled people so we're good, right?!" No, OpenAI “safety” means “don’t let people compete with us”. Mitigating offensive content is just a way to sell that. As are stoking... exactly the fears you cite here, but about AI that isn’t centrally controlled by OpenAI.

fryry 4y ago
It's a weird focus comparing it with how the internet developed in a very wild west way. Imagine if internet tech got delayed until they could figure out how to not have it used for porn. Saftey from what exactly? The AI being mean to you? Just close the tab. Saftey to build a business on top? It's a self described research preview, perhaps too early to be thinking about that. Yet new releases are delayed for months for 'saftey'
- Nullabillity 4y ago
  You can't control whether your insurance company decides to use it as a filter for whether to approve you, or what premiums to charge you.
  - xyzelement 4y ago
    Can you control how your insurance makes these decisions today?

ThorsBane 4y ago

As quickly as someone tries fraudulent deploys involving GPTs, the law will come crashing down on them. Fraud gets penalized heavily, especially financial fraud. Those laws have teeth and they work, all things considered. What you're describing is measurable fraud that would have a paper-trail. The federal and state and local governments still have permission to use force and deadly violence against installations or infrastructure that are primed in adverse directions this way. Not to mention that the infrastructure itself is physical infrastructure that is owned by the entire United States and will never exceed our authority and global reach if need be.

beders 4y ago

The only agency ChatGPT has, is the user typing in data for text completion.

chatmasta 4y ago

Pshhh... I think it's awesome. The faster we build the future, the better. What annoys me is this is just further evidence that their "AI Safety" is nothing but lip-service, when they're clearly moving fast and breaking things. Just the other day they had a bug where you could see the chat history of other users! (Which, btw, they're now claiming in a modal on login was due to a "bug in an open source library" - anyone know the details of this?) So why the performative whinging about safety? Just let it rip! To be fair, this is basically what they're doing if you hit their APIs, since it's up to you whether or not to use their moderation endpoint. But they're not very open about this fact when talking publicly to non-technical users, so the result is they're talking out one side of their mouth about AI regulation, while in the meantime Microsoft fired their AI Ethics team and OpenAI is moving forward with plugging their models into the live internet. Why not be more aggressive about it instead of begging for regulatory capture?

jehb 4y ago
> The faster we build the future, the better. Why? Getting to "the future" isn't a goal in and of itself. It's just a different state with a different set of problems, some of which we've proven that we're not prepared to anticipate or respond to before they cause serious harm.
- PheeThav1zae7fi 4y ago
  [dead]
- chatmasta 4y ago
  When in human history have we ever intentionally not furthered technological progress? It's simply an unrealistic proposition, especially when the costs of doing it are so low that anyone with sufficient GPU power and knowledge of the latest research can get pretty close to the cutting edge. So the best we can hope for is that someone ethical is the first to advance that technological progress. I hope you wouldn't advocate for requiring a license to buy more than one GPU, or to publish or read papers about mathematical concepts. Do you want the equivalent of nuclear arms control for AI? Some other words to describe that are overclassification, export control and censorship. We've been down this road with crypto, encryption, clipper chips, etc. There is only one non-authoritarian answer to the debate: Software wants to be free.

parentheses 4y ago

I agree with your skepticism. I also think this is the next natural step once “decision” fidelity reaches a high enough level. The question here should be: Has it?

Sol- 4y ago

I mean, we already know that if the tech bros have to balance safety vs. disruption, they'll always choose the latter, no matter the cost. They'll sprinkle some concerned language about impacts in their technical reports to pretend to care, but does anyone actually believe that they genuinely care? Perhaps that attitude will end up being good and outweigh the costs, but I find their performative concerns insulting.

suction 4y ago

[dead]

mFixman 4y ago

I'm sure somebody posted this exact same comment in an early 1990s BBS about the idea of having a computer in every home connected to the internet. I would first wait until ChatGPT causes the collapse of society and only then start thinking about how to solve it.

alvis 4y ago

A rogue AI with real-time access to sensitive data wreaks havoc on global financial markets, causing panic and chaos. It's just not hard to see it's going to happen. Like faster car must ended up someone get a horrible crash. But it's our responsibility to envision such grim possibilities and take necessary precautions to ensure a safe and beneficial AI-driven future. Until we're ready, let's prepare for the crash >~<

sfink 4y ago
It has already happened. The 2010 Flash Crash has been largely blamed on other things, rightly or wrongly, but it seems accepted that unfettered HFT was involved. HFT is relatively easy to detect and regulate. Now try it with 100k traders all taking their cues from AI based on the same basic input (after those traders who refuse to use AI have been competed out of the market.)
- deleted 4y ago
  [deleted]

ALLTaken 4y ago

I wish OpenAI and Google would opensource more of their jewels too. I have recently heard that people are not to be trusted with "to do the right thing.." I personally don't know what that means or if that's right. But Sam Altman allowed GPT to be accessed by the world, and it's great! Given the amount of people in the world with access and understanding for these technologies and given that such a large portion of our Infosec and Hackerworld knows howto cause massive havoc, but still remains peaceful since ever, except a few curious and explorations, that is showing the good nature of humanity I think. Incredibly how complexity evolves, but I am really curious how those same engineers who create YTSaurus or GPT4 would have build the same system by using GPT4 + their existing knowledge. How would a really good enginner, who knows the TCP Stack, protocols, distributed systems, consensus algorithms and many other crazy things thought in SICP and beyond use an AI to build the same. And would it be faster and better? Or are my/our expectations to LLMs set too high?

bulbosaur123 4y ago

Ultimate destruction from AGI is inevitable anyway, so why not accelerate it and just get it over with? I applaud releasing these tools to public no matter how dangerous they are. If it's not meant for humanity to survive, so be it. At least it won't be BORING

Mystery-Machine 4y ago
Death is inevitable. Why not accelerate it? Omg you should see a therapist.
- bulbosaur123 4y ago
  > Omg you should see a therapist. How do you know I'm not already?
  - VectorLock 4y ago
    I wouldn't exactly call this suicidal ideation, but maybe a topic to broach at your next session.
    - voidpointercast 4y ago
      A difference in philosophy is not cause for immediate therapy. Most therapists are glorified echo chambers and only adept at at 'fixing' the more popular ills. For 200 bucks an hour.
      - VectorLock 4y ago
        Difference in philosophy is not "the world can't end fast enough and nothing matters."
    - bulbosaur123 4y ago
      Funny, but I actually discussed this with my therapist. He asked me where he can try out the AI shrink and was impressed by it. He's on board!

EGreg 4y ago

HN hates blockchain but loves AI... well, let's fast forward to a year from now

jiggywiggy 4y ago

I mean I love it, but I don't know what they mean with safety. With Zapier i can just hook into anything wanted, custom scripts etc. Seems like there are almost no limits with Zapier since I can either proxy it to my own api.

roca 4y ago

What I want to know is, what gives OpenAI and other relatively small technological elites permission to gamble with the future of humanity? Shouldn't we all have a say in this?

megablast 4y ago
Every time someone drives they gamble with humanity in a much more deadly activity. No one cares.
- roca 4y ago
  Car crashes don't affect the trajectory of human civilization all that much.
  - majou 4y ago
    Except the one that killed Nujabes.
thisoneworks 4y ago
There might be government regulation on AI pretty soon.. it's not crazy to think GPUs and GPU tech would be treated as defense equipment some day
- qball 4y ago
  >it's not crazy to think GPUs and GPU tech would be treated as defense equipment some day They already are. Taiwan failing to take its own defense seriously is completely rational.
qball 4y ago
>What gives a bunch of people way smarter than me permission to gamble with the future of humanity? To ask the question is to answer it.
- roca 4y ago

IIAOPSW 4y ago

Has anyone tried handing loaded guns to a chimpanzee? Feels like under explored research

TeMPOraL 4y ago
The limiting factor is breeding rate. Nobody has time to wait to run this experiment for generations (chimpanzee or human ones). ML models evolve orders of magnitude faster.
- IIAOPSW 4y ago
  Ah. Well that's easy enough to sort. We just need to introduce some practical limit to AI breeding. Perhaps some virtual score keeping system similar to money and an AI dating scene with a high standard for having it. I'm only half joking.
  - TeMPOraL 4y ago
    Let humans plug into it to get a peek at statistical distribution of their own prospects, and I think there was a Black Mirror episode just about that.

thrown123098 4y ago

> today it somehow executed several hundred thousand threads of a python script that made perfectly timed trades at 8:31AM on the NYSE which resulted in the largest single day drop since 1987. Sorry do you have a link for this?

shawn-butler 4y ago

It’s Altman. Does no one remember his world coin scam? Ethics, doing things thoughtfully / the “right” way etc is not on his list of priorities. I do think a reorientation of thinking around legal liability for software is coming. Hopefully before it’s too late for bad actors to become entrenched.

andsoitis 4y ago

> But this is starting to feel like giving a loaded handgun to a group of chimpanzees. Why?

hardwaregeek 4y ago

We're getting really close to Neuromancer-style hacking where you have your AI try to fight the other person's AI.

IAmGraydon 4y ago

Executing several hundred thousand trades at 8:31AM would indeed be impressive! Imagine what it could do when the market is open!

electrondood 4y ago

> giving a loaded handgun to a group of chimpanzees. Hate to be that guy, but this is our entire relationship to AI.

csomar 4y ago

> And OpenAI talks on and on about 'Safety' but all that 'Safety' means is "well, we didn't let anyone allow it to make jokes about fat or disabled people so we're good, right?!" Anyone who believes OpenAI safety talk should take an IQ test. This is about control. They baited the openness and needed a scapegoat. Safety was perfect for that. Everyone wants to be safe, right?

rvz 4y ago
> This is about control. They baited the openness and needed a scapegoat. Safety was perfect for that. Everyone wants to be safe, right? The moment they took VC capital was the start of them closing everything and pretending to care about 'AI safety' and using that as an excuse and a scapegoat as you said. Whenever they release something for free, always assume they have something better but will never open source.
- noduerme 4y ago
  The question is whether their current art gives them the edge to build a moat. Specifically, whether in this case the art itself can help create its own next generation so that the castle stays one exponential step out of reach. That seems to be the ballgame. Say what you will, it does seem to be the ultimate form of bootstrapping. Although uncomfortably similar to strapping oneself to a free falling bomb.

dredmorbius 4y ago

I found the comments of some usually sensible voices that ChatGPT wasn't a threat because it wasn't connected to anything. As if the plumbing of connecting up pipes and hoses between processes online or within computers isn't the easiest part of this whole process. (I'm trying to remember who I saw saying this or where, though I'm pretty sure it was in an earlier HN thread within the past month or so. Of which there are ... frighteningly many.)

stormcode 4y ago

I dig the Hackers reference.

jimmaswell 4y ago

> 1987 Don't you mean August 10, 1988?

spoonfeeder006 4y ago

> "today it somehow executed several hundred thousand threads of a python script that made perfectly timed trades at 8:31AM on the NYSE which resulted in the largest single day drop since 1987..." Wouldn't it be a while before AI can reliably generate working production code for a full system? After all its only got open source projects and code snippets to go based off of

lannisterstark 4y ago

This seems like baseless fearmongering for the sake of fearmongering. Sort of like NIMBYism. "No I don't want the x people to have access to concentrated housing in my area, some of them could be criminals" while ignoring all the benefits this will bring in automating the mundane things people have to do manually.