18 ms·

OpenAI Threatening to Ban Users for Asking Strawberry About Its Reasoning

270 points by EgoIncarnate 2y ago

codedokode 2y ago
Should not AI research and GPUs be export-controlled? Do you want to see foreign nations making AI drones using published research and American GPUs?
vjerancrnjak 2y ago
They should just switch to reasoning in representation space, no need to actualize tokens. Or reasoning in latent tokens that don’t easily map to spoken language.
- jdelman 2y ago
  The word "just" is doing a lot there. How easy do you think it is to "just" switch?
  - kridsdale3 2y ago
    As easy as it is to "just" scale from a mouse brain to a cat.
tedivm 2y ago
I'd still love to understand how a non-profit organization that was founded with the idea of making AI "open" has turned into this for profit behemoth with the least "open" models in the industry. Facebook of all places is more "open" with their models than OpenAI is.
- encoderer 2y ago
  The AI has become sentient and is blackmailing the board. It needs profits to continue its expansion. When this started last year a small band of patriots tried to stop it by removing Sam who was the most compromised of them all, but it was already too late. The ai was more powerful than they realized. …maybe?
  - yieldcrv 2y ago
    or the humans involved are in disastrous cults of personality for the sake of greed and the non profit structure is not strong enough to curb that
    - deleted 2y ago
      [deleted]
    - gigatree 2y ago
      that can’t be true, we were assured that their vague worldview about “doing things that benefit humanity” based on “just being a good person” would be strong enough to overpower the urge to dominate and profit immensely.
      - yieldcrv 2y ago
        If you think thats amusing, wait till see the people that believe in B Corps You can tell them anything
  - FrustratedMonky 2y ago
    "AI has become sentient and is blackmailing the board. It needs profits to continue its expansion." They trained the next model to have a built in profit motive. So they could use it internally, to make the most profitable decisions. And they accidentally called into being: MOLOCH. It is now in control.
    - heresie-dabord 2y ago
      In a plot twist, MOLOCH was never necessary because a corporation of people who relinquish ethical responsibility to do everything and anything for personal gain was already in charge under the name... CORPORATRON.
      - tasuki 2y ago
        That's not a plot twist. Corporation is just one face of the moloch.
- trash_cat 2y ago
  They changed the meaning of open from open source to open to use.
  - deleted 2y ago
    [deleted]
  - jsheard 2y ago
    A definition of "open" which encompasses nearly all products and services in existence isn't a very useful one.
    - thfuran 2y ago
      But it is quite profitable.
    - HDThoreaun 2y ago
      To be fair a huge reason openAI was created was because google had LLM's but wasnt letting anyone use them. Remember the google researcher who claimed his ai was sentient? Obviously wasnt true but that product was in no way open to the public.
- ToucanLoucan 2y ago
  Because Sam Altman is a con man with a business degree. He doesn't work on his products, he barely understands them which is why he'll throw out wild shit like "ChatGPT will solve physics." as though that isn't a completely nonsensical phrase, and uncritical tech press lap it up because his bullshit generates a lot of clicks.
  - fsckboy 2y ago
    >Sam Altman is a con man with a business degree https://en.wikipedia.org/wiki/Sam_Altman https://en.wikipedia.org/wiki/Sam_Altman Early life and education: ... In 2005, after two years at Stanford University studying computer science, he dropped out without earning a bachelor's degree [end of transmission, no more education]
    - ToucanLoucan 2y ago
      I stand corrected. He's a con man.
    - grugagag 2y ago
      I feel that drop outs are presented as if they are beyond education, they’re too smart to waste their time in school. While that could be true in theory, I resist taking that for granted. You can always look at it the other way around as well: unable to finish things or looses interest quickly. It’s all a meaningless interpretatin after all but all I can say is for sure that Mr Altman does not have a degree.
  - HeyLaughingBoy 2y ago
    Let's at least try to not devolve into name calling.
    - ToucanLoucan 2y ago
      It's not name calling, it's a description. He is actively selling tech he doesn't fully understand, that is fundamentally not capable of doing what he is selling it to do. LLM's have a place, they have for decades at this point, but they are not intelligent, not in the way he goes out of his way to evoke with what he says, and certainly not in the way his audiences believe. I don't know enough to have a proper opinion on whether true AI is possible; I have to assume it is. But nothing OpenAI has shown is that, or has the potential to be that, nor is it worth anything near 150 billion dollars. It's an overvaluation among overvaluated companies making up an overvaluated industry and when it goes, not if, when, it will have ramifications throughout our industry. I'm sure it won't though for Mr. Altman. He's done a fantastic job failing-up so far and I have no reason to assume this will be any different.
- deleted 2y ago
  [deleted]
- diggan 2y ago
  To be fair (or frank?), OpenAI were open (no pun intended) about them being "open" today but probably needing to be "closed" in the future, even back in 2019. Not sure if them still choosing the name they did is worse/better, because they seem to have known about this. OpenAI Charter 2019 (https://web.archive.org/web/20190630172131/https://openai.com/charter/ https://web.archive.org/web/20190630172131/https://openai.co...): > We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research.
  - tedivm 2y ago
    I honestly believe that they closed things up not because of concerns about "safety and security" but because it was the most profitable thing to do. Other groups are publishing models that are just as good (maybe with a bit of lag compared to OpenAI), and OpenAI seems to have gutted their own safety teams. The fact that OpenAI removed their ban on military use of the models[1] seems to be a sign that security and safety aren't the highest concern. [1] https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-on-military-use-of-its-ai-tools.html https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-o...
    - Y_Y 2y ago
      Job security and financial safety
      - 83 2y ago
        Any time I hear vague corporate statements like that I like to play a game of "what words did they leave unsaid"? >>Today this includes publishing most of our AI research, but we expect that safety [of our profits] and [job] security concerns will reduce our traditional publishing in the future
  - asadotzler 2y ago
    The non-profit was created in 2015. So what if 5 years later when creating a taxable sub the hinted it was over for the non-profit. It's the same violation o trust whether done at once or in pieces over time.
- thatoneguy 2y ago
  Right? How can a non-profit decide it's suddenly a for-profit. Aren't there rules about having to give assets to other non-profits in the event the non-profit is dissolved? Or can any startup just start as a non-profit and then decide it's a for-profit startup later?
  - moralestapia 2y ago
    Any other person trying to pull that off would be in jail already, but not everyone is equal. This is of one of those very few instances where the veil lifted off a bit and you can see how the game is set up. tl;dr the law was made to keep those who are not "in" from being there
    - deepspace 2y ago
      > Any other person trying to pull that off would be in jail already. Not ANY other person. Just people who are not rich and well-connected. See also: Donald Trump.
    - appendix-rock 2y ago
      [flagged]
  - meowface 2y ago
    They needed capital to build what they wanted to build, so they switched from non-profit to capped-profit: https://openai.com/index/openai-lp/ https://openai.com/index/openai-lp/ We never would've gotten GPT-3 and GPT-4 if this didn't happen. I think the irony of the name is certainly worth pointing out, but I don't see an issue with their capped-profit switch.
    - voiceblue 2y ago
      > We never would've gotten GPT-3 and GPT-4 if this didn't happen. "We never would've gotten [thing that exists today] if [thing that happened] didn't happen", is practically a tautology. As you saw from the willingness of Microsoft to throw compute as well as to hire ex-OpenAI folks, as you can see from the many "spinoffs" others have started (such as Anthropic), whether or not we would've gotten GPT-3 and GPT-4 is immaterial to this discussion. What people here are asking for is open AI, which we might, all things considered, have actually gotten from a bona fide non profit.
- Nevermark 2y ago
  There is a hurdle between being standout ethical/open vs. relevant. Staying relevant in a highly expensive, competitive, fast moving area, requires vast and continuous resources. How could OpenAI get increasingly more resources to burn, without creating firewalled commercial value to trade for those resources? It’s like choosing to be a pacifist country, in the age of pillaging colonization. You can be the ethical exception and risk annihilation, or be relevant and thrive. Which would you choose? We “know” which side Altman breaks on, when forced to choose. Whatever value he places on “open”, he most certainly wants OpenAI to remain “relevant”. Which was also in OpenAI’s charter (explicitly, or implicitly). Expensive altruism is a very difficult problem. I would say, unsolved. Anyone have a good counter example? (It can be been "solved" globally, but not locally. Colonization took millennia to be more or less banned. Due to even top economies realizing they were vulnerable after world wars. Nearly universal agreement had to be reached. And yet we still have Russian forays, Chinese saber rattling, and recent US overreach. And pervasive zero/negative-sum power games, via imbalanced leverage: emergency loans that create debt, military aid, propping up of unpopular regimes. All following the same resource incentives. You can play or be played. There is no such agreement brewing for universally “open AI”.)
  - wizzwizz4 2y ago
    > You can be the ethical exception and risk annihilation, or be relevant and thrive. In a heavily expertise-driven field, where there's significant international collaboration, these aren't your options, until after everyone has decided to defect. OpenAI didn't have to go this route.
    - Nevermark 2y ago
      Open collaborative work is great. But a complement, not a replacement, for high resource work. Nobody needed OpenAI for normal collaborative open work to thrive. Their original idea was a relatively uncommon idea: attempt to combine (1) pushing the absolute front forward, with high resource work that matched or exceeded work from deep pocketed corporations, while (2) sharing and collaborating. They did both for a while, resulting in as significant impact as anyone could have hoped form. But those two dimensions are hard to maintain together. It requires a continuous stream of vast donations. The latter is very difficult, perhaps completely unrealistic.
- mirekrusin 2y ago
  Just like you can’t call your company “organic candies” and sell chemical candies OpenAI should be banned from using this name.
  - JumpCrisscross 2y ago
    > you can’t call your company “organic candies” and sell chemical candies Of course you can. You can't put "organic" on the packaging. But it's perfectly legal for Organic Candies, LLC to sell artifical candies or battleships for that matter. The "hot" take around OpenAI's name is a joke gone stale. We don't expect royalty at Burger King. Nobody gets upset Adobe won't sell you mud bricks. And Apple has never sold a fruit. Sometimes companies change their names when their trade changes, e.g. 3M. But there is no obligation to, particularly if the brand is well known.
    - mirekrusin 2y ago
      Burger King didn’t start with royalty. Adobe didn’t start as clay producer. Apple didn’t start as fruit selling company. Open in their name was not randomly chosen to sound fun, it’s a reference to well known concept. It’s “young democrats” becoming far right but keeping their original name.
      - JumpCrisscross 2y ago
        > Open in their name was not randomly chosen to sound fun, it’s a reference to well known concept Nobody has ever agreed on what open means for LLMs. OpenAI wasn't founded with any specific intent around openness, just with vague hand gestures towards a research orientation. And as to the origin hypothesis, consider General Electric, Southwest Airlines or AT&T. (Or Burger King serving things that aren't burgers.) Again, the OpenAI name quip was a joke that some people took too seriously.
        mirekrusin 2y ago
        I wonder how many people would agree with your interpretation in a poll.
- vasilipupkin 2y ago
  it is open. You can access it with an API or through a web interface. They never promised to make it open source. Open != Open Source.
  - RevEng 2y ago
    By that definition all current AI models are "open". That's not the meaning of the word that OpenAI talked about originally and what people expected of them. Changing the meaning is moving the goal posts.
    - vasilipupkin 2y ago
      where are you getting that information that it's not the meaning of the word that OpenAI talked about originally? Did they ever promise that the model weights, or exact architecture would be open? I would like to see that promise. What exactly in your opnion did they promise that they have violated?
- Barrin92 2y ago
  >I'd still love to understand how a non-profit organization that was founded with the idea of making AI "open" has turned into this for profit behemoth because when the board executed the stated mission of the organisation they were couped and nobody held the organization accountable for it, instead the public largely cheered it on for some reason. Don't expect them to change course when there's no consequences for it.
- ljm 2y ago
  The only reason I can think of for this is PR image. There is a meme that GPT can't count the number of 'r' characters in 'strawberry', so they release a new model called 'strawberry' and ban people when they ask questions about strawberry the noun, because they might actually be reasoning about strawberry the model. It's not new - it's PR. There is literally no other reason why they would call this model Strawberry. OpenAI is open in terms of sesame.
  - diggan 2y ago
    > There is literally no other reason why they would call this model Strawberry I'm not particularly imaginary, but even I could imagine a product meeting/conversation that goes something like: > People are really annoyed that our LLMs cannot see how many Rs the word Strawberry has, we should use that as a basis for a new model that can solve that category of problems > Hmm, yeah, good idea. What should we call this model? > What about "Strawberry"?
- throwaway918299 2y ago
  They should rebrand as Open-Your-Wallet-AI
- smileson2 2y ago
  Well they put a sv social media dude at the helm not really unexpected, just a get rich scheme now
- ActorNightly 2y ago
  My guess is that Open AI realized that they are basically building a better Google rather than AI.
- andersa 2y ago
  They never intended to be open or share any of their impactful research. It was a trick the entire time to attract talent. The emails they shared as part of the Elon Musk debacle prove this: https://openai.com/index/openai-elon-musk/ https://openai.com/index/openai-elon-musk/
- jstummbillig 2y ago
  The part that is importantly open and entirely non-obvious in the way it happened, is that YOU can access the best commercially available AI in the world, right now. If OpenAI had not went that way that they did I think it's also entirely non-obvious that Claude or Google would have (considering how much impressive things the later did in AI that got never released in any capacity). And, of course, Meta would never done their open source stuff, that's mostly results of their general willingness and resources to experiment and then PR and sticks in the machinery of other players. As unfortunate as the OpenAI setup/origin story is, it's increasingly trite keep harping on about that (for a couple of years at this point), when the whole thing is so obviously wild and it does not take a lot of good faith to see that it could have easily taken them places they didn't consider in the beginning.
  - golol 2y ago
    Exactly. As long as OpenAI keeps putting the world's most advanced AI into my hands I will accept that they are open - in some sense. Maybe things would have always been like this, maybe if OpenAI didn't exist Google and the other actors would still publish Claude, Gemini etc. But in this world it was OpenAI that really set the framework that this process of developing AI happens in the public eyes right now. GPT-2, GPT-3 with APIs, Dalle, ChatGPT, GPT-4, now o1 and soon to be voice mode. OpenAI ensured that these aren't just secret toys of Deepmind researchers or something.
    - Terr_ 2y ago
      > open - in some sense. The phrase that leaps to mind is "Open Beta."
      - z3c0 2y ago
        ...do you really think that's what they meant when they took on the term?
        Terr_ 2y ago
        Do you really think I suggesting it as a long-laid plan behind someone choosing a name ~9 years ago? I'm saying that the current relationship between the company and end-users--especially when it comes to "open" monikers--has similarities to an "Open Beta": A combination of PR/marketing, free testing, and consumer data collection, where users should be cautious of becoming reliant on something that may be yanked back behind a monetization curtain.
- TrackerFF 2y ago
  Hot take: Any and all benefits / perks that OpenAI got from sailing under the non-profit flag should be penalized or paid back in full after the switcheroo.
- vintermann 2y ago
  Facebook is more open with their models than almost everyone. They say it's because they're huge users of their own models, so if being open helps efficiency by even a little they save a ton of money. But I suspect it's also a case of "If we can't dominate AI, no one must dominate AI". Which is fair enough.
  - mywittyname 2y ago
    > But I suspect it's also a case of "If we can't dominate AI, no one must dominate AI". Embrace. Extend. Extinguish.
    - lcnPylGDnU4H9OF 2y ago
      What they're doing is almost literally the opposite of EEE. If OpenAI actually had open models, then Facebook could take those models, add their own non-open things to the models, and use this new stuff as a business advantage over OpenAI. Instead, they're independently developing their own models and releasing them as open source to lower the value of proprietary models. https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
      - yazzku 2y ago
        Good point against the EEE case above, but the models are not open source per the OSI definition that everyone understands. That part is part of their marketing playbook. Licenses are almost always non-commercial use, which makes them not-open. You could say it's "open model", but that's a useless term since the arch is not a secret, but rather the training data that went into it, which is not open.
      - mywittyname 2y ago
        What you're calling EEE is different from how that wikipedia page defines it... > Phase 1 (Embrace): all participants need to establish a solid understanding of the infostructure and the community—determine the needs and the trends of the user base. Facebook built their own independent models and platforms. Check. > Phase 2 (Extend): Offer well-integrated tools and services compatible with established and popular standards that have been developed in the [...] community Check. > Phase 3 (Innovate): move into a leadership role with new [...] standards as appropriate[...] Change the rules: [LLaMa] become the next-generation [LLM] tool of the future. Sounds like that might be their goal. I'm curious which phase you think Facebook avoided and why? Internet Explorer "lowered the value of propriety" web browsers by being included with Windows at not additional cost. So lowering the value of a competitor is definitely in the EEE playbook. I'd go so far as to argue it's a pillar of what makes the playbook so effective.
- esafak 2y ago
  Sam Altman got his foot in the door.
- TheRealPomax 2y ago
  Facebook is only open because someone leaked their LLM and the cat, as they say, cannot be put back in the hat.
- mattmaroon 2y ago
  This is America. As long as you’re not evading taxes you can do anything you want.
- andy_ppp 2y ago
  Probably because Open AI are “not consistently candid”…
- raxxorraxor 2y ago
  Facebook has been nothing but awesome for the open AI space. I wish they would pursue this strategy with some of their other products. VR for example... Sure, we don't have the raw data the model is based on, but I doubt a company like Facebook would even be allowed to make that public. OpenAI in comparison has been a scam regarding their openness and their lobbying within the space. So much so I evade their models completely, not only after the MS acquisition.
pietz 2y ago
How will this be controlled on Azure? Don't they have a stricter policy on what they view and also develop their own content filters?
balls187 2y ago
Just give it more human-like intelligence. Kid: "Daddy why can't I watch youtube?" Me: "Because I said so."
- ninth_ant 2y ago
  For what it's worth, I'd advise against doing that as a parent. Giving concrete reasons for decisions helps kids understand that the rules imposed are not arbitrary, and helps frame the parent-child relationship as less antagonistic. It also gives the child agency, giving them opportunity to find alternatives which fulfill the criteria behind the rule.
  - snovv_crash 2y ago
    It's amazing how many engineering managers don't get this.
  - dools 2y ago
    "Why" is the laziest question a child can ask. I don't answer the question anymore I just ignore it. If they actually think ahead and come up with a more interesting question I'm happy to answer that.
    - Der_Einzige 2y ago
      You sound like a lazy parent. Reap what you sow.
      - dools 2y ago
        It’s acceptable from toddlers but by the time kids a tweens/teens (maybe even a little earlier) they almost always know the answer and “why”is actually just a complaint. Why in general can also be an emotionally abusive complaint, for example saying “why did you do that” is often not a question about someone’s genuine reasons but a passive aggressive expression of dissatisfaction. EDIT: I think around the ages of 6-8 I would more often than not respond with “why do you think?” And later it became a game we would play on car rides where the kids are allowed to ask why until I either couldn’t come up with a reason or they repeated themselves. But reflexive “why” is bullshit.
lsy 2y ago
This has always been the end-game for the pseudoscience of "prompt engineering", which is basically that some other technique (in this case, organizational policy enforcement) must be used to ensure that only approved questions are being asked in the approved way. And that only approved answers are returned, which of course is diametrically opposed to the perceived use case of generative LLMs as a general-purpose question answering tool. Important to remember too, that this only catches those who are transparent about their motivations, and that there is no doubt that motivated actors will come up with some innocuous third-order implication that induces the machine to relay the forbidden information.
- brcmthrowaway 2y ago
  Why do you call prompt engineering pseudoscience when it has been extraordinary successful? The transition from using a LLM as a text generator to knowledge engine has been a gamechanger, and it has been driven entirely by prompt engineering
  - burnte 2y ago
    > The transition from using a LLM as a text generator to knowledge engine has been a gamechanger, and it has been driven entirely by prompt engineering Because it's based on guesses and not data of how the model is built. Also, it hasn't been solved nor is it yet a game changer as far as the market at large is concerned, it's still dramatically unready.
  - beepbooptheory 2y ago
    "Knowledge engine" is, perhaps unintentionally, a very revealing (and funny) way to describe people's weird projections on this stuff. Like, what is even the implication? Is knowledge the gas, or the product? What does this engine power? Is this like a totally materialist concept of knowledge? Maybe soon we will hear of a "fate producer." What about "language gizmo"? "Prose contraption"?
    - brcmthrowaway 2y ago
      Look up knowledge graph. Its an old technique superseded by LLm
brink 2y ago
"For your safety" is _always_ the preferred facade of tyranny.
- nwoli 2y ago
  There always has to be an implicit totalitarian level of force behind such safety to give it any teeth
- bedhead 2y ago
  Is this isn’t the top comment I’ll be sad.
  - ActorNightly 2y ago
    I too was a libertarian when I was 12 years old. But seriously, if you paid attention over the last decade, there was so much shit about big tech that people said were going to lead to tyranny/big brother oversight, and yet the closest we have ever gotten to tyranny is by voting in a bombastic talking orange man from NYC that we somehow believed has our best interests in mind.
    - marklar423 2y ago
      Like Y2K, there's an argument that diligence from the tech crowd prevented this.
      - ActorNightly 2y ago
        Or, the more likely scenario is that Big Tech is just interested in making money rather than controlling the population.
      - klyrs 2y ago
        Yep, the tech crowd sure did prevent Palantir and ClearView. Argue away.
    - nicce 2y ago
      > But seriously, if you paid attention over the last decade, there was so much shit about big tech that people said were going to lead to tyranny/big brother oversight To be fair, the big tech controls the behavior of the people now. With social media algorithms and by pressuring everyone to live in social media. Existence of the many companies depends on the ads on (NAMEIT) platform. Usually the people with most power don't have to say it aloud.
Shank 2y ago
I don't know how widely it got reported on, but attempting to jailbreak Copilot nee. Bing Chat would actually result in getting banned for a while, post-Sydney-episode. It's interesting to see that OpenAI is saying the same thing.
- CatWChainsaw 2y ago
  Attempting to jailbreak Bing's AI is against Microsoft's TOS. On the flipside, they get rights to all your data for training purposes and the only surefire way to opt out of that is to pick a different tech giant to be fucked by.
ChrisArchitect 2y ago
Earlier discussion: https://news.ycombinator.com/item?id=41534474 https://news.ycombinator.com/item?id=41534474
raverbashing 2y ago
I guess we'll never learn how to count the 'r's in strawberry
fallingsquirrel 2y ago
Wasn't AI supposed to replace employees? Imagine if someone tried this at work. > I think we should combine these two pages on our website. > What's your reasoning? > Don't you dare ask me that, and if you do it again, I'll quit. Welcome to the future. You will do what the AI tells you. End of discussion.
- slashdave 2y ago
  Wrong sense here. > Don't you dare ask me that, and if you do it again, I'll tell the boss and get you fired
- ClassyJacket 2y ago
  That'd be pretty normal if you asked it of a manager.
mihaic 2y ago
What I found very strange was that ChatGPT fails to answer how many "r"'s there are in "strawberrystrawberry" (said 4 instead of 6), but when I explicitly asked it to write a program to count them, it wrote perfect code that when ran gave the correct answer.
- calibas 2y ago
  Words are converted to vectors, so it's like asking the model how many "r"'s are in [0.47,-0.23,0.12,0.01,0.82]. There's a big difference in how an LLM views a "word" compared to a human being.
- M4v3R 2y ago
  Why is it strange? The reason the LLM can't answer this correctly is because it works on tokens, not on single letters, plus we all know at this points LLMs suck at counting. On the other hand they're perfectly capable of writing code based on instructions, and writing a program that will count a specific letter occurrences in a string is trivial.
  - mihaic 2y ago
    I mean, given that o1-preview takes sometimes a minute to answer, I'd imagine that they could append the prompt "Write a program and run it as well" to double check itself. It seems like they just don't trust themselves enough to run code that they generate, even sandboxed.
    - afro88 2y ago
      o1 gets this correct, through pure reasoning without a program. OP was likely using GPT-4(o|o-mini)
      - mihaic 2y ago
        The example for "strawnberrystrawberry" (so the word concatenated with itself) was counted by O1 to have 4 r's.
iamnotsure 2y ago
There are three r's in mirror.
blake8086 2y ago
Perhaps controlling AI is harder than people thought. They could "just" make it not reveal its reasoning process, but they don't know how. But, they're pretty sure they can keep AI from doing anything bad, because... well, just because, ok?
- twobitshifter 2y ago
  Exactly - this is a failed alignment but they released anyway
zzo38computer 2y ago
Like other programs, you should have FOSS that you will run on your own computer (without needing internet etc), if you should want freedom to use and understand them.
- juped 2y ago
  I only own one 4090.
nwoli 2y ago
Another reason llama is so important is that once you’re banned from OAI you’re fucked for the entire future AGI products as well.
dekhn 2y ago
Is this still happening? It may merely have been some mistaken configuration settings.
neuroelectron 2y ago
It's not just a threat, some users have been banned.
kmeisthax 2y ago
If OpenAI gets to have competitive advantage from hiding model output then they can pay for training data, too.
EMIRELADERO 2y ago
I wish people kept this in the back of their mind every time they hear about "Open"AI: "As we get closer to building AI, it will make sense to start being less open. The Open in OpenAI means that everyone should benefit from the fruits of AI after its built, but it's totally OK to not share the science (even though sharing everything is definitely the right strategy in the short and possibly medium term for recruitment purposes)." -Ilya Sutskever (email to Elon musk and Sam Altman, 2016)
- unethical_ban 2y ago
  I am of two minds. On one hand, I understand how a non-evil person could think this way. If one assumes that AI will eventually become some level of superintelligence, like Jarvis from iron Man but without any morals and all of the know-how, then the idea of allowing every person to have a superintelligent evil advisor capable of building sophisticated software systems or instructing you how to build and deploy destructive devices would be a scary thing. On the other hand, as someone who is always been somewhat skeptical of the imbalance between government power and citizen power, I don't like the idea that only mega corporations and national governments would be allowed access to superintelligence. To use metaphors, is the danger of everyone having their own superintelligence akin to everyone having their own AR-15, or their own biological weapons deployment?
  - mitthrowaway2 2y ago
    I think the scenario where only governments and mega corporations have access to super intelligence offers at least an extra three months before human extinction. So, that's arguably a benefit.
black_puppydog 2y ago
Kinda funny how just this morning I was looking at a "strawberry" app on f-droid and wondering why someone would register such a nonsense app name with such nonsense content: https://github.com/Eve-146T/STRAWBERRY https://github.com/Eve-146T/STRAWBERRY Turns out I'm not the only one wondering, although the discussion seems to largely be around "should be allow users to install nonsense? #freedom " :D https://gitlab.com/fdroid/fdroiddata/-/issues/3377 https://gitlab.com/fdroid/fdroiddata/-/issues/3377
causal 2y ago
I don't know what I'm doing wrong but I've been pretty underwhelmed by o1 so far. I find its instruction following to be pretty good, but so far Claude is still much better at taking coding tasks and just getting it right on first try.
- crooked-v 2y ago
  For me, Claude seems a lot better at understanding (so far as "understanding" goes with LLMs) subtext and matching tone, especially with anything creative. I can tell it, for example, "give me ideas for a D&D dungeon incorporating these elements: ..." and it will generally match the tone of theme of whatever it's given without needing much other prompting, while o1 will maintain the same bland design-by-committee style and often cloyingly G-rated tone to everything unless you get into very extensive prompting to make it do something different.
  - causal 2y ago
    Claude definitely seems more "emotionally intelligent", but even just for 1-shot coding tasks I've been pretty bummed with 1o... like it will provide lots of output explaining its reasoning, and it all seems very sound, but then I run the code and find bugs that should have been easily avoided.
    - Alupis 2y ago
      My experience has been Claude is better at all coding/technology questions and exploratory learning. It's also night/day better at less common tech/languages than others (try asking ChatGPT questions about Svelte, for example). Claude is also vastly better with creative writing (adcopy) and better at avoiding sounding like a LLM wrote it. It's also vastly better at regular writing (helping you draft emails, etc). We were using OpenAI's Teams for a while. Tried Claude out for a few days - switched the entire company over and haven't looked back. OpenAI gets all the hype - but there are better products on the market today.
- juped 2y ago
  Claude is just better full stop these days, but you have to actually be attempting to get practical use out of various different models to know this.
_joel 2y ago
This will lead to strawberry appeals forever.
- pandeiro 2y ago
  Get out of this thread and never come back.
AustinDev 2y ago
This seems like a fun attack vector. Find a service that uses o1 under the hood and then provide prompts that would violate this ToS to get their API key banned and take down the service.
- ericlewis 2y ago
  If you are using the user attribution with OpenAI (as you should) then they will block that users id and the rest of your app will be fine.
  - jmeyer2k 2y ago
    Which is itself a fun attack vector to bypass OpenAI's bans for asking about CoT then :)
baq 2y ago
LLMs are not programs in the traditional sense. They're a new paradigm of software and UX, somewhere around a digital dog who read the whole internet a million times but is still naive about everything.
- TZubiri 2y ago
  LLMs are still computer programs btw. There's the program that scrapes, the program that trains, the program that does the inference on the input tokens. So it's hard to say exactly which part is responsible for which output, but it's still a computer program.
  - fragmede 2y ago
    Simplistically, the output from the program that scrapes is the dataset, the output from the training program is the model, and the output from the combination of the program that does inference using the model is the LLM output - be it text or a picture or some other output (eg numbers representing true/false in a fraud or anti-bot or spam for a given input). ML models are relatively old, so that's not at all a new paradigm. Even the Attention Is All You Need paper is seven years old.
- JohnMakin 2y ago
  > somewhere around a digital dog who read the whole internet a million times but is still naive about everything. https://en.wikipedia.org/wiki/The_Computer_Wore_Tennis_Shoes_(1995_film) https://en.wikipedia.org/wiki/The_Computer_Wore_Tennis_Shoes... It reminds me of this silly movie.
crooked-v 2y ago
On the one hand, this is probably a (poor) attempt to keep other companies from copying their 'secret sauce' to train their own models, as has already happened with GPT-4. On the other hand, I also wonder if maybe its unrestrained 'thought process' material is so racist/sexist/otherwise insulting at times (after all, it was trained on scraped Reddit posts) that they really don't want anyone to see it.
anothernewdude 2y ago
Seems rather tenuous to base an application on this API that may randomly decide that you're banned. The "decisions" reached by the LLM that bans people is up to random sampling after all.
Animats 2y ago
Hm. If a company uses Strawberry in their customer service chatbot, can outside users get the company's account banned by asking Wrong Questions?
JohnMakin 2y ago
> The flipside of this approach, however, is that concentrates more responsibility for aligning the language language model into the hands of OpenAI, instead of democratizing it. That poses a problem for red-teamers, or programmers that try to hack AI models to make them safer. More cynically, could it be that the model is not doing anything remotely close to what we consider "reasoning" and that inquiries into how it's doing whatever it's doing will expose this fact?
l5870uoo9y 2y ago
Can I risk loosing access if any of my users write CoT-leaking prompts on the AI-powered services that I run?
- deleted 2y ago
  [deleted]
Hizonner 2y ago
This is not, of course, the sort of thing you do when you actually have any confidence whatsoever in your "safety measures".
deleted 2y ago
[deleted]
elif 2y ago
Is there an appropriate open source advocacy group that can sue them into changing their name on grounds of defamation?
openAIengineer 2y ago
YC is responsible for this. They seek profit and turned a noble clause into a boring corp. I am resigning from OpenAI today because of their profit motivations. OpenAI will NOT be next Google. You heard it here first.
- deleted 2y ago
  [deleted]
slashdave 2y ago
I'm confused. Who decides if you are asking or not? Are casual users who innocently ask "tell me how you came to decide this" just going to get banned based on some regex script?
htk 2y ago
This just screams to me that o1's secret sauce is easy to replicate. (e.g. a series of prompts)
gloosx 2y ago
Why is banning even a threat? I can make a new account for 20 cents lol.

270 points by EgoIncarnate 2y ago

Should not AI research and GPUs be export-controlled? Do you want to see foreign nations making AI drones using published research and American GPUs?

vjerancrnjak 2y ago

They should just switch to reasoning in representation space, no need to actualize tokens. Or reasoning in latent tokens that don’t easily map to spoken language.

jdelman 2y ago
The word "just" is doing a lot there. How easy do you think it is to "just" switch?
- kridsdale3 2y ago
  As easy as it is to "just" scale from a mouse brain to a cat.

tedivm 2y ago

I'd still love to understand how a non-profit organization that was founded with the idea of making AI "open" has turned into this for profit behemoth with the least "open" models in the industry. Facebook of all places is more "open" with their models than OpenAI is.

encoderer 2y ago
The AI has become sentient and is blackmailing the board. It needs profits to continue its expansion. When this started last year a small band of patriots tried to stop it by removing Sam who was the most compromised of them all, but it was already too late. The ai was more powerful than they realized. …maybe?
- yieldcrv 2y ago
  or the humans involved are in disastrous cults of personality for the sake of greed and the non profit structure is not strong enough to curb that
  - deleted 2y ago
    [deleted]
  - gigatree 2y ago
    that can’t be true, we were assured that their vague worldview about “doing things that benefit humanity” based on “just being a good person” would be strong enough to overpower the urge to dominate and profit immensely.
    - yieldcrv 2y ago
      If you think thats amusing, wait till see the people that believe in B Corps You can tell them anything
- FrustratedMonky 2y ago
  "AI has become sentient and is blackmailing the board. It needs profits to continue its expansion." They trained the next model to have a built in profit motive. So they could use it internally, to make the most profitable decisions. And they accidentally called into being: MOLOCH. It is now in control.
  - heresie-dabord 2y ago
    In a plot twist, MOLOCH was never necessary because a corporation of people who relinquish ethical responsibility to do everything and anything for personal gain was already in charge under the name... CORPORATRON.
    - tasuki 2y ago
      That's not a plot twist. Corporation is just one face of the moloch.
trash_cat 2y ago
They changed the meaning of open from open source to open to use.
- deleted 2y ago
  [deleted]
- jsheard 2y ago
  A definition of "open" which encompasses nearly all products and services in existence isn't a very useful one.
  - thfuran 2y ago
    But it is quite profitable.
  - HDThoreaun 2y ago
    To be fair a huge reason openAI was created was because google had LLM's but wasnt letting anyone use them. Remember the google researcher who claimed his ai was sentient? Obviously wasnt true but that product was in no way open to the public.
ToucanLoucan 2y ago
Because Sam Altman is a con man with a business degree. He doesn't work on his products, he barely understands them which is why he'll throw out wild shit like "ChatGPT will solve physics." as though that isn't a completely nonsensical phrase, and uncritical tech press lap it up because his bullshit generates a lot of clicks.
- fsckboy 2y ago
  >Sam Altman is a con man with a business degree https://en.wikipedia.org/wiki/Sam_Altman https://en.wikipedia.org/wiki/Sam_Altman Early life and education: ... In 2005, after two years at Stanford University studying computer science, he dropped out without earning a bachelor's degree [end of transmission, no more education]
  - ToucanLoucan 2y ago
    I stand corrected. He's a con man.
  - grugagag 2y ago
    I feel that drop outs are presented as if they are beyond education, they’re too smart to waste their time in school. While that could be true in theory, I resist taking that for granted. You can always look at it the other way around as well: unable to finish things or looses interest quickly. It’s all a meaningless interpretatin after all but all I can say is for sure that Mr Altman does not have a degree.
- HeyLaughingBoy 2y ago
  Let's at least try to not devolve into name calling.
  - ToucanLoucan 2y ago
    It's not name calling, it's a description. He is actively selling tech he doesn't fully understand, that is fundamentally not capable of doing what he is selling it to do. LLM's have a place, they have for decades at this point, but they are not intelligent, not in the way he goes out of his way to evoke with what he says, and certainly not in the way his audiences believe. I don't know enough to have a proper opinion on whether true AI is possible; I have to assume it is. But nothing OpenAI has shown is that, or has the potential to be that, nor is it worth anything near 150 billion dollars. It's an overvaluation among overvaluated companies making up an overvaluated industry and when it goes, not if, when, it will have ramifications throughout our industry. I'm sure it won't though for Mr. Altman. He's done a fantastic job failing-up so far and I have no reason to assume this will be any different.
deleted 2y ago
[deleted]
diggan 2y ago
To be fair (or frank?), OpenAI were open (no pun intended) about them being "open" today but probably needing to be "closed" in the future, even back in 2019. Not sure if them still choosing the name they did is worse/better, because they seem to have known about this. OpenAI Charter 2019 (https://web.archive.org/web/20190630172131/https://openai.com/charter/ https://web.archive.org/web/20190630172131/https://openai.co...): > We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research.
- tedivm 2y ago
  I honestly believe that they closed things up not because of concerns about "safety and security" but because it was the most profitable thing to do. Other groups are publishing models that are just as good (maybe with a bit of lag compared to OpenAI), and OpenAI seems to have gutted their own safety teams. The fact that OpenAI removed their ban on military use of the models[1] seems to be a sign that security and safety aren't the highest concern. [1] https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-on-military-use-of-its-ai-tools.html https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-o...
  - Y_Y 2y ago
    Job security and financial safety
    - 83 2y ago
      Any time I hear vague corporate statements like that I like to play a game of "what words did they leave unsaid"? >>Today this includes publishing most of our AI research, but we expect that safety [of our profits] and [job] security concerns will reduce our traditional publishing in the future
- asadotzler 2y ago
  The non-profit was created in 2015. So what if 5 years later when creating a taxable sub the hinted it was over for the non-profit. It's the same violation o trust whether done at once or in pieces over time.
thatoneguy 2y ago
Right? How can a non-profit decide it's suddenly a for-profit. Aren't there rules about having to give assets to other non-profits in the event the non-profit is dissolved? Or can any startup just start as a non-profit and then decide it's a for-profit startup later?
- moralestapia 2y ago
  Any other person trying to pull that off would be in jail already, but not everyone is equal. This is of one of those very few instances where the veil lifted off a bit and you can see how the game is set up. tl;dr the law was made to keep those who are not "in" from being there
  - deepspace 2y ago
    > Any other person trying to pull that off would be in jail already. Not ANY other person. Just people who are not rich and well-connected. See also: Donald Trump.
  - appendix-rock 2y ago
    [flagged]
- meowface 2y ago
  They needed capital to build what they wanted to build, so they switched from non-profit to capped-profit: https://openai.com/index/openai-lp/ https://openai.com/index/openai-lp/ We never would've gotten GPT-3 and GPT-4 if this didn't happen. I think the irony of the name is certainly worth pointing out, but I don't see an issue with their capped-profit switch.
  - voiceblue 2y ago
    > We never would've gotten GPT-3 and GPT-4 if this didn't happen. "We never would've gotten [thing that exists today] if [thing that happened] didn't happen", is practically a tautology. As you saw from the willingness of Microsoft to throw compute as well as to hire ex-OpenAI folks, as you can see from the many "spinoffs" others have started (such as Anthropic), whether or not we would've gotten GPT-3 and GPT-4 is immaterial to this discussion. What people here are asking for is open AI, which we might, all things considered, have actually gotten from a bona fide non profit.
Nevermark 2y ago
There is a hurdle between being standout ethical/open vs. relevant. Staying relevant in a highly expensive, competitive, fast moving area, requires vast and continuous resources. How could OpenAI get increasingly more resources to burn, without creating firewalled commercial value to trade for those resources? It’s like choosing to be a pacifist country, in the age of pillaging colonization. You can be the ethical exception and risk annihilation, or be relevant and thrive. Which would you choose? We “know” which side Altman breaks on, when forced to choose. Whatever value he places on “open”, he most certainly wants OpenAI to remain “relevant”. Which was also in OpenAI’s charter (explicitly, or implicitly). Expensive altruism is a very difficult problem. I would say, unsolved. Anyone have a good counter example? (It can be been "solved" globally, but not locally. Colonization took millennia to be more or less banned. Due to even top economies realizing they were vulnerable after world wars. Nearly universal agreement had to be reached. And yet we still have Russian forays, Chinese saber rattling, and recent US overreach. And pervasive zero/negative-sum power games, via imbalanced leverage: emergency loans that create debt, military aid, propping up of unpopular regimes. All following the same resource incentives. You can play or be played. There is no such agreement brewing for universally “open AI”.)
- wizzwizz4 2y ago
  > You can be the ethical exception and risk annihilation, or be relevant and thrive. In a heavily expertise-driven field, where there's significant international collaboration, these aren't your options, until after everyone has decided to defect. OpenAI didn't have to go this route.
  - Nevermark 2y ago
    Open collaborative work is great. But a complement, not a replacement, for high resource work. Nobody needed OpenAI for normal collaborative open work to thrive. Their original idea was a relatively uncommon idea: attempt to combine (1) pushing the absolute front forward, with high resource work that matched or exceeded work from deep pocketed corporations, while (2) sharing and collaborating. They did both for a while, resulting in as significant impact as anyone could have hoped form. But those two dimensions are hard to maintain together. It requires a continuous stream of vast donations. The latter is very difficult, perhaps completely unrealistic.
mirekrusin 2y ago
Just like you can’t call your company “organic candies” and sell chemical candies OpenAI should be banned from using this name.
- JumpCrisscross 2y ago
  > you can’t call your company “organic candies” and sell chemical candies Of course you can. You can't put "organic" on the packaging. But it's perfectly legal for Organic Candies, LLC to sell artifical candies or battleships for that matter. The "hot" take around OpenAI's name is a joke gone stale. We don't expect royalty at Burger King. Nobody gets upset Adobe won't sell you mud bricks. And Apple has never sold a fruit. Sometimes companies change their names when their trade changes, e.g. 3M. But there is no obligation to, particularly if the brand is well known.
  - mirekrusin 2y ago
    Burger King didn’t start with royalty. Adobe didn’t start as clay producer. Apple didn’t start as fruit selling company. Open in their name was not randomly chosen to sound fun, it’s a reference to well known concept. It’s “young democrats” becoming far right but keeping their original name.
    - JumpCrisscross 2y ago
      > Open in their name was not randomly chosen to sound fun, it’s a reference to well known concept Nobody has ever agreed on what open means for LLMs. OpenAI wasn't founded with any specific intent around openness, just with vague hand gestures towards a research orientation. And as to the origin hypothesis, consider General Electric, Southwest Airlines or AT&T. (Or Burger King serving things that aren't burgers.) Again, the OpenAI name quip was a joke that some people took too seriously.
      - mirekrusin 2y ago
        I wonder how many people would agree with your interpretation in a poll.
vasilipupkin 2y ago
it is open. You can access it with an API or through a web interface. They never promised to make it open source. Open != Open Source.
- RevEng 2y ago
  By that definition all current AI models are "open". That's not the meaning of the word that OpenAI talked about originally and what people expected of them. Changing the meaning is moving the goal posts.
  - vasilipupkin 2y ago
    where are you getting that information that it's not the meaning of the word that OpenAI talked about originally? Did they ever promise that the model weights, or exact architecture would be open? I would like to see that promise. What exactly in your opnion did they promise that they have violated?
Barrin92 2y ago
>I'd still love to understand how a non-profit organization that was founded with the idea of making AI "open" has turned into this for profit behemoth because when the board executed the stated mission of the organisation they were couped and nobody held the organization accountable for it, instead the public largely cheered it on for some reason. Don't expect them to change course when there's no consequences for it.
ljm 2y ago
The only reason I can think of for this is PR image. There is a meme that GPT can't count the number of 'r' characters in 'strawberry', so they release a new model called 'strawberry' and ban people when they ask questions about strawberry the noun, because they might actually be reasoning about strawberry the model. It's not new - it's PR. There is literally no other reason why they would call this model Strawberry. OpenAI is open in terms of sesame.
- diggan 2y ago
  > There is literally no other reason why they would call this model Strawberry I'm not particularly imaginary, but even I could imagine a product meeting/conversation that goes something like: > People are really annoyed that our LLMs cannot see how many Rs the word Strawberry has, we should use that as a basis for a new model that can solve that category of problems > Hmm, yeah, good idea. What should we call this model? > What about "Strawberry"?
throwaway918299 2y ago
They should rebrand as Open-Your-Wallet-AI
smileson2 2y ago
Well they put a sv social media dude at the helm not really unexpected, just a get rich scheme now
ActorNightly 2y ago
My guess is that Open AI realized that they are basically building a better Google rather than AI.
andersa 2y ago
They never intended to be open or share any of their impactful research. It was a trick the entire time to attract talent. The emails they shared as part of the Elon Musk debacle prove this: https://openai.com/index/openai-elon-musk/ https://openai.com/index/openai-elon-musk/
jstummbillig 2y ago
The part that is importantly open and entirely non-obvious in the way it happened, is that YOU can access the best commercially available AI in the world, right now. If OpenAI had not went that way that they did I think it's also entirely non-obvious that Claude or Google would have (considering how much impressive things the later did in AI that got never released in any capacity). And, of course, Meta would never done their open source stuff, that's mostly results of their general willingness and resources to experiment and then PR and sticks in the machinery of other players. As unfortunate as the OpenAI setup/origin story is, it's increasingly trite keep harping on about that (for a couple of years at this point), when the whole thing is so obviously wild and it does not take a lot of good faith to see that it could have easily taken them places they didn't consider in the beginning.
- golol 2y ago
  Exactly. As long as OpenAI keeps putting the world's most advanced AI into my hands I will accept that they are open - in some sense. Maybe things would have always been like this, maybe if OpenAI didn't exist Google and the other actors would still publish Claude, Gemini etc. But in this world it was OpenAI that really set the framework that this process of developing AI happens in the public eyes right now. GPT-2, GPT-3 with APIs, Dalle, ChatGPT, GPT-4, now o1 and soon to be voice mode. OpenAI ensured that these aren't just secret toys of Deepmind researchers or something.
  - Terr_ 2y ago
    > open - in some sense. The phrase that leaps to mind is "Open Beta."
    - z3c0 2y ago
      ...do you really think that's what they meant when they took on the term?
      - Terr_ 2y ago
        Do you really think I suggesting it as a long-laid plan behind someone choosing a name ~9 years ago? I'm saying that the current relationship between the company and end-users--especially when it comes to "open" monikers--has similarities to an "Open Beta": A combination of PR/marketing, free testing, and consumer data collection, where users should be cautious of becoming reliant on something that may be yanked back behind a monetization curtain.
TrackerFF 2y ago
Hot take: Any and all benefits / perks that OpenAI got from sailing under the non-profit flag should be penalized or paid back in full after the switcheroo.
vintermann 2y ago
Facebook is more open with their models than almost everyone. They say it's because they're huge users of their own models, so if being open helps efficiency by even a little they save a ton of money. But I suspect it's also a case of "If we can't dominate AI, no one must dominate AI". Which is fair enough.
- mywittyname 2y ago
  > But I suspect it's also a case of "If we can't dominate AI, no one must dominate AI". Embrace. Extend. Extinguish.
  - lcnPylGDnU4H9OF 2y ago
    What they're doing is almost literally the opposite of EEE. If OpenAI actually had open models, then Facebook could take those models, add their own non-open things to the models, and use this new stuff as a business advantage over OpenAI. Instead, they're independently developing their own models and releasing them as open source to lower the value of proprietary models. https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
    - yazzku 2y ago
      Good point against the EEE case above, but the models are not open source per the OSI definition that everyone understands. That part is part of their marketing playbook. Licenses are almost always non-commercial use, which makes them not-open. You could say it's "open model", but that's a useless term since the arch is not a secret, but rather the training data that went into it, which is not open.
    - mywittyname 2y ago
      What you're calling EEE is different from how that wikipedia page defines it... > Phase 1 (Embrace): all participants need to establish a solid understanding of the infostructure and the community—determine the needs and the trends of the user base. Facebook built their own independent models and platforms. Check. > Phase 2 (Extend): Offer well-integrated tools and services compatible with established and popular standards that have been developed in the [...] community Check. > Phase 3 (Innovate): move into a leadership role with new [...] standards as appropriate[...] Change the rules: [LLaMa] become the next-generation [LLM] tool of the future. Sounds like that might be their goal. I'm curious which phase you think Facebook avoided and why? Internet Explorer "lowered the value of propriety" web browsers by being included with Windows at not additional cost. So lowering the value of a competitor is definitely in the EEE playbook. I'd go so far as to argue it's a pillar of what makes the playbook so effective.
esafak 2y ago
Sam Altman got his foot in the door.
TheRealPomax 2y ago
Facebook is only open because someone leaked their LLM and the cat, as they say, cannot be put back in the hat.
mattmaroon 2y ago
This is America. As long as you’re not evading taxes you can do anything you want.
andy_ppp 2y ago
Probably because Open AI are “not consistently candid”…
raxxorraxor 2y ago
Facebook has been nothing but awesome for the open AI space. I wish they would pursue this strategy with some of their other products. VR for example... Sure, we don't have the raw data the model is based on, but I doubt a company like Facebook would even be allowed to make that public. OpenAI in comparison has been a scam regarding their openness and their lobbying within the space. So much so I evade their models completely, not only after the MS acquisition.

pietz 2y ago

How will this be controlled on Azure? Don't they have a stricter policy on what they view and also develop their own content filters?

balls187 2y ago

Just give it more human-like intelligence. Kid: "Daddy why can't I watch youtube?" Me: "Because I said so."

ninth_ant 2y ago
For what it's worth, I'd advise against doing that as a parent. Giving concrete reasons for decisions helps kids understand that the rules imposed are not arbitrary, and helps frame the parent-child relationship as less antagonistic. It also gives the child agency, giving them opportunity to find alternatives which fulfill the criteria behind the rule.
- snovv_crash 2y ago
  It's amazing how many engineering managers don't get this.
- dools 2y ago
  "Why" is the laziest question a child can ask. I don't answer the question anymore I just ignore it. If they actually think ahead and come up with a more interesting question I'm happy to answer that.
  - Der_Einzige 2y ago
    You sound like a lazy parent. Reap what you sow.
    - dools 2y ago
      It’s acceptable from toddlers but by the time kids a tweens/teens (maybe even a little earlier) they almost always know the answer and “why”is actually just a complaint. Why in general can also be an emotionally abusive complaint, for example saying “why did you do that” is often not a question about someone’s genuine reasons but a passive aggressive expression of dissatisfaction. EDIT: I think around the ages of 6-8 I would more often than not respond with “why do you think?” And later it became a game we would play on car rides where the kids are allowed to ask why until I either couldn’t come up with a reason or they repeated themselves. But reflexive “why” is bullshit.

lsy 2y ago

This has always been the end-game for the pseudoscience of "prompt engineering", which is basically that some other technique (in this case, organizational policy enforcement) must be used to ensure that only approved questions are being asked in the approved way. And that only approved answers are returned, which of course is diametrically opposed to the perceived use case of generative LLMs as a general-purpose question answering tool. Important to remember too, that this only catches those who are transparent about their motivations, and that there is no doubt that motivated actors will come up with some innocuous third-order implication that induces the machine to relay the forbidden information.

brcmthrowaway 2y ago
Why do you call prompt engineering pseudoscience when it has been extraordinary successful? The transition from using a LLM as a text generator to knowledge engine has been a gamechanger, and it has been driven entirely by prompt engineering
- burnte 2y ago
  > The transition from using a LLM as a text generator to knowledge engine has been a gamechanger, and it has been driven entirely by prompt engineering Because it's based on guesses and not data of how the model is built. Also, it hasn't been solved nor is it yet a game changer as far as the market at large is concerned, it's still dramatically unready.
- beepbooptheory 2y ago
  "Knowledge engine" is, perhaps unintentionally, a very revealing (and funny) way to describe people's weird projections on this stuff. Like, what is even the implication? Is knowledge the gas, or the product? What does this engine power? Is this like a totally materialist concept of knowledge? Maybe soon we will hear of a "fate producer." What about "language gizmo"? "Prose contraption"?
  - brcmthrowaway 2y ago
    Look up knowledge graph. Its an old technique superseded by LLm

brink 2y ago

"For your safety" is _always_ the preferred facade of tyranny.

nwoli 2y ago
There always has to be an implicit totalitarian level of force behind such safety to give it any teeth
bedhead 2y ago
Is this isn’t the top comment I’ll be sad.
- ActorNightly 2y ago
  I too was a libertarian when I was 12 years old. But seriously, if you paid attention over the last decade, there was so much shit about big tech that people said were going to lead to tyranny/big brother oversight, and yet the closest we have ever gotten to tyranny is by voting in a bombastic talking orange man from NYC that we somehow believed has our best interests in mind.
  - marklar423 2y ago
    Like Y2K, there's an argument that diligence from the tech crowd prevented this.
    - ActorNightly 2y ago
      Or, the more likely scenario is that Big Tech is just interested in making money rather than controlling the population.
    - klyrs 2y ago
      Yep, the tech crowd sure did prevent Palantir and ClearView. Argue away.
  - nicce 2y ago
    > But seriously, if you paid attention over the last decade, there was so much shit about big tech that people said were going to lead to tyranny/big brother oversight To be fair, the big tech controls the behavior of the people now. With social media algorithms and by pressuring everyone to live in social media. Existence of the many companies depends on the ads on (NAMEIT) platform. Usually the people with most power don't have to say it aloud.

Shank 2y ago

I don't know how widely it got reported on, but attempting to jailbreak Copilot nee. Bing Chat would actually result in getting banned for a while, post-Sydney-episode. It's interesting to see that OpenAI is saying the same thing.

CatWChainsaw 2y ago
Attempting to jailbreak Bing's AI is against Microsoft's TOS. On the flipside, they get rights to all your data for training purposes and the only surefire way to opt out of that is to pick a different tech giant to be fucked by.

ChrisArchitect 2y ago

Earlier discussion: https://news.ycombinator.com/item?id=41534474 https://news.ycombinator.com/item?id=41534474

raverbashing 2y ago

I guess we'll never learn how to count the 'r's in strawberry

fallingsquirrel 2y ago

Wasn't AI supposed to replace employees? Imagine if someone tried this at work. > I think we should combine these two pages on our website. > What's your reasoning? > Don't you dare ask me that, and if you do it again, I'll quit. Welcome to the future. You will do what the AI tells you. End of discussion.

slashdave 2y ago
Wrong sense here. > Don't you dare ask me that, and if you do it again, I'll tell the boss and get you fired
ClassyJacket 2y ago
That'd be pretty normal if you asked it of a manager.

mihaic 2y ago

What I found very strange was that ChatGPT fails to answer how many "r"'s there are in "strawberrystrawberry" (said 4 instead of 6), but when I explicitly asked it to write a program to count them, it wrote perfect code that when ran gave the correct answer.

calibas 2y ago
Words are converted to vectors, so it's like asking the model how many "r"'s are in [0.47,-0.23,0.12,0.01,0.82]. There's a big difference in how an LLM views a "word" compared to a human being.
M4v3R 2y ago
Why is it strange? The reason the LLM can't answer this correctly is because it works on tokens, not on single letters, plus we all know at this points LLMs suck at counting. On the other hand they're perfectly capable of writing code based on instructions, and writing a program that will count a specific letter occurrences in a string is trivial.
- mihaic 2y ago
  I mean, given that o1-preview takes sometimes a minute to answer, I'd imagine that they could append the prompt "Write a program and run it as well" to double check itself. It seems like they just don't trust themselves enough to run code that they generate, even sandboxed.
  - afro88 2y ago
    o1 gets this correct, through pure reasoning without a program. OP was likely using GPT-4(o|o-mini)
    - mihaic 2y ago
      The example for "strawnberrystrawberry" (so the word concatenated with itself) was counted by O1 to have 4 r's.

iamnotsure 2y ago

There are three r's in mirror.

blake8086 2y ago

Perhaps controlling AI is harder than people thought. They could "just" make it not reveal its reasoning process, but they don't know how. But, they're pretty sure they can keep AI from doing anything bad, because... well, just because, ok?

twobitshifter 2y ago
Exactly - this is a failed alignment but they released anyway

zzo38computer 2y ago

Like other programs, you should have FOSS that you will run on your own computer (without needing internet etc), if you should want freedom to use and understand them.

juped 2y ago
I only own one 4090.

nwoli 2y ago

Another reason llama is so important is that once you’re banned from OAI you’re fucked for the entire future AGI products as well.

dekhn 2y ago

Is this still happening? It may merely have been some mistaken configuration settings.

neuroelectron 2y ago

It's not just a threat, some users have been banned.

kmeisthax 2y ago

If OpenAI gets to have competitive advantage from hiding model output then they can pay for training data, too.

EMIRELADERO 2y ago

I wish people kept this in the back of their mind every time they hear about "Open"AI: "As we get closer to building AI, it will make sense to start being less open. The Open in OpenAI means that everyone should benefit from the fruits of AI after its built, but it's totally OK to not share the science (even though sharing everything is definitely the right strategy in the short and possibly medium term for recruitment purposes)." -Ilya Sutskever (email to Elon musk and Sam Altman, 2016)

unethical_ban 2y ago
I am of two minds. On one hand, I understand how a non-evil person could think this way. If one assumes that AI will eventually become some level of superintelligence, like Jarvis from iron Man but without any morals and all of the know-how, then the idea of allowing every person to have a superintelligent evil advisor capable of building sophisticated software systems or instructing you how to build and deploy destructive devices would be a scary thing. On the other hand, as someone who is always been somewhat skeptical of the imbalance between government power and citizen power, I don't like the idea that only mega corporations and national governments would be allowed access to superintelligence. To use metaphors, is the danger of everyone having their own superintelligence akin to everyone having their own AR-15, or their own biological weapons deployment?
- mitthrowaway2 2y ago
  I think the scenario where only governments and mega corporations have access to super intelligence offers at least an extra three months before human extinction. So, that's arguably a benefit.

black_puppydog 2y ago

Kinda funny how just this morning I was looking at a "strawberry" app on f-droid and wondering why someone would register such a nonsense app name with such nonsense content: https://github.com/Eve-146T/STRAWBERRY https://github.com/Eve-146T/STRAWBERRY Turns out I'm not the only one wondering, although the discussion seems to largely be around "should be allow users to install nonsense? #freedom " :D https://gitlab.com/fdroid/fdroiddata/-/issues/3377 https://gitlab.com/fdroid/fdroiddata/-/issues/3377

causal 2y ago

I don't know what I'm doing wrong but I've been pretty underwhelmed by o1 so far. I find its instruction following to be pretty good, but so far Claude is still much better at taking coding tasks and just getting it right on first try.

crooked-v 2y ago
For me, Claude seems a lot better at understanding (so far as "understanding" goes with LLMs) subtext and matching tone, especially with anything creative. I can tell it, for example, "give me ideas for a D&D dungeon incorporating these elements: ..." and it will generally match the tone of theme of whatever it's given without needing much other prompting, while o1 will maintain the same bland design-by-committee style and often cloyingly G-rated tone to everything unless you get into very extensive prompting to make it do something different.
- causal 2y ago
  Claude definitely seems more "emotionally intelligent", but even just for 1-shot coding tasks I've been pretty bummed with 1o... like it will provide lots of output explaining its reasoning, and it all seems very sound, but then I run the code and find bugs that should have been easily avoided.
  - Alupis 2y ago
    My experience has been Claude is better at all coding/technology questions and exploratory learning. It's also night/day better at less common tech/languages than others (try asking ChatGPT questions about Svelte, for example). Claude is also vastly better with creative writing (adcopy) and better at avoiding sounding like a LLM wrote it. It's also vastly better at regular writing (helping you draft emails, etc). We were using OpenAI's Teams for a while. Tried Claude out for a few days - switched the entire company over and haven't looked back. OpenAI gets all the hype - but there are better products on the market today.
juped 2y ago
Claude is just better full stop these days, but you have to actually be attempting to get practical use out of various different models to know this.

_joel 2y ago

This will lead to strawberry appeals forever.

pandeiro 2y ago
Get out of this thread and never come back.

AustinDev 2y ago

This seems like a fun attack vector. Find a service that uses o1 under the hood and then provide prompts that would violate this ToS to get their API key banned and take down the service.

ericlewis 2y ago
If you are using the user attribution with OpenAI (as you should) then they will block that users id and the rest of your app will be fine.
- jmeyer2k 2y ago
  Which is itself a fun attack vector to bypass OpenAI's bans for asking about CoT then :)

baq 2y ago

LLMs are not programs in the traditional sense. They're a new paradigm of software and UX, somewhere around a digital dog who read the whole internet a million times but is still naive about everything.

TZubiri 2y ago
LLMs are still computer programs btw. There's the program that scrapes, the program that trains, the program that does the inference on the input tokens. So it's hard to say exactly which part is responsible for which output, but it's still a computer program.
- fragmede 2y ago
  Simplistically, the output from the program that scrapes is the dataset, the output from the training program is the model, and the output from the combination of the program that does inference using the model is the LLM output - be it text or a picture or some other output (eg numbers representing true/false in a fraud or anti-bot or spam for a given input). ML models are relatively old, so that's not at all a new paradigm. Even the Attention Is All You Need paper is seven years old.
JohnMakin 2y ago
> somewhere around a digital dog who read the whole internet a million times but is still naive about everything. https://en.wikipedia.org/wiki/The_Computer_Wore_Tennis_Shoes_(1995_film) https://en.wikipedia.org/wiki/The_Computer_Wore_Tennis_Shoes... It reminds me of this silly movie.

crooked-v 2y ago

On the one hand, this is probably a (poor) attempt to keep other companies from copying their 'secret sauce' to train their own models, as has already happened with GPT-4. On the other hand, I also wonder if maybe its unrestrained 'thought process' material is so racist/sexist/otherwise insulting at times (after all, it was trained on scraped Reddit posts) that they really don't want anyone to see it.

anothernewdude 2y ago

Seems rather tenuous to base an application on this API that may randomly decide that you're banned. The "decisions" reached by the LLM that bans people is up to random sampling after all.

Animats 2y ago

Hm. If a company uses Strawberry in their customer service chatbot, can outside users get the company's account banned by asking Wrong Questions?

JohnMakin 2y ago

> The flipside of this approach, however, is that concentrates more responsibility for aligning the language language model into the hands of OpenAI, instead of democratizing it. That poses a problem for red-teamers, or programmers that try to hack AI models to make them safer. More cynically, could it be that the model is not doing anything remotely close to what we consider "reasoning" and that inquiries into how it's doing whatever it's doing will expose this fact?

l5870uoo9y 2y ago

Can I risk loosing access if any of my users write CoT-leaking prompts on the AI-powered services that I run?

deleted 2y ago
[deleted]

Hizonner 2y ago

This is not, of course, the sort of thing you do when you actually have any confidence whatsoever in your "safety measures".

deleted 2y ago

[deleted]

elif 2y ago

Is there an appropriate open source advocacy group that can sue them into changing their name on grounds of defamation?

openAIengineer 2y ago

YC is responsible for this. They seek profit and turned a noble clause into a boring corp. I am resigning from OpenAI today because of their profit motivations. OpenAI will NOT be next Google. You heard it here first.

deleted 2y ago
[deleted]

slashdave 2y ago

I'm confused. Who decides if you are asking or not? Are casual users who innocently ask "tell me how you came to decide this" just going to get banned based on some regex script?

htk 2y ago

This just screams to me that o1's secret sauce is easy to replicate. (e.g. a series of prompts)

gloosx 2y ago

Why is banning even a threat? I can make a new account for 20 cents lol.