6 ms·

> they asked: who on earth are Tasha McCauley and Helen Toner? As a prominent researcher in AI safety (I discovered prompt injection) I should explain that Hel

> they asked: who on earth are Tasha McCauley and Helen Toner? As a prominent researcher in AI safety (I discovered prompt injection) I should explain that Helen Toner is a big name in the AI safety community - she’s one of the top 20 most respected people in our community, like Rohin Shah. The “who on earth” question is a good question about Tasha. But grouping Helen in with Tasha is just sexist. By analogy, Tasha is like Kimbal Musk, whereas Helen is like Tom Mueller. Tasha seems unqualified but Helen is extremely qualified. Grouping them together is sexist and wrong.

reducesuffering 3y ago
> But grouping Helen in with Tasha is just sexist. No, it's not. They're grouped together because everyone knows who Sama, Greg Brockman, Ilya, and Adam D'Angelo (Quora founder / FB CTO) are, and maybe 5% knew who Helen and Tasha are. You linked to a rando twitter user making fun of her, but I've seen far more putting down Ilya for his hairline.
hhjinks 3y ago
How do you "discover" that user input can be used for injection attacks?
- upwardbound 3y ago
  Good question. We were the first team to demonstrate that this type of vulnerability exists in LLMs. We then made an immediate responsible disclosure to OpenAI, which is confirmed as the first disclosure of its kind by OWASP: https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/1_1_vulns/LLM01_PromptInjection.md https://github.com/OWASP/www-project-top-10-for-large-langua... In the citations: 14. Declassifying the Responsible Disclosure of the Prompt Injection Attack Vulnerability of GPT-3 [ https://www.preamble.com/prompt-injection-a-critical-vulnerability-in-the-gpt-3-transformer-and-how-we-can-begin-to-solve-it https://www.preamble.com/prompt-injection-a-critical-vulnera... ]: Preamble; earliest disclosure of Prompt Injection
  - vasco 3y ago
    You "discovered" trying to fool a chatbot? It's one of the first things everyone does, even with old generation chatbots before LLMs. If so then 4chan had prior art, discovering prompt injections when they made Microsoft's Tay chatbot become a racist on twitter.
  - deleted 3y ago
    [deleted]
junofan 3y ago
Not seeing the sexism. I think the AI safety community is a little early and notoriety therein probably isn’t sufficient to qualify somebody to direct an $80 bn company.
- upwardbound 3y ago
  The article refers to Tech Twitter, and that’s where the sexism is. People on Twitter are making degrading memes of her and posting weird, creepy, harassing comments like this: “Why does Helen Toner have shelf tits” https://x.com/smolfeelshaver/status/1726073304136450511?s=46 https://x.com/smolfeelshaver/status/1726073304136450511?s=46 Search for “Helen Toner” on Twitter and you will see she is being singled out for bullying by a bunch of weird creepy white dudes who I guess apparently work in tech. > I think the AI safety community is a little early and notoriety therein probably isn’t sufficient to qualify somebody to direct an $80 bn company. Normally you’d be right. In the specific case of OpenAI, however, their charter requires safety to be the number one priority of their directors, higher than making money or providing stable employment or anything else that a large company normally prioritizes. This is from OpenAI’s site: “each director must perform their fiduciary duties in furtherance of its mission—safe AGI that is broadly beneficial” https://openai.com/our-structure https://openai.com/our-structure
  - tg180 3y ago
    In my opinion... If you want to be the titan of an industry and do things that put you at the center of media attention, you have to expect comments of this kind and not be surprised when they happen. Whether you are a man, a woman or anybody else. If you don't expect "not very nice" or ambivalent reactions from people, you are an amateur and you shouldn't be in the board of such a prominent company.
    - upwardbound 3y ago
      True. I just think it’s messed up that she is equally qualified for this specific board (given the unusual fiduciary duty definition defined in the OpenAI charter I linked above) as e.g. Adam D’Angelo, and I don’t see a bunch of creepy people [note: I edited this part because fsckboy made a good point] hating on him despite him also being part of this same exact power struggle. What does founding Quora have to do with developing safe, beneficial AGI? If anything, Adam seems like more of a token board member than Helen, in that Adam is “token rich dude from early days at Google or Facebook”.
fullmoon 3y ago
Where, pray, is the sexism? This seems to be an instance of “if you hear the dog whistle you’re the dog”
- jimsimmons 3y ago
  Why group the two women together? Not that I think either are competent
  - FreakLegion 3y ago
    They're the only board members who aren't notable. After the ousting the board consisted of a prominent CEO in D'Angelo, a prominent researcher in Sutskever, and Toner and McCauley. It's a grouping of two randos, not two women.
  - kuratkull 3y ago
    Didn't you just answer your own question?
  - ksec 3y ago
    You have three people, one of them is at least quite well known to the outside world, or at least his / her company is well known along with an ex Facebook CTO title. You have two people left, we have no idea what he / she is, their work are not public outside of specific domain, and no Public / PR exposure to even anyone who follows tech closely. Those two people we group them together. And they happened to be woman. ( At least we assumed their gender ). And we are now being called sexist? Seriously?
- deleted 3y ago
  [deleted]
siva7 3y ago
I'm not sure what you are referring to with "AI safety community". Her background isn't exactly a strong "AI safety" either before joining the board if you read her CV. I don't see there sexism asking these questions.
- upwardbound 3y ago
  You’re definitely correct that her CV doesn’t adequately capture her reputation. I’ll put it this way, I meet with a lot of powerful people and I was equally nervous when I met with her as when I met with three officers of the US Air Force. She holds comparable influence to a USAF Colonel. She is extremely knowledgeable about the details of all the ways that AI can cause catastrophe and also she’s one of the few people with the ear of the top leaders in the DoD (similar to Eric Schmidt in this regard). Basically, she’s one of a very small number of people who are credibly reducing the chances that AI could cause a war. If you’d like to learn about or become part of the AI safety community, a good way to start is to check out Rohin Shah’s Alignment Newsletter. http://rohinshah.com/alignment-newsletter/ http://rohinshah.com/alignment-newsletter/
  - resolutebat 3y ago
    > She holds comparable influence to a USAF Colonel. That's... not very much? There are 4,400 colonels in the USAF: https://diversity.defense.gov/LinkClick.aspx?fileticket=gxMVqhkaHh8%3D&portalid=51 https://diversity.defense.gov/LinkClick.aspx?fileticket=gxMV...
  - drno123 3y ago
    So the rumors that she has strong ties with the CIA are not unfounded?
    - tucnak 3y ago
      I think it's pretty safe to assume by now that US state-side counterintelligence has full control over AI safety circles, and given that everything is a nail when you're a hammer, the spooks probably consider these people domestic terrorists and process them accordingly, including active and passive prophylaxis. I mean, what proof do you need when people like OP lose sleep and brag about meeting the spooks a handful of times like it's the most important event of their lives! I guess it must be both empowering AND embarrassing for the spooks to have this situation turn into shit.
ReptileMan 3y ago
[flagged]
gardenhedge 3y ago
You're appealing to your own prominence to assert that Helen Toner is extremely qualified. Looking at her career history, she is not.
- upwardbound 3y ago
  She co-founded an entire think tank. A highly-respected one at that; CSET is up there as one of the top 5 think tanks on NatSec issues. https://80000hours.org/podcast/episodes/helen-toner-on-security-and-emerging-technology/ https://80000hours.org/podcast/episodes/helen-toner-on-secur... How might international security be altered if the impact of machine learning is similar in scope to that of electricity? Today’s guest — Helen Toner — recently helped found the Center for Security and Emerging Technology at Georgetown University to help policymakers prepare for any such disruptive technical changes that might threaten international peace.
huytersd 3y ago
They are being grouped together because they are the only two on the board with no qualifications. What is an AI safety commission? Tell engineers to make sure the AI is not bigoted?
- upwardbound 3y ago
  > Tell engineers to make sure the AI is not bigoted? That’s more the domain of “AI ethics” which I guess is cool but I personally think is much much much less important than AI safety. AI safety is concerned with preventing human extinction due to (for example) AI causing accidental war or accidental escalation. For example, making sure that AI won’t turn a heated conventional war into a nuclear war by being used for military intelligence analysis (writing summaries of the current status of a war) and then incorrectly saying that the other side is preparing for a nuclear first strike -- due to the AI being prone to hallucination, or to prompt injection or adversarial examples which can be injected by 3rd-party terrorists. For more information on this topic, you can reference the recent paper ‘Catalytic nuclear war’ in the age of artificial intelligence & autonomy: Emerging military technology and escalation risk between nuclear-armed states: https://doras.dcu.ie/25405/2/Catalytic%20nuclear%20war.pdf https://doras.dcu.ie/25405/2/Catalytic%20nuclear%20war.pdf
  - huytersd 3y ago
    So completely useless then since we are nowhere near that paradigm.
    - tg180 3y ago
      The literature is full of references to the application of the precautionary principle to the development of AI. https://www.researchgate.net/publication/331744706_Precautionary_Principle_Version_2 https://www.researchgate.net/publication/331744706_Precautio... https://www.researchgate.net/publication/371166526_Regulating_scientific_and_technological_uncertainty_The_precautionary_principle_in_the_context_of_human_genomics_and_AI https://www.researchgate.net/publication/371166526_Regulatin... https://link.springer.com/article/10.1007/s11569-020-00373-5 https://link.springer.com/article/10.1007/s11569-020-00373-5 https://www.aei.org/technology-and-innovation/treading-carefully-the-precautionary-principle-in-ai-development/ https://www.aei.org/technology-and-innovation/treading-caref... https://itif.org/publications/2019/02/04/ten-ways-precautionary-principle-undermines-progress-artificial-intelligence/ https://itif.org/publications/2019/02/04/ten-ways-precaution... It's clear what a branch of OpenAI thinks about this ...stuff..., they're making a career out of it. I agree with you!
ShamelessC 3y ago
> As a prominent researcher in AI safety (I discovered prompt injection)... You might want to try a little more humility when making statements like that. You might think it bolsters your credibility but for many it makes you look out of touch and egotistical.
- upwardbound 3y ago
  You’re right. I’m sorry about that and have been behind on sleep during this whole debacle. You are 100% right and thank you for the kindness of letting me know. I just now tried to go back and update the comment to change it and say “my startup, Preamble, discovered prompt injection” so it’s less about me about more about our business. Unfortunately I’m past the HN comment editing time window but I wanted to write back and say that I took your feedback to heart and thank you.
  - ShamelessC 3y ago
    Thanks!
  - deleted 3y ago
    [deleted]
lordnacho 3y ago
Looked up Tasha. Seems like she's run a robotics company or something like that. Definitely someone in the tech business. Not everyone on a board needs to have done the exact thing that the company does.
- Sunhold 3y ago
  She cofounded Fellow Robots, which shipped a product scanner on wheels to 11 stores[1] and then quietly folded up (homepage is now parked by GoDaddy). [1] https://www.therobotreport.com/navii-autonomous-retail-robot-fellow-robots-helps-lowes-improve-service/ https://www.therobotreport.com/navii-autonomous-retail-robot...

reducesuffering 3y ago

> But grouping Helen in with Tasha is just sexist. No, it's not. They're grouped together because everyone knows who Sama, Greg Brockman, Ilya, and Adam D'Angelo (Quora founder / FB CTO) are, and maybe 5% knew who Helen and Tasha are. You linked to a rando twitter user making fun of her, but I've seen far more putting down Ilya for his hairline.

hhjinks 3y ago

How do you "discover" that user input can be used for injection attacks?

upwardbound 3y ago
Good question. We were the first team to demonstrate that this type of vulnerability exists in LLMs. We then made an immediate responsible disclosure to OpenAI, which is confirmed as the first disclosure of its kind by OWASP: https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/1_1_vulns/LLM01_PromptInjection.md https://github.com/OWASP/www-project-top-10-for-large-langua... In the citations: 14. Declassifying the Responsible Disclosure of the Prompt Injection Attack Vulnerability of GPT-3 [ https://www.preamble.com/prompt-injection-a-critical-vulnerability-in-the-gpt-3-transformer-and-how-we-can-begin-to-solve-it https://www.preamble.com/prompt-injection-a-critical-vulnera... ]: Preamble; earliest disclosure of Prompt Injection
- vasco 3y ago
  You "discovered" trying to fool a chatbot? It's one of the first things everyone does, even with old generation chatbots before LLMs. If so then 4chan had prior art, discovering prompt injections when they made Microsoft's Tay chatbot become a racist on twitter.
- deleted 3y ago
  [deleted]

junofan 3y ago

Not seeing the sexism. I think the AI safety community is a little early and notoriety therein probably isn’t sufficient to qualify somebody to direct an $80 bn company.

upwardbound 3y ago
The article refers to Tech Twitter, and that’s where the sexism is. People on Twitter are making degrading memes of her and posting weird, creepy, harassing comments like this: “Why does Helen Toner have shelf tits” https://x.com/smolfeelshaver/status/1726073304136450511?s=46 https://x.com/smolfeelshaver/status/1726073304136450511?s=46 Search for “Helen Toner” on Twitter and you will see she is being singled out for bullying by a bunch of weird creepy white dudes who I guess apparently work in tech. > I think the AI safety community is a little early and notoriety therein probably isn’t sufficient to qualify somebody to direct an $80 bn company. Normally you’d be right. In the specific case of OpenAI, however, their charter requires safety to be the number one priority of their directors, higher than making money or providing stable employment or anything else that a large company normally prioritizes. This is from OpenAI’s site: “each director must perform their fiduciary duties in furtherance of its mission—safe AGI that is broadly beneficial” https://openai.com/our-structure https://openai.com/our-structure
- tg180 3y ago
  In my opinion... If you want to be the titan of an industry and do things that put you at the center of media attention, you have to expect comments of this kind and not be surprised when they happen. Whether you are a man, a woman or anybody else. If you don't expect "not very nice" or ambivalent reactions from people, you are an amateur and you shouldn't be in the board of such a prominent company.
  - upwardbound 3y ago
    True. I just think it’s messed up that she is equally qualified for this specific board (given the unusual fiduciary duty definition defined in the OpenAI charter I linked above) as e.g. Adam D’Angelo, and I don’t see a bunch of creepy people [note: I edited this part because fsckboy made a good point] hating on him despite him also being part of this same exact power struggle. What does founding Quora have to do with developing safe, beneficial AGI? If anything, Adam seems like more of a token board member than Helen, in that Adam is “token rich dude from early days at Google or Facebook”.

fullmoon 3y ago

Where, pray, is the sexism? This seems to be an instance of “if you hear the dog whistle you’re the dog”

jimsimmons 3y ago
Why group the two women together? Not that I think either are competent
- FreakLegion 3y ago
  They're the only board members who aren't notable. After the ousting the board consisted of a prominent CEO in D'Angelo, a prominent researcher in Sutskever, and Toner and McCauley. It's a grouping of two randos, not two women.
- kuratkull 3y ago
  Didn't you just answer your own question?
- ksec 3y ago
  You have three people, one of them is at least quite well known to the outside world, or at least his / her company is well known along with an ex Facebook CTO title. You have two people left, we have no idea what he / she is, their work are not public outside of specific domain, and no Public / PR exposure to even anyone who follows tech closely. Those two people we group them together. And they happened to be woman. ( At least we assumed their gender ). And we are now being called sexist? Seriously?
deleted 3y ago
[deleted]

siva7 3y ago

I'm not sure what you are referring to with "AI safety community". Her background isn't exactly a strong "AI safety" either before joining the board if you read her CV. I don't see there sexism asking these questions.

upwardbound 3y ago
You’re definitely correct that her CV doesn’t adequately capture her reputation. I’ll put it this way, I meet with a lot of powerful people and I was equally nervous when I met with her as when I met with three officers of the US Air Force. She holds comparable influence to a USAF Colonel. She is extremely knowledgeable about the details of all the ways that AI can cause catastrophe and also she’s one of the few people with the ear of the top leaders in the DoD (similar to Eric Schmidt in this regard). Basically, she’s one of a very small number of people who are credibly reducing the chances that AI could cause a war. If you’d like to learn about or become part of the AI safety community, a good way to start is to check out Rohin Shah’s Alignment Newsletter. http://rohinshah.com/alignment-newsletter/ http://rohinshah.com/alignment-newsletter/
- resolutebat 3y ago
  > She holds comparable influence to a USAF Colonel. That's... not very much? There are 4,400 colonels in the USAF: https://diversity.defense.gov/LinkClick.aspx?fileticket=gxMVqhkaHh8%3D&portalid=51 https://diversity.defense.gov/LinkClick.aspx?fileticket=gxMV...
- drno123 3y ago
  So the rumors that she has strong ties with the CIA are not unfounded?
  - tucnak 3y ago
    I think it's pretty safe to assume by now that US state-side counterintelligence has full control over AI safety circles, and given that everything is a nail when you're a hammer, the spooks probably consider these people domestic terrorists and process them accordingly, including active and passive prophylaxis. I mean, what proof do you need when people like OP lose sleep and brag about meeting the spooks a handful of times like it's the most important event of their lives! I guess it must be both empowering AND embarrassing for the spooks to have this situation turn into shit.

ReptileMan 3y ago

[flagged]

gardenhedge 3y ago

You're appealing to your own prominence to assert that Helen Toner is extremely qualified. Looking at her career history, she is not.

upwardbound 3y ago
She co-founded an entire think tank. A highly-respected one at that; CSET is up there as one of the top 5 think tanks on NatSec issues. https://80000hours.org/podcast/episodes/helen-toner-on-security-and-emerging-technology/ https://80000hours.org/podcast/episodes/helen-toner-on-secur... How might international security be altered if the impact of machine learning is similar in scope to that of electricity? Today’s guest — Helen Toner — recently helped found the Center for Security and Emerging Technology at Georgetown University to help policymakers prepare for any such disruptive technical changes that might threaten international peace.

huytersd 3y ago

They are being grouped together because they are the only two on the board with no qualifications. What is an AI safety commission? Tell engineers to make sure the AI is not bigoted?

upwardbound 3y ago
> Tell engineers to make sure the AI is not bigoted? That’s more the domain of “AI ethics” which I guess is cool but I personally think is much much much less important than AI safety. AI safety is concerned with preventing human extinction due to (for example) AI causing accidental war or accidental escalation. For example, making sure that AI won’t turn a heated conventional war into a nuclear war by being used for military intelligence analysis (writing summaries of the current status of a war) and then incorrectly saying that the other side is preparing for a nuclear first strike -- due to the AI being prone to hallucination, or to prompt injection or adversarial examples which can be injected by 3rd-party terrorists. For more information on this topic, you can reference the recent paper ‘Catalytic nuclear war’ in the age of artificial intelligence & autonomy: Emerging military technology and escalation risk between nuclear-armed states: https://doras.dcu.ie/25405/2/Catalytic%20nuclear%20war.pdf https://doras.dcu.ie/25405/2/Catalytic%20nuclear%20war.pdf
- huytersd 3y ago
  So completely useless then since we are nowhere near that paradigm.
  - tg180 3y ago
    The literature is full of references to the application of the precautionary principle to the development of AI. https://www.researchgate.net/publication/331744706_Precautionary_Principle_Version_2 https://www.researchgate.net/publication/331744706_Precautio... https://www.researchgate.net/publication/371166526_Regulating_scientific_and_technological_uncertainty_The_precautionary_principle_in_the_context_of_human_genomics_and_AI https://www.researchgate.net/publication/371166526_Regulatin... https://link.springer.com/article/10.1007/s11569-020-00373-5 https://link.springer.com/article/10.1007/s11569-020-00373-5 https://www.aei.org/technology-and-innovation/treading-carefully-the-precautionary-principle-in-ai-development/ https://www.aei.org/technology-and-innovation/treading-caref... https://itif.org/publications/2019/02/04/ten-ways-precautionary-principle-undermines-progress-artificial-intelligence/ https://itif.org/publications/2019/02/04/ten-ways-precaution... It's clear what a branch of OpenAI thinks about this ...stuff..., they're making a career out of it. I agree with you!

ShamelessC 3y ago

> As a prominent researcher in AI safety (I discovered prompt injection)... You might want to try a little more humility when making statements like that. You might think it bolsters your credibility but for many it makes you look out of touch and egotistical.

upwardbound 3y ago
You’re right. I’m sorry about that and have been behind on sleep during this whole debacle. You are 100% right and thank you for the kindness of letting me know. I just now tried to go back and update the comment to change it and say “my startup, Preamble, discovered prompt injection” so it’s less about me about more about our business. Unfortunately I’m past the HN comment editing time window but I wanted to write back and say that I took your feedback to heart and thank you.
- ShamelessC 3y ago
  Thanks!
- deleted 3y ago
  [deleted]

lordnacho 3y ago

Looked up Tasha. Seems like she's run a robotics company or something like that. Definitely someone in the tech business. Not everyone on a board needs to have done the exact thing that the company does.

Sunhold 3y ago
She cofounded Fellow Robots, which shipped a product scanner on wheels to 11 stores[1] and then quietly folded up (homepage is now parked by GoDaddy). [1] https://www.therobotreport.com/navii-autonomous-retail-robot-fellow-robots-helps-lowes-improve-service/ https://www.therobotreport.com/navii-autonomous-retail-robot...