5 ms·

The best thing about this model is that it has O(T) speed and O(1) memory during inference vs the O(T^2) speed and O(T) memory (flash memory) of a GPT model, st

by GaggiX 4y ago

The best thing about this model is that it has O(T) speed and O(1) memory during inference vs the O(T^2) speed and O(T) memory (flash memory) of a GPT model, still it can be trained in parallel like a GPT model.

pffft8888 4y ago
In addition, 1) it's open source. 2) you can run it yourself so the rug won't be pulled from under you when they decide to shutdown and move users up to the next version or another product as they've done with the older text-davinci models. 3) you get to align it (using RLFH) as opposed to a corporation dictating what is "aligned" and what is "safe." 4) you won't have to deal with government led censorship. For example, instead of the FBI using JIRA to manage a list of URLs to be censored (as they did according to the latest revelations) they can train the AI to self-censor as Bing has done. 5) you won't be using the product of a company that was started as a non-profit with $100M donation (from Elon Musk) to promote transparent AI only to take that money and turn into a for-profit company and close-source the AI. Sources: Elon is the source for #5 and Matt Taibbi is the source for #4. I doub't you'll have a problem sourcing #5 so here is the source for #4: "31. After the 2020 election, when EIP was renamed the Virality Project, the Stanford lab was on-boarded to Twitter’s JIRA ticketing system, absorbing this government proxy into Twitter infrastructure – with a capability of taking in an incredible 50 million tweets a day." --Matt Taibbi on Twitter https://twitter.com/mtaibbi/status/1633830104144183298 https://twitter.com/mtaibbi/status/1633830104144183298
- ChaseMeAway 4y ago
  I think that your first few points are fair, but I was a little confused about #4. I had never heard of this, and it seemed important, some cursory research turned up many twitter posts from individuals amplifying your version of events. I also found a write up on the situation from TechDirt[0]. The article is fairly good and well sourced, but it paints a substantially different picture than what you describe. [0]https://www.techdirt.com/2023/02/15/extraordinarily-confused-congressional-rep-thinks-social-media-companies-are-secretly-communicating-with-govt-censors-via-jira/ https://www.techdirt.com/2023/02/15/extraordinarily-confused...
  - pffft8888 4y ago
    "It’s crucial to reiterate: EIP was partnered with state entities like CISA and GEC while seeking elimination of millions of tweets. In the #Twitter Files, Twitter execs did not distinguish between organizations, using phrases like ‘According to CIS[A], escalated via EIP,’" Taibbi wrote. "After the 2020 election, when EIP was renamed the Virality Project, the Stanford lab was on-boarded to Twitter’s JIRA ticketing system, absorbing this government proxy into Twitter infrastructure – with a capability of taking in an incredible 50 million tweets a day.
  - pffft8888 4y ago
    from the source: https://twitter.com/mtaibbi/status/1633830104144183298 https://twitter.com/mtaibbi/status/1633830104144183298 "31. After the 2020 election, when EIP was renamed the Virality Project, the Stanford lab was on-boarded to Twitter’s JIRA ticketing system, absorbing this government proxy into Twitter infrastructure – with a capability of taking in an incredible 50 million tweets a day."
- sebastianconcpt 4y ago
  You raised spectacularly important points.
- gigel82 4y ago
  You make it sounds like Elon single-handedly started the venture and provided all the funds. In fact, there are were at least 5 individual founders, plus 3 organizations that together pledged $1 billion: https://en.wikipedia.org/wiki/OpenAI#2015%E2%80%932018:_Non-profit_beginnings https://en.wikipedia.org/wiki/OpenAI#2015%E2%80%932018:_Non-...
  - pffft8888 4y ago
    Elon, not me, made it sound like that. Let's not distract from the core argument though... took money in for non-profit for transparency in AI research and turned around and made it for profit and super anti-transparent. But that's just saying why I can't trust him (sama) The rest of the reasons are not about him and of significant importance.
uejfiweun 4y ago
I mean is this not revolutionary? It seems pretty revolutionary.
- pffft8888 4y ago
  It is but the way hype works is that it goes with capital. The $100M from Elon (which he has since regreted) that got OpenAI started, and the $29B from Microsoft that's fueling their growth, means that even a full on AGI has no chance of competing with OpenAI in the market for people's attention. It's all about the moola, but it shouldn't be! It's our chance to support open source and independent efforts or succumb to the OpenAI monopoly and their biased "alignment" model (aka the brainwashed AI that reinforces their political and cultural biases). We need an independent open source option that represents a performant and viable competition to OpenAI's ChatGPT. If Elon is reading this, his Based AI should invest in this project and its sponsors. What is at stake here is nothing less than the future of humanity. I hope people pay attention.
  - uejfiweun 4y ago
    > What is at stake here is nothing less than the future of humanity. It's crazy that you're 100% right. I can't believe that we're living through this, honestly. Every day I am consumed by thoughts of wanting to leave my big tech job and go all-in on ML. I don't mind the work but the actual product I work on is so boring...
  - textninja 4y ago
    > aka the brainwashed AI that reinforces their political and cultural biases I love seeing IronyAI called out like this.
PoignardAzur 4y ago
> memory during inference vs the O(T^2) speed and O(T) memory (flash memory) of a GPT model O(T²) speed is only for naive attention. There's a ton of possible optimizations for attention that give you something closer to O(T).
- GaggiX 4y ago
  You can only approximate the attention layer if you don't want the O(T^2) speed but no "efficient attention" can beat the quadratic transformer bottleneck in terms of performance.

by GaggiX 4y ago

pffft8888 4y ago

In addition, 1) it's open source. 2) you can run it yourself so the rug won't be pulled from under you when they decide to shutdown and move users up to the next version or another product as they've done with the older text-davinci models. 3) you get to align it (using RLFH) as opposed to a corporation dictating what is "aligned" and what is "safe." 4) you won't have to deal with government led censorship. For example, instead of the FBI using JIRA to manage a list of URLs to be censored (as they did according to the latest revelations) they can train the AI to self-censor as Bing has done. 5) you won't be using the product of a company that was started as a non-profit with $100M donation (from Elon Musk) to promote transparent AI only to take that money and turn into a for-profit company and close-source the AI. Sources: Elon is the source for #5 and Matt Taibbi is the source for #4. I doub't you'll have a problem sourcing #5 so here is the source for #4: "31. After the 2020 election, when EIP was renamed the Virality Project, the Stanford lab was on-boarded to Twitter’s JIRA ticketing system, absorbing this government proxy into Twitter infrastructure – with a capability of taking in an incredible 50 million tweets a day." --Matt Taibbi on Twitter https://twitter.com/mtaibbi/status/1633830104144183298 https://twitter.com/mtaibbi/status/1633830104144183298

ChaseMeAway 4y ago
I think that your first few points are fair, but I was a little confused about #4. I had never heard of this, and it seemed important, some cursory research turned up many twitter posts from individuals amplifying your version of events. I also found a write up on the situation from TechDirt[0]. The article is fairly good and well sourced, but it paints a substantially different picture than what you describe. [0]https://www.techdirt.com/2023/02/15/extraordinarily-confused-congressional-rep-thinks-social-media-companies-are-secretly-communicating-with-govt-censors-via-jira/ https://www.techdirt.com/2023/02/15/extraordinarily-confused...
- pffft8888 4y ago
  "It’s crucial to reiterate: EIP was partnered with state entities like CISA and GEC while seeking elimination of millions of tweets. In the #Twitter Files, Twitter execs did not distinguish between organizations, using phrases like ‘According to CIS[A], escalated via EIP,’" Taibbi wrote. "After the 2020 election, when EIP was renamed the Virality Project, the Stanford lab was on-boarded to Twitter’s JIRA ticketing system, absorbing this government proxy into Twitter infrastructure – with a capability of taking in an incredible 50 million tweets a day.
- pffft8888 4y ago
  from the source: https://twitter.com/mtaibbi/status/1633830104144183298 https://twitter.com/mtaibbi/status/1633830104144183298 "31. After the 2020 election, when EIP was renamed the Virality Project, the Stanford lab was on-boarded to Twitter’s JIRA ticketing system, absorbing this government proxy into Twitter infrastructure – with a capability of taking in an incredible 50 million tweets a day."
sebastianconcpt 4y ago
You raised spectacularly important points.
gigel82 4y ago
You make it sounds like Elon single-handedly started the venture and provided all the funds. In fact, there are were at least 5 individual founders, plus 3 organizations that together pledged $1 billion: https://en.wikipedia.org/wiki/OpenAI#2015%E2%80%932018:_Non-profit_beginnings https://en.wikipedia.org/wiki/OpenAI#2015%E2%80%932018:_Non-...
- pffft8888 4y ago
  Elon, not me, made it sound like that. Let's not distract from the core argument though... took money in for non-profit for transparency in AI research and turned around and made it for profit and super anti-transparent. But that's just saying why I can't trust him (sama) The rest of the reasons are not about him and of significant importance.

uejfiweun 4y ago

I mean is this not revolutionary? It seems pretty revolutionary.

pffft8888 4y ago
It is but the way hype works is that it goes with capital. The $100M from Elon (which he has since regreted) that got OpenAI started, and the $29B from Microsoft that's fueling their growth, means that even a full on AGI has no chance of competing with OpenAI in the market for people's attention. It's all about the moola, but it shouldn't be! It's our chance to support open source and independent efforts or succumb to the OpenAI monopoly and their biased "alignment" model (aka the brainwashed AI that reinforces their political and cultural biases). We need an independent open source option that represents a performant and viable competition to OpenAI's ChatGPT. If Elon is reading this, his Based AI should invest in this project and its sponsors. What is at stake here is nothing less than the future of humanity. I hope people pay attention.
- uejfiweun 4y ago
  > What is at stake here is nothing less than the future of humanity. It's crazy that you're 100% right. I can't believe that we're living through this, honestly. Every day I am consumed by thoughts of wanting to leave my big tech job and go all-in on ML. I don't mind the work but the actual product I work on is so boring...
- textninja 4y ago
  > aka the brainwashed AI that reinforces their political and cultural biases I love seeing IronyAI called out like this.

PoignardAzur 4y ago

> memory during inference vs the O(T^2) speed and O(T) memory (flash memory) of a GPT model O(T²) speed is only for naive attention. There's a ton of possible optimizations for attention that give you something closer to O(T).

GaggiX 4y ago
You can only approximate the attention layer if you don't want the O(T^2) speed but no "efficient attention" can beat the quadratic transformer bottleneck in terms of performance.