6 ms·

I just finished reading the 'paper' and I'm astonished that they aren't even publishing the # of parameters or even a vague outline of the architecture changes.

I just finished reading the 'paper' and I'm astonished that they aren't even publishing the # of parameters or even a vague outline of the architecture changes. It feels like such a slap in the face to all the academic AI researchers that their work is built off over the years, to just say 'yeah we're not telling you how any of this is possible because reasons'. Not even the damned parameter count. Christ.

soheil 4y ago
I wouldn't be surprised if this is due do some national security concerns and if the government has already been involved in every aspect of what OpenAI is doing.
- hackernewds 4y ago
  Highly unlikely
swatcoder 4y ago
In the old days of flashy tech conferences, that was precisely the sign of business-driven demo wizardry. The prerecorded videos, the staff-presented demos, the empty hardware chassis, the suggestive technical details, etc They have “reasons” for not giving away details, but there are good odds that the ultimate reason is that this is a superficial product update with a lot of flashy patchwork rather than that fundamental advance in AI technology we’d assume from the name.
- sebzim4500 4y ago
  You can use the product now though, they aren't pulling a Google.
- hnfong 4y ago
  No, the reason is they don’t want other companies to replicate their results so that they can maintain their first mover advantage. You can use the product today, right now.
  - dmix 4y ago
    Yeah it's a bit silly to act like this is all marketing fluff when the actual product is released to the public and we can all compare it to results of GPT3.5. A mining company protecting access to the gold mine is different than a company with a fools gold mine limiting access to the mine to delay analysis. There might be an "empty chassis" in the academic paper but that's different than tech companies betting on their closed sourced licensing/marketing to spin something less-than-whole.
  - VHRanger 4y ago
    People have, and it gaslit them into thinking it was 2022
- circuit10 4y ago
  They did a live demo though, that wasn’t pre-recorded
zpeti 4y ago
The sceptical me says its more of a marketing ploy, for people not subscribed to chatgpt pro yet, getting v4 is a good reason. I wouldn't be suprised if they get millions of new subscribers today.
- precompute 4y ago
  Agreed, seeing how Bing chat has now been confirmed that it was using GPT-4.
whazor 4y ago
I think it is important to know, as a user, how things roughly work. Now we don't know how they fixed previous flaws or what the drawbacks are.
DiogenesKynikos 4y ago
Ironic, given that their name is "OpenAI."
- JBiserkov 4y ago
  "Take the opposite of your biggest flaw and blast it on repeat 24/7 in your media campaigns." Here they've managed 2 - it's not open and it's not AI.
hackernewds 4y ago
Ironic their name is OpenAI to imply and borrow from the toils of previous academics
- pram 4y ago
  Neither Open nor AI
radu_floricica 4y ago
Because... it's past that? It's a huge commercial enterprise, by number of new subscribers possible the biggest in history. Complaining about paper details is a bit offtopic - it's nice they made a token effort to release one, but it hasn't been that kind of thing at least since November.
oezi 4y ago
Can anybody give an educated guess based on the published pricing or reading between the lines of the report? How big is this model and what did they do differently (ELI5 please)?
- espadrine 4y ago
  My educated guess is that they use a MoE-style model similar to the Switch transformer[0], and combine a similar encoding as that of Kosmos-1[1] (with an “image” latch token, and a ViT-style transformer to process images). As a result, the parameter count is likely bigger, but since not all of them are involved in a forward pass, it is not as meaningful. [0]: https://arxiv.org/pdf/2302.14045.pdf https://arxiv.org/pdf/2302.14045.pdf [1]: https://arxiv.org/pdf/2101.03961.pdf https://arxiv.org/pdf/2101.03961.pdf
- pixel_tracing 4y ago
  You could copy the abstract now and ask GPT4 to explain itself in the style of ELI5
clircle 4y ago
What would an academic do with the parameter count? It’s just a marketing number
- dx034 4y ago
  Maybe it wasn't parameter count increase that made any of this possible but they don't want to give that away. By keeping all developments vague, it's harder to determine if they found some novel technique they don't want others to know.
- mk_stjames 4y ago
  It's not just a marketing number, its a main indicator of model size and memory usage. Some of what is happening now is trying to see how 'large' the LLMs need to be to function at certain level, for instance it was claimed Llama (65B) had GPT-3 (175B) level performance but at 65B parameters that is a lot less memory usage. It's rough high level indicator of the computational requirements to run the model.
  - mirker 4y ago
    Without accounting for data and model architecture, it’s not a very useful number. For all we know, they may have sparse approximations which would throw this off by a lot. For example, if you measure a fully connected model over images of size N^2 and compare it to a convolutional one, the former would have O(N^4) parameters and the latter would have O(K^2) parameters, for K<N window size. It’s only useful if you know they essentially stacked additional layers on top of GPT3.5, which we know is not the case as they added a vision head.
- redox99 4y ago
  If this is like a multi trillion parameter model, then you know to replicate it it's probably cranking up the parameter count. If this is a <100M model, then you know there is some breakthrough they found that you need to find out, instead of wasting time and money with more parameters.
globular-toast 4y ago
Unfortunately screaming "that's not fair" doesn't tend to achieve anything. This is Microsoft. This is what they do.
deleted 4y ago
[deleted]
MichaelMoser123 4y ago
a new kind of science; they don't tell you anything, you just have to believe that it works - somehow.
- lyu07282 4y ago
  But you can see with your own eyes that it does in fact, work. They just stopped telling you how exactly it works. Is this science? No it doesn't advance humanities understanding of anything. Openai research is pure for-profit, proprietary r&d, not exactly a new thing.
  - MichaelMoser123 4y ago
    > But you can see with your own eyes that it does in fact, work. I can't tell if the answers of the tool are objective or if the answers aren't biases by the training corpus (or some collective bias ingrained within the training corpus). I really can't tell much about it. It's very much a black box - take it or leave it. I think that's a great deal. I mean science is all about objectivity (or so i was told)
whiplash451 4y ago
We’re talking about dozens of billions of dollars in valuation/revenue here. Time for a reality check.
evanrmurphy 4y ago
It is frustrating to other researchers and may be self-interested as other commenters mentioned. But these models are also now capable enough that if they are going to be developed, publishing architectural details could be a serious infohazard. It's good when AI labs don't publish some details about powerful models, for the same reason that it's good when bio research labs don't publish details about dangerous viruses.
- feanaro 4y ago
  Do you believe that these models will not be replicated outside OpenAI? And do you believe OpenAI will remain relatively benevolent long-term if they are not replicated elsewhere?
  - evanrmurphy 4y ago
    I believe they will be replicated outside OpenAI, given enough time. But the fewer details OpenAI releases, the longer it will take for someone else to replicate them. To your second question, I am worried about the power dynamics of one lab having a monopoly on super-powerful models. But by far the worst risk I'm worried about (and it's my job to try and help mitigate) is catastrophic accidents from someone creating a super-powerful model without the right alignment techniques and safeguards. And that kind of risk is heightened when there are more actors competitively racing to build AGI.

soheil 4y ago

I wouldn't be surprised if this is due do some national security concerns and if the government has already been involved in every aspect of what OpenAI is doing.

hackernewds 4y ago
Highly unlikely

swatcoder 4y ago

In the old days of flashy tech conferences, that was precisely the sign of business-driven demo wizardry. The prerecorded videos, the staff-presented demos, the empty hardware chassis, the suggestive technical details, etc They have “reasons” for not giving away details, but there are good odds that the ultimate reason is that this is a superficial product update with a lot of flashy patchwork rather than that fundamental advance in AI technology we’d assume from the name.

sebzim4500 4y ago
You can use the product now though, they aren't pulling a Google.
hnfong 4y ago
No, the reason is they don’t want other companies to replicate their results so that they can maintain their first mover advantage. You can use the product today, right now.
- dmix 4y ago
  Yeah it's a bit silly to act like this is all marketing fluff when the actual product is released to the public and we can all compare it to results of GPT3.5. A mining company protecting access to the gold mine is different than a company with a fools gold mine limiting access to the mine to delay analysis. There might be an "empty chassis" in the academic paper but that's different than tech companies betting on their closed sourced licensing/marketing to spin something less-than-whole.
- VHRanger 4y ago
  People have, and it gaslit them into thinking it was 2022
circuit10 4y ago
They did a live demo though, that wasn’t pre-recorded

zpeti 4y ago

The sceptical me says its more of a marketing ploy, for people not subscribed to chatgpt pro yet, getting v4 is a good reason. I wouldn't be suprised if they get millions of new subscribers today.

precompute 4y ago
Agreed, seeing how Bing chat has now been confirmed that it was using GPT-4.

whazor 4y ago

I think it is important to know, as a user, how things roughly work. Now we don't know how they fixed previous flaws or what the drawbacks are.

DiogenesKynikos 4y ago

Ironic, given that their name is "OpenAI."

JBiserkov 4y ago
"Take the opposite of your biggest flaw and blast it on repeat 24/7 in your media campaigns." Here they've managed 2 - it's not open and it's not AI.

hackernewds 4y ago

Ironic their name is OpenAI to imply and borrow from the toils of previous academics

pram 4y ago
Neither Open nor AI

radu_floricica 4y ago

Because... it's past that? It's a huge commercial enterprise, by number of new subscribers possible the biggest in history. Complaining about paper details is a bit offtopic - it's nice they made a token effort to release one, but it hasn't been that kind of thing at least since November.

oezi 4y ago

Can anybody give an educated guess based on the published pricing or reading between the lines of the report? How big is this model and what did they do differently (ELI5 please)?

espadrine 4y ago
My educated guess is that they use a MoE-style model similar to the Switch transformer[0], and combine a similar encoding as that of Kosmos-1[1] (with an “image” latch token, and a ViT-style transformer to process images). As a result, the parameter count is likely bigger, but since not all of them are involved in a forward pass, it is not as meaningful. [0]: https://arxiv.org/pdf/2302.14045.pdf https://arxiv.org/pdf/2302.14045.pdf [1]: https://arxiv.org/pdf/2101.03961.pdf https://arxiv.org/pdf/2101.03961.pdf
pixel_tracing 4y ago
You could copy the abstract now and ask GPT4 to explain itself in the style of ELI5

clircle 4y ago

What would an academic do with the parameter count? It’s just a marketing number

dx034 4y ago
Maybe it wasn't parameter count increase that made any of this possible but they don't want to give that away. By keeping all developments vague, it's harder to determine if they found some novel technique they don't want others to know.
mk_stjames 4y ago
It's not just a marketing number, its a main indicator of model size and memory usage. Some of what is happening now is trying to see how 'large' the LLMs need to be to function at certain level, for instance it was claimed Llama (65B) had GPT-3 (175B) level performance but at 65B parameters that is a lot less memory usage. It's rough high level indicator of the computational requirements to run the model.
- mirker 4y ago
  Without accounting for data and model architecture, it’s not a very useful number. For all we know, they may have sparse approximations which would throw this off by a lot. For example, if you measure a fully connected model over images of size N^2 and compare it to a convolutional one, the former would have O(N^4) parameters and the latter would have O(K^2) parameters, for K<N window size. It’s only useful if you know they essentially stacked additional layers on top of GPT3.5, which we know is not the case as they added a vision head.
redox99 4y ago
If this is like a multi trillion parameter model, then you know to replicate it it's probably cranking up the parameter count. If this is a <100M model, then you know there is some breakthrough they found that you need to find out, instead of wasting time and money with more parameters.

globular-toast 4y ago

Unfortunately screaming "that's not fair" doesn't tend to achieve anything. This is Microsoft. This is what they do.

deleted 4y ago

[deleted]

MichaelMoser123 4y ago

a new kind of science; they don't tell you anything, you just have to believe that it works - somehow.

lyu07282 4y ago
But you can see with your own eyes that it does in fact, work. They just stopped telling you how exactly it works. Is this science? No it doesn't advance humanities understanding of anything. Openai research is pure for-profit, proprietary r&d, not exactly a new thing.
- MichaelMoser123 4y ago
  > But you can see with your own eyes that it does in fact, work. I can't tell if the answers of the tool are objective or if the answers aren't biases by the training corpus (or some collective bias ingrained within the training corpus). I really can't tell much about it. It's very much a black box - take it or leave it. I think that's a great deal. I mean science is all about objectivity (or so i was told)

whiplash451 4y ago

We’re talking about dozens of billions of dollars in valuation/revenue here. Time for a reality check.

evanrmurphy 4y ago

It is frustrating to other researchers and may be self-interested as other commenters mentioned. But these models are also now capable enough that if they are going to be developed, publishing architectural details could be a serious infohazard. It's good when AI labs don't publish some details about powerful models, for the same reason that it's good when bio research labs don't publish details about dangerous viruses.

feanaro 4y ago
Do you believe that these models will not be replicated outside OpenAI? And do you believe OpenAI will remain relatively benevolent long-term if they are not replicated elsewhere?
- evanrmurphy 4y ago
  I believe they will be replicated outside OpenAI, given enough time. But the fewer details OpenAI releases, the longer it will take for someone else to replicate them. To your second question, I am worried about the power dynamics of one lab having a monopoly on super-powerful models. But by far the worst risk I'm worried about (and it's my job to try and help mitigate) is catastrophic accidents from someone creating a super-powerful model without the right alignment techniques and safeguards. And that kind of risk is heightened when there are more actors competitively racing to build AGI.