3 ms·

I tried all the recent wave of text/image to 3D model services, some touting 100 MM+ valuations and tens of millions raised and found them all to produce unusab

I tried all the recent wave of text/image to 3D model services, some touting 100 MM+ valuations and tens of millions raised and found them all to produce unusable garbage.

jampekka 2y ago
The gap from demos/papers to reality is huge. ML has a bad replication crisis.
- SV_BubbleTime 2y ago
  >The gap from demos/papers to reality is huge. SAI showed Stable Diffusion 3 pictures of women laying on grass. If you haven’t been following SD3… https://arstechnica.com/information-technology/2024/06/ridiculed-stable-diffusion-3-release-excels-at-ai-generated-body-horror/ https://arstechnica.com/information-technology/2024/06/ridic...
- freeone3000 2y ago
  This is not a “replication crisis”. Running the paper gets you the same results as the author; it’s uniquely replicable. The results not being useful in a product is not the same as a fundamental failure in our use of the scientific process.
  - jampekka 2y ago
    That is reproducibility. Replicability means that the results hold for replication outside the specific circumstances of one study.
    - fngjdflmdflg 2y ago
      >Replicability means that the results hold for replication outside the specific circumstances of one study. If by "hold for replication outside the specific circumstances of one study" you mean "useful for real world problems" as implied by your previous comment then I don't think you are correct. From a quick search it seems there are multiple definitions of Reproducibility and Replicability with some using the words interchangeably but the most favorable one I found to what you are saying is this definition: >Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. >[...] >In general, whenever new data are obtained that constitute the results of a study aimed at answering the same scientific question as another study, the degree of consistency of the results from the two studies constitutes their degree of replication.[0] However I think this holds true for a lot of ML research going on. The issue is not that the solutions do not generalize. It's that the solution itself is not useful for most real world applications. I don't see what replicability has to do with it. you can train a given model with a different but similar dataset and you will get the same quality non-useful results. I'm not sure exactly what definition of replicability you are using though if there is one I missed please point it out. [0] https://www.ncbi.nlm.nih.gov/books/NBK547546/ https://www.ncbi.nlm.nih.gov/books/NBK547546/
arrobbiato 2y ago
I have too, and you’re quite right. Also the various 2D-to-3D face generators are mostly awful. I’ve done a deep dive on that and nearly all of them seem to only create slight perturbations on some base model, regardless of the input.
dgellow 2y ago
Haven’t tried all, but yeah, pretty bad so far
dudus 2y ago
SOTA text-to-image 5 years ago was complete garbage. Most people would think the same. Look how good it got now. You have to look at this as stepping stone research.
- raincole 2y ago
  Did they got such high valuation 5 years ago? Genuine question.
  - dinglestepup 2y ago
    No. With one partial exception being OpenAI that got $1B investment ~5 years ago from MS before they launched DALL-E v1 (and even before GPT-3).
  - gpm 2y ago
    I'm not sure I'd expect valuations to be at all similar. The potential target market is significantly different in scale (I assume, I haven't tried to estimate either). The potential competitors are... already in existence. It seems more likely now that we'll succeed at good 3d-generative-AI then it seemed before we got good 2d-generative-AI that we would succeed at that...
ddtaylor 2y ago
We tried them too. My wife is a 3D artist, but we needed a lot of assets that frankly weren't that important. The plan was to use the output as a starting point and improve as needed manually. The problem is that the output you get is just baked meshes. If the object connects together or has a few pieces you'll have to essentially undo some of that work. Similar problems with textures as the AI doesn't work normally like other artists do. All of this is also on top of the output being basically garbage. Input photos ultimately fail in ways that would require so much work to fix it invalidates the concept. By the time you start to get something approaching decent output you've put in more work or money than just having someone make it to begin with while essentially also losing all control over the art pipeline.
guyomes 2y ago
The field evolves quickly. The Meta 3D Gen paper states that Rodin Gen-1 [1,2,3] has a clean topology. As a non professional, the wireframes from some examples indeed look nice. [1]: https://github.com/CLAY-3D/OpenCLAY https://github.com/CLAY-3D/OpenCLAY [2]: https://hyperhuman.deemos.com/ https://hyperhuman.deemos.com/ [3]: https://huggingface.co/spaces/DEEMOSTECH/Rodin https://huggingface.co/spaces/DEEMOSTECH/Rodin

I tried all the recent wave of text/image to 3D model services, some touting 100 MM+ valuations and tens of millions raised and found them all to produce unusable garbage.

jampekka 2y ago

The gap from demos/papers to reality is huge. ML has a bad replication crisis.

SV_BubbleTime 2y ago
>The gap from demos/papers to reality is huge. SAI showed Stable Diffusion 3 pictures of women laying on grass. If you haven’t been following SD3… https://arstechnica.com/information-technology/2024/06/ridiculed-stable-diffusion-3-release-excels-at-ai-generated-body-horror/ https://arstechnica.com/information-technology/2024/06/ridic...
freeone3000 2y ago
This is not a “replication crisis”. Running the paper gets you the same results as the author; it’s uniquely replicable. The results not being useful in a product is not the same as a fundamental failure in our use of the scientific process.
- jampekka 2y ago
  That is reproducibility. Replicability means that the results hold for replication outside the specific circumstances of one study.
  - fngjdflmdflg 2y ago
    >Replicability means that the results hold for replication outside the specific circumstances of one study. If by "hold for replication outside the specific circumstances of one study" you mean "useful for real world problems" as implied by your previous comment then I don't think you are correct. From a quick search it seems there are multiple definitions of Reproducibility and Replicability with some using the words interchangeably but the most favorable one I found to what you are saying is this definition: >Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. >[...] >In general, whenever new data are obtained that constitute the results of a study aimed at answering the same scientific question as another study, the degree of consistency of the results from the two studies constitutes their degree of replication.[0] However I think this holds true for a lot of ML research going on. The issue is not that the solutions do not generalize. It's that the solution itself is not useful for most real world applications. I don't see what replicability has to do with it. you can train a given model with a different but similar dataset and you will get the same quality non-useful results. I'm not sure exactly what definition of replicability you are using though if there is one I missed please point it out. [0] https://www.ncbi.nlm.nih.gov/books/NBK547546/ https://www.ncbi.nlm.nih.gov/books/NBK547546/

arrobbiato 2y ago

I have too, and you’re quite right. Also the various 2D-to-3D face generators are mostly awful. I’ve done a deep dive on that and nearly all of them seem to only create slight perturbations on some base model, regardless of the input.

dgellow 2y ago

Haven’t tried all, but yeah, pretty bad so far

dudus 2y ago

SOTA text-to-image 5 years ago was complete garbage. Most people would think the same. Look how good it got now. You have to look at this as stepping stone research.

raincole 2y ago
Did they got such high valuation 5 years ago? Genuine question.
- dinglestepup 2y ago
  No. With one partial exception being OpenAI that got $1B investment ~5 years ago from MS before they launched DALL-E v1 (and even before GPT-3).
- gpm 2y ago
  I'm not sure I'd expect valuations to be at all similar. The potential target market is significantly different in scale (I assume, I haven't tried to estimate either). The potential competitors are... already in existence. It seems more likely now that we'll succeed at good 3d-generative-AI then it seemed before we got good 2d-generative-AI that we would succeed at that...

ddtaylor 2y ago

We tried them too. My wife is a 3D artist, but we needed a lot of assets that frankly weren't that important. The plan was to use the output as a starting point and improve as needed manually. The problem is that the output you get is just baked meshes. If the object connects together or has a few pieces you'll have to essentially undo some of that work. Similar problems with textures as the AI doesn't work normally like other artists do. All of this is also on top of the output being basically garbage. Input photos ultimately fail in ways that would require so much work to fix it invalidates the concept. By the time you start to get something approaching decent output you've put in more work or money than just having someone make it to begin with while essentially also losing all control over the art pipeline.

guyomes 2y ago

The field evolves quickly. The Meta 3D Gen paper states that Rodin Gen-1 [1,2,3] has a clean topology. As a non professional, the wireframes from some examples indeed look nice. [1]: https://github.com/CLAY-3D/OpenCLAY https://github.com/CLAY-3D/OpenCLAY [2]: https://hyperhuman.deemos.com/ https://hyperhuman.deemos.com/ [3]: https://huggingface.co/spaces/DEEMOSTECH/Rodin https://huggingface.co/spaces/DEEMOSTECH/Rodin