7 ms·
Meta 3D Gen
- explaininjs 2y agoLooks fine, but you can tell the topology isn’t good based on the lack of wireframes.
- nuz 2y agoSuch a silly argument. Fixing topology is a nearly solved problem in geometry processing. (Or just start with a good topology and 'paste' a texture onto it like they develop techniques for here.)
- explaininjs 2y agoNo… it’s not. But if you know something I don’t the 5 primes will certainly be happy to pay you handsomely for the implementation!
- nuz 2y agohttps://github.com/wjakob/instant-meshes https://github.com/wjakob/instant-meshes
- deleted 2y ago[deleted]
- explaininjs 2y agoA piece of software that hasn’t been touched in 5 years, let alone adopted in any professional production environment? Cool…
- portaouflop 2y agoAFAICT it’s used in professional applications and software does not need to be constantly updated, especially if it’s not for the web.
- explaininjs 2y agoIf the claim was that the problem was solved, sure it might make sense that the package does not need to be touched (in reality the field isn’t as slow as you presume, but I digress). Instead, the claim is that it’s “nearly^{TM}” solved, so the proof being an abandoned repo from half a decade ago actually speaks volumes: it’s solved except for the hard part, and nobody knows how to solve the hard part.
- portaouflop 2y agoWell yea I wasn’t claiming the problem is solved, that was GP. I don’t think you can truly “solve” any problem if you think about it.
- explaininjs 2y agoI'm talking satisfying business needs not philosophical mumbo jumbo. If you need a model to move, you give it to a rigger. If the mesh is bad, they will first need to remesh it or the rigging wont work right. This is the problem. They will solve this problem using a manual, labor intensive, process. It's not particularly difficult, any 3D artist considering themselves a professional ought to be able to do it. But it's not the sort of thing where you just press a button and turn some knobs and you're done either. It takes a lot of work, and in particularly bad cases it's easier to just start from scratch - remeshing a garbage mesh is indeed harder than modeling from scratch in many cases. Once either of these mechanisms have been applied, the problem will be solved and the rigger can move on to rigging. So yes, algorithms exist that pretend to remesh, and every professional modeling systems has one built in (because your sales guys don't want to be the ones without one), but professionals do not use them in production environments (my original claim, if you recall) because their results are so bad. Indeed I'm told several meme accounts exist dedicated to how badly they screw things up when folks do try to take the shortcuts. If this project was aiming to solve the problem (which is possible, as I have just explained), they would not have given up 5 years ago. Because it sure isn't solved now.
- spookie 2y agoI love that tool but it really doesn't fix bad topology. It gets you somewhere closer, but not a fix. Moreover, depending on what you have at hand, the resolution of your remeshing might destroy a LOT of detail or is unable to accomodate thin sections. Retopo isn't a solved problem. It only is for really basic, convex meshes.
- RicoElectrico 2y agoIt's an essential skill for reading scientific papers to notice what isn't there. It's as important as what is there. In my field, analog IC design, if we face a wall, we often do some literature review with a colleague and more often than not, results are not relevant for commercial application. Forget about Monte Carlo, sometimes even there aren't full PVT corners.
- jampekka 2y agoThis is indeed a side effect from research papers being read more outside academia (which is strictly a good thing in itself). In research one learns that most (almost all) papers oversell their results and a lot of stuff is hidden in the "Limitations" section. This is a significant problem, but not that big a problem within academia as everybody, at least within the field, knows to take the results with a grain of salt. But those outside academia, or outside the field, often don't take this into account. Academic papers should be read a bit like marketing material or pitch decks.
- zemo 2y agodepends what you're talking about and what your criteria is. In gamedev, studios typically use a retopology tool like topogun (https://www.topogun.com/ https://www.topogun.com/) to aid in the creation of efficient topologies, but it's still a manual task, as different topologies have different tradeoffs in terms of poly count, texture detail, options for how the model deforms when animated, etc. For example you may know that you're working on a model of a player character in a 3rd person game where the camera is typically behind you, so you want to spend more of your budget on the _back_ of the model than the _front_, because the player is typically looking at their character's back. If your criteria is "find the minimum number of polygons", sure, it's solved. That's just one of many different goals, and not the goal that is typically used by gamedev, which I assume to be a primary audience of this research.
- efilife 2y agoFyi, we use asterisks to put emphasis on text on HN
- TrevorJ 2y agoHard disagree, as someone in the industry.
- jsheard 2y agoCredit where it's due, unlike most of these papers they do at least show some of their models sans textures on page 11, so you can see how undefined the actual geometry is (e.g. none of the characters have eyes until they are painted on).
- SV_BubbleTime 2y agoSans texture is not wireframe though. They have a texture, it’s just all white. The wire frame is going to be unrecognizable-bad. Still a ways to go.
- tobyjsullivan 2y agoThey seem to admit as much in Table 1 which indicates this model is not capable of "clean topology". Somewhat annoyingly, they do not discuss topology anywhere else in the paper (at least, I could not find the word "topology" via Ctrl+F).
- dyauspitr 2y agoThat doesn’t matter for things like 3D printing and CNC machining. Additionally, there are other mesh fixer AI tools. This is going to be gold for me.
- jsheard 2y agoHowever if you 3DP/CNC these you'll only get the base shape, without any of the fake details it painted over the surface. Expectation vs. reality: https://i.imgur.com/82R5DAc.png https://i.imgur.com/82R5DAc.png
- dyauspitr 2y agoThat’s still not bad. I can use the normal and texture maps to generate appropriate depth maps to put the details in and do some final Wacom touch ups. Way better than making the whole thing from scratch.
- eropple 2y ago> That doesn’t matter for things like 3D printing and CNC machining It absolutely does. But great, let's look forward to Printables being ruined by off-model nonsense.
- SV_BubbleTime 2y agoIt matters so much more, GP is just being hopeful and soon to be disappointed.
- dyauspitr 2y agoWhy does it matter. As long as there are no holes, my vectric software doesn’t care.
- TylerE 2y agoIf your normals are flipped, your cnc cutter is going to try to cut from inside up to the surface. That's no bueno.
- torginus 2y agoAfaik, there's no topology - it outputs signed distance fields, not meshes.
- explaininjs 2y agoThis is incorrect. > Given a text prompt provided by the user, Stage I creates […] a 3D mesh.
- GaggiX 2y agoIn the comparison between the models only Rodin seems to produce clean topology, hopefully in the future we will see a model with the strength of both, hopefully from Meta as Rodin is a commercial model.
- cchance 2y agoYa would be cool if we had something open that competed with rodin, but just like elevenlabs for voice, seems closed is gonna be ahead for a while
- kgraves 2y agoCan this be used for image to 3D generation? What is the SOTA in this area these days?
- tobyjsullivan 2y agoThe paper suggests Rodin Gen-1 [0] is capable of image-to-shape generation. [0] https://hyperhuman.deemos.com/rodin https://hyperhuman.deemos.com/rodin
- Fripplebubby 2y agoI think what they did here was go text prompt -> generate multiple 2d views -> reconstruction network to go multiple 2d images to 3d representation -> mesh extraction from 3d representation. That's a long way of saying, no, I don't think that this introduces a component that specifically goes 2d -> 3d from a single 2d image.
- anditherobot 2y agoCan this potentially support : - Image Input to 3D model Output - 3D model(format) as Input Question: What is the current state of the art commercially available product in that niche?
- moffkalast 2y agoMeshroom, if you have enough images ;)
- egnehots 2y agoThis a pipeline for text to 3D. But it's using for 3D gen, a model that is more flexible: https://assetgen.github.io/ https://assetgen.github.io/ It can be conditioned on text or image.
- Simon_ORourke 2y agoAre those guys still banging on about that Metaverse? That's taken a decided back seat to all the AI innovation in the past 18 months.
- dvngnt_ 2y agozuck has said before that ML will help make the "metaverse" more viable. he still needs a moat with its own ecosystem like the iphone
- yieldcrv 2y agoMeta has spent like $50bn on their Metaverse line item since 2021 and hasn't stopped that probably means a bunch of H100's now for this Meta 3D Gen thing, and other yet unnannounced things still incubating in a womb of datasets
- localfirst 2y agocan somebody please please integrate SAM with 3d primitive RAGging? This is the golden chalice solution as a 3d modeler, having one of those "blobs" generated by Luma and likes aren't very useful
- rebuilder 2y agoI’m puzzled by the poor texture quality in these. The colours are just bad - it looks like the textures are blown out (the detail at the bright end clip into white) and much too contrasty ( the turkey does that transition from red to white via a band of yellow). I wonder why that is - was the training data just done on the cheap?
- firtoz 2y agoIt seems to be very well compared to the alternatives, however there's a long way to go forward indeed
- wkat4242 2y agoI can't wait for this to become usable. I love VR but the content generation is just sooooo labour intensive. Help creating 3D models would help so much and be the #1 enabler for the metaverse IMO.
- jsheard 2y agoVR is especially unforgiving of "fake" detailing, you need as much detail as possible in the actual geometry to really sell it. That's the opposite how these models currently work, they output goopy low-res geometry and approximate most of the detailing with textures, which would be immediately register as fake with stereoscopic depth perception.
- SV_BubbleTime 2y agoAgreed. Everyone I see text to 3D, it’s ALWAYS textured. That is the obvious give-away that it is still garbage. Show me text to wireframe that looks good and I’ll get excited.
- deleted 2y ago[deleted]
- spookie 2y agoYup. I'm doing a VR project, urban environment. Haven't really found a good enough solution for 3D reconstruction from images. Yes, there is gaussian splatting, NeRF and derivatives, but their outputs _really don't look good_. It's also necessary to have the surface remeshed if you go through that route, and then you need to retexture it. Crazy thing being able to see things up to scale and so close up :)
- ibrarmalik 2y agoBy output you mean the extracted surface geometry? Or are you directly rendering NeRFs in VR.
- iamleppert 2y agoI tried all the recent wave of text/image to 3D model services, some touting 100 MM+ valuations and tens of millions raised and found them all to produce unusable garbage.
- jampekka 2y agoThe gap from demos/papers to reality is huge. ML has a bad replication crisis.
- SV_BubbleTime 2y ago>The gap from demos/papers to reality is huge. SAI showed Stable Diffusion 3 pictures of women laying on grass. If you haven’t been following SD3… https://arstechnica.com/information-technology/2024/06/ridiculed-stable-diffusion-3-release-excels-at-ai-generated-body-horror/ https://arstechnica.com/information-technology/2024/06/ridic...
- freeone3000 2y agoThis is not a “replication crisis”. Running the paper gets you the same results as the author; it’s uniquely replicable. The results not being useful in a product is not the same as a fundamental failure in our use of the scientific process.
- jampekka 2y agoThat is reproducibility. Replicability means that the results hold for replication outside the specific circumstances of one study.
- fngjdflmdflg 2y ago>Replicability means that the results hold for replication outside the specific circumstances of one study. If by "hold for replication outside the specific circumstances of one study" you mean "useful for real world problems" as implied by your previous comment then I don't think you are correct. From a quick search it seems there are multiple definitions of Reproducibility and Replicability with some using the words interchangeably but the most favorable one I found to what you are saying is this definition: >Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. >[...] >In general, whenever new data are obtained that constitute the results of a study aimed at answering the same scientific question as another study, the degree of consistency of the results from the two studies constitutes their degree of replication.[0] However I think this holds true for a lot of ML research going on. The issue is not that the solutions do not generalize. It's that the solution itself is not useful for most real world applications. I don't see what replicability has to do with it. you can train a given model with a different but similar dataset and you will get the same quality non-useful results. I'm not sure exactly what definition of replicability you are using though if there is one I missed please point it out. [0] https://www.ncbi.nlm.nih.gov/books/NBK547546/ https://www.ncbi.nlm.nih.gov/books/NBK547546/
- 999900000999 2y agoWould love for an artist to provide some input, but I imagine this could be really good if it generates models that you can edit or start from later . Or, just throw a PS1 filter on top and make some retro games
- doctorpangloss 2y ago> for an artist to provide some input Sure, the results are excellent. > Or, just throw a PS1 filter on top and make some retro games There's so many creative ways to use these workflows. Consider how much people achieved with NES graphics. The biggest obstacles are tools and marketplaces.
- testfrequency 2y agoI question that you’re actually an 3D artist. I’m an artist (as is my partner) and we both agree this looks better than most examples..but it still looks incredibly lackluster, poorly colored, and texturally continues to have weird uncanny smoothness to it that is distracting/obviously generated. I don’t have time to leave a longer reply, and I still need to read over their entire white paper later tonight, but I’m surprised to see someone who claims to be an artist be convinced that this is “incredible”.
- raytopia 2y agoUnless the topology is good it may not be worth it.
- dorkwood 2y agoSure. The paper doesn't show topology, UVs or the output texture, so we're left to assume the models look something like what you'd find when using photogrammetry: triangulated blobs with highly segmented UV islands and very large textures. Fine for background elements in a 3D render, but unsuitable for use in a game engine or real-time pipeline. In my job I've sometimes been given 3D scans and asked to include them in a game. They require extensive cleanup to become usable, if you care about visual quality and performance at all.
- LarsDu88 2y agoThis is crazy impressive, and the fact they have the whole thing running with a PBR texturing pipeline is really cool. That being said, I wonder if the use of signed distance fields (SDFs) results in bad topology. I saw a paper earlier this week that was recently released that seems to build "game-ready" topology --- stuff that might actually be riggable for animation. https://github.com/buaacyw/MeshAnything https://github.com/buaacyw/MeshAnything
- jsheard 2y agoThe obvious major caveat with MeshAnything is that it only scales up to outputs with about 800 polygons, so even if their claims about the quality of their topology hold up it's not actually good for much as it stands. For reference a modern AAA game character model can easily exceed 100,000 polygons, and models made to be rendered offline can be an order of magnitude bigger still.
- LarsDu88 2y agoI do some 3d modeling on the side with my side project (https://roguestargun.com https://roguestargun.com), and I suspect those 800 polygons with good topology may be more useful to a lot of 3d artists than blobby fully textured SDF derived models. A low poly model with good topology can be very easily subdivided and details extruded for higher definition ala Ian Hubert's famous vending machine tutorial: https://www.youtube.com/watch?v=v_ikG-u_6r0 https://www.youtube.com/watch?v=v_ikG-u_6r0 And of course I'm sure those folks in Shanghai making the Mesh Anything paper did not have access to the datasets or compute power the Meta team had.
- surfingdino 2y agoNot sure how adding Gen AI is going to make VR any better? I wanted to type "it's like throwing good money after bad", but that's not quite right. Both are black holes where VC money is turned into papers and demos.
- Filligree 2y agoThe ultimate end goal is a VR game with infinite detail. Sword Art Online, however, remains fiction. Perhaps for the best.
- vletal 2y agoSeeems like simple enough 3D-to-3D will be possible soon! I'll use it to upscale 8x all meshes and textures in the original Mafia and Unreal Tournament, write a good bye letter to my family and disappear. I think the kids will understand when they grow up.
- f0e4c2f7 2y agoIs there a way to try this yet?
- carbocation 2y agoFor starters, I'd love to just see a rock-solid neural network replacement for screened poisson surface reconstruction. (I have seen MeshAnything and I don't think that's the end-game.)
- timeon 2y agoWhy these pages want to bother visitors with popups? Just use "only essential" as default.
- w_for_wumbo 2y agoI think this is another precursor step in recreating our reality digitally. As long as you're able to react to the persons' state, with enough metrics you're able to recreate environments and scenarios within a 'safe environment' for people to push through and learn to cope with the scenarios they don't feel safe to address in the 'real' world. When the person then emerges from this virtual world, it'll be like an egg hatching into a new birth, having learned the lessons in their virtual cocoon. If you don't like this idea, it's an interesting thought experiment regardless as we can't verify, we're not already in a form of this.
- polterguy1000 2y agoMeta 3D Gen represents a significant step forward in the realm of 3D content generation, particularly for VR applications. The ability to generate detailed 3D models from text inputs could drastically reduce the labor-intensive process of content creation, making it more accessible and scalable. However, as some commenters have pointed out, the current technology still faces challenges, especially in producing high-quality, detailed geometry that holds up under the scrutiny of VR’s stereoscopic depth perception. The integration of PBR texturing is a promising feature, but the real test will be in how well these models can be refined and utilized in practical applications. It’s an exciting development, but there’s still a long way to go before it can fully meet the needs of VR developers and artists.
- xena 2y agoGenerally these things are useless for 3d artists because the wireframe is useless for them.
- guiomie 2y agoThat would be great. I've learnt some Unity, building my own little VR game, and I dread having to learn Blender or any other tool to make more detailed shapes/models. I've tried a few GenAI tool to create 3D models and the quality is not useable.
- mintone 2y agoI've been bullish[1] on this as a major aspect of generative AI for a while now, so it's great to see this paper published. 3D has an extremely steep learning curve once you try to do anything non-trivial, especially in terms of asset creation for VR etc. but my real interest is where this leads in terms of real-world items. One of the major hurdles is that in the real-world we aren't as forgiving as we are in VR/games. I'm not entirely surprised to see that most of the outputs are "artistic" ones, but I'm really interested to see where this ends up when we can give AI combined inputs from text/photos/LIDAR etc and have it make the model for a physical item that can be 3D printed. [1] https://www.technicalchops.com/articles/ai-inputs-and-outputs-in-the-real-world/ https://www.technicalchops.com/articles/ai-inputs-and-output...
- milofeynman 2y agoAutodesk has been building practical 3d models for years with generative design. I have to imagine it's only getting better with these recent advances, but I'm not immersed in the space.
- slidehero 2y ago>I've been bullish[1] on this as a major aspect of generative AI for a while now, so it's great to see this paper published. Me too. My first thought when seeing 2D AI generated images was that 3D would be a logical next step. Compared to pixels, there's so much additional data to work with when training these models I just assumed that 3D would be an easier problem to solve than 2D image generation. You have 3D point data, edges, edge loops, bone systems etc and a lot of the existing data is pretty well labeled too.
- lallysingh 2y agoI'm excited to see the main deterrents to Indy gave dev: art and sound, get AI'd away. A single developer could use an off-the-shelf game engine and some AI generated assets (perhaps combining with whatever they can buy cheap) to develop some fun games. Still, getting from still models to something that animates is necessary:(
- 2y ago
- antman 2y agoWork like this is the only way to revive the now defunct Metaverse, I was wondering whether Meta would fund research such as this that could lower the financial barrier to entry for Metaverse participants
- floppiplopp 2y agoInteresting, but what are the practical uses of 3d assets beyond gaming, where does it create a real advantage over what we already use as visual information and user interfaces? I cannot see VR replacing the interactions we have. It requires cumbersome, expensive hardware, it floods the users with additional mostly useless information (image, sound, 3d itself) they have to process, it's slow and expensive to create and maintain, in short: it's inefficient compared to established tech, which will always run circles around the lame try of imitating real world interactions in a 3d virtual space. The potential availability of very expensively (in terms of computing power) generated assets doesn't change that. It's still hard to do right, and even if done right, it seems like only a gimmick hardly anyone can stomach for more then a couple of hours at best. It's information overload to most people, and they have better alternatives.
- Ukv 2y ago> Interesting, but what are the practical uses of 3d assets beyond gaming Probably many areas that we already use 3D assets/texturing for. Maybe objects to fill out an architectural render, CG in movies/TV shows, 3D printing, or just as an inspiration/mock-up to build off of. I'd imagine this generator is less useful for product design/manufacturing at the moment due to lack precise constraints - but maybe once we get the equivalent of ControlNets. If weights are released, it may also serve as a nice foundation model, or synthetic data generator, for other 3D tasks (including non-generative tasks like defect detection), in the same way Stable Diffusion and Segment Anything have for 2D tasks. > I cannot see VR replacing the interactions we have. It requires cumbersome, expensive hardware Currently sure, but it's been a reasonably safe bet that hardware will get smaller and cheaper. Something like the Bigscreen Beyond already has a fairly small form factor. But, I feel you're basing judgement of a 3D generator on one currently-niche potential use of 3D assets, that being VR/AR user interfaces (and in particular ones intended to replace a phone rather than, for instance, the interactive interfaces within VR games/experiences). > The potential availability of very expensively (in terms of computing power) generated assets doesn't change that Even just comparing computing power and not the human labour required, this is probably going to be an extremely cheap way to generate assets. The paper reports 30 seconds for AssetGen, then a further 20 seconds for TextureGen - both being feed-forward generators. They don't mention which GPU, but previous similar models have ran in a couple of minutes on consumer GPUs.
- tiborsaas 2y agoProbably this is the best way to build the Metaverse. Publish all the research, let people build products over it and soon we'll in need for a place and platform to make use of all the instant assets in virtual spaces. Well played, Meta.
- ziofill 2y agoIs this going toward 3D games entirely "hallucinated"? That would be amazing.
- nightowl_games 2y agoIs this just a paper or can I run the program and generate some stuff?