Point-E – An AI that Generates 3D Models

The world of artificial intelligence known simply as AI has been making news for some years now. However, it is only in recent times that it has impacted some of us working in non-IT fields, often making us wonder what could happen in 2023. If ChatGPT left us gasping with what it does and what it could do in the future, wait till you check out Point-E. 

Yes, this is being perceived as the next breakthrough in the world of AI. Point-E is an open-source machine learning system developed by OpenAI that creates 3D objects with just a text prompt. The idea  is still in the realms of development but experiments suggest that it can produce 3D models in a minute or two with a single Nvidia V100 GPU. 

Is it too good to be true though?

There is a catch though. Point-E does not create 3D objects as we know. It generates point clouds – data points in space – that are easier to synthesize from the point of view of computing as these do not capture an object’s fine-grained shape or texture. The researchers working on the project believe this is a smarter way as it reduces 3D generation of an object. 

However, the fact remains that this is indeed a limitation to overcome which the Point-E team has set up another AI system to convert these clouds into meshes, which are merely a group of vertices, edges and faces that define an object. However, this conversion and the outcome isn’t yet kosher as the model often creates distorted shapes. 

How does it all come together?

In addition to the mesh generation model, Point-E comprises a text-to-image model as well as an image-to-3D model. The first of these was trained on labeled images to understand the link between words and visual concepts. The second was programmed with images paired to 3D objects in order to make it associate one with the other. 

Once several million such objects and images along with associated metadata was parsed through the system. Point-E began generating colored point clouds that matched text prompts on several occasions, the research paper says. However, they admit that the outcome is far from satisfactory as Point-E fails to understand the image from the text-to-image model.

“While our method performs worse on this evaluation than state-of-the-art techniques, it produces samples in a small fraction of the time. This could make it more practical for certain applications, or could allow for the discovery of higher-quality 3D objects,” the researchers say in their note. 

What are the use cases and what about copyright?

Now, if you’re wondering what this is all about and who can use it, the note says Point-E point clouds could fabricate real-world objects while the mesh-conversion could find its way into gaming and animation development workflows. Of course, this isn’t the first of its kind as Google had released DreamFusion earlier in 2022, a generative 3D system, unveiled back in 2021. 

And what about copyright issues, given the extensively large market for 3D models. Does Point-E technology impinge on the creators’ intellectual property, given that modern generative AI borrows heavily from training data that these teams or individuals have created with their existing 3D models? 

More so, since Point-E doesn’t credit or cite any of the artists who may have imagined or influenced the creation of these models. In fact, even OpenAI seems to have side-stepped the matter for now as neither the research paper nor their GitHub page alludes to copyright.  

What they do refer to instead is the probability of Point-E suffering from biases inherited from the training data they’ve used as well as the absence of safeguards around models that may have been used to create dangerous objects. Maybe, that’s the one reason behind why the research team refer to Point-E as merely a starting point.