Shap-E OpenAI Tool Turns Text Into 3D Objects

OpenAI reignited global interest in artificial intelligence with ChatGPT, which generates journals, research papers, and texts from simple commands. Yet, the tech trailblazer continues to innovate with AI tools. That is why it recently launched Shap-E, a tool that turns text prompts into three-dimensional objects.

ChatGPT is already disrupting numerous industries, so imagine what Shap-E can do! It can become a game-changer for digital artists since it can “draft” objects to help them start projects sooner. Also, it could facilitate tasks for engineers and architects by making structures quickly instead of hand drawing them.

This article will discuss what Shop-E can do. More importantly, I will explain how to install and use it on your computer. This is a great topic for 3D modeling enthusiasts and others who want to see what’s next for artificial intelligence.

How does Shap-E work?

Photo Credit: digitaltrends.com

OpenAI researchers Jun Haewoo and Alex Nichol published a research paper regarding Shap-E, “a conditional generative model for 3D assets.” The program creates 3D models based on user prompts.

For example, tell it to “make a penguin,” and it will show you a 3D recreation of this arctic bird. Like other OpenAI programs, you gain better results by adding details.

For instance, it can create a chair that looks like a tree. Moreover, Shap-E can turn 2D pictures into 3D models, transform three-dimensional assets into something else, and re-render them.

You may also like: OpenAI Is Developing ChatGPT Robots

The new OpenAI tool saves 3D models as PLY files you can open in Paint 3D. The latter is a built-in app for Windows 11 computers. Alternatively, you can open them with other 3D modeling programs like AutoCAD.

You may also convert them into STL files acceptable to 3D printers, turning them into physical models. Unlike other OpenAI programs, it does not require an internet connection and runs locally on your PC.

However, you must allot several hours to configure Shap-E. You need significant time to install the program and its numerous dependencies. For example, you need the free 3D modeling app Blender to re-render 3D objects.

How do I use Shap-E?

OpenAI quietly dropped a new AI.

It's called Shap-E and it's all about text to 3D-models 👇pic.twitter.com/IDVxvOQNhA

— Product Hunt 😸 (@ProductHunt) May 8, 2023

The best thing about this project is that you can download and use it for free. You need a powerful computer running an NVIDIA GPU. More importantly, you must know how to use Linux because the current version has numerous problems in Windows. If you meet these requirements, follow these steps:

1. Install Anaconda or Miniconda in Linux via the Conda website.
2. Create a Conda environment named shap-e with Python 3.9.
3. Next, activate the shap-e environment.
4. Install Pytorch using this command: conda install pytorch=1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
5. Then, build Pytorch with this code: pip install “git+https://github.com/facebookresearch/pytorch3d.git”
6. Install Jupyter Notebook using Conda with this command: conda install -c anaconda jupyter
7. Clone the shap-e repo using this command: git clone https://github.com/openai/shap-e
8. Open the shap-e folder and install using pip using these codes: “cd shap-e” and “pip install -e”

How to setup Shap-E

Yes.

OpenAI releasing Text-to-3D soon: SHAP-E

They dropped a research paper and Github code for a model called SHAP-E for creating 3D models through text prompts.

We are probably 1-2 months away from seeing Text-to-3D printers, as in text-to-object. pic.twitter.com/1peku6whdP

— AI Breakfast (@AiBreakfast) May 5, 2023

Launch a Jupyter Notebook and go to the localhost URL the software shows you. Its name is “https://localhost:8888?token=.” Then, follow these additional instructions:

1. Browse to shap-e/examples and double-click “sample_text_to_3d.ipynb”
2. Then, highlight each code section and click the Run button.
3. Wait for it to finish. It should show four 3D shark models in your browser. Also, your examples folder will have four PLY files.
4. Next, double-click “sample_image_to_3d.ipynb” to try the image-to-3d script.
5. Afterward, highlight each section and click the Run button.
6. Change the image location to modify the image resolution.

You may also like: Spanish Startup Offers 3D Printed Steaks

You must run a specific Python script from Tom’s Hardware and save it as “text-to-3d.py” or another name. It will let you generate PLY files using text prompts. Click here to access it.

Then, run your program and enter a prompt when the program requires it. Shap-E will give a PLY output, which you can modify if you know Python.

What are the applications of Shap-E?

I'm experimenting with OpenAI’s Shap-E text-to-3d this weekend.

First results seems promising.

1. "a shark" 3d model as rendered by Shap-E
2. video passed through controlnet (3 models – canny, depth zoe and shuffle)
3. interpolated with RunwayML
4. original+final overlayed pic.twitter.com/D0J1Z4riZm

— fofr (@fofrAI) May 6, 2023

Text-to-3D programs will likely revolutionize industries that rely on 3D modeling. For example, you can use Shap-E to create temporary characters and assets for your movie.

This process is called storyboarding, and it allows a movie team to plan scenes before making 3D models. It could become faster and easier thanks to Shap-E.

You may also like: 3D Organ Printing Is The Future Of Cardiology

Engineers and architects could use text-to-3D to expedite their tasks. For example, civil engineers could create temporary buildings for city planning. Architects could submit ideas and see how it looks immediately instead of drawing them by hand.

Also, it could become a boon for educators. For example, teachers could promote creativity with Shap-E. It could turn student ideas into physical objects to encourage them to cultivate artistic skills.

Conclusion

OpenAI researchers published a paper explaining their latest project, Shap-E. It turns user text prompts into 3D models. Fortunately, you can download and install it today.

Note that Shap-E is a work in progress. The 3D objects may have numerous errors and flaws. Still, it would likely become a powerful tool as artificial intelligence advances faster.

You can read the complete research paper on the Cornell University webpage. Also, you can check more Inquirer Tech articles for the latest digital trends.

Frequently asked questions about Shap-E

How can I use Shap-E?

You need a powerful computer with an NVIDIA graphics card to use OpenAI’s text-to-3D model. Also, you must install numerous programs to use Shap-E’s other features. For example, you need Blender to re-render and transform 3D objects. If you want to make physical objects, you need to link Shap-E to a 3D printer.

Do I need skills to use Shap-E?

You need to know how to use Linux because the current version only functions properly on that system. Running it on Windows presents numerous problems. Also, you should know how to use Python because the tool requires inputting codes. Lastly, you need artistic skills to have visually appealing results from Shap-E.

Can I use Shap-E for free?

You can use this text-to-3D program for free on GitHub. Also, it does not require you to have a constant internet connection because it runs locally on your computer. Shap-E does not require a ChatGPT API, so it will not charge fees while in use. However, you may need to spend money on programs and devices for its other functions.