Science textbooks come alive via new, interactive AI technology

Students worldwide may soon access interactive, 3D versions of textbook diagrams, transforming how they learn physics. Thanks to a new tool, students could take static textbook images and convert them into living simulations—viewable with just an iPad.

Imagine opening your physics book to find a diagram of a skier perched on a hill. Instead of imagining the skier’s movement, this tool enables you to record the image with your device, set parameters, and watch the skier zoom down the slope and leap into the air.

Ryo Suzuki and his team at the University of Colorado Boulder are pioneering this new approach to make textbooks dynamic and immersive, taking home a “best paper” award for their work at the ACM Symposium on User Interface Software and Technology.

"Usually, those diagrams are fixed. We have to imagine what happens,” said Suzuki, an assistant professor in the ATLAS Institute and Department of Computer Science. “But what if we could take any static diagram from any textbook and make it interactive?”

Dubbed "Augmented Physics," the tool uses artificial intelligence (AI) to simulate physics concepts on screen, letting students see ideas like kinetic energy and momentum in action. Students and teachers select objects within a diagram, such as a skier and a ski jump, and assign each object a role. AI applies basic physics principles, such as gravity and friction, to make the skier zoom faster or slower with a few swipes.

Aditya Gunturu, a master's student and lead author of the study, says he once struggled with visualizing physics in high school. “I was so frustrated,” said Gunturu. “I wanted to visualize these concepts.” His mother, a biology teacher in India, faces similar challenges, often sketching concepts like cell division on a blackboard since textbooks can only show static images.

The team’s work is grounded in a need for accessible, interactive learning tools. Interactive simulations are known to help students grasp abstract concepts better than static images or videos alone. Suzuki’s project gives students a unique chance to actively engage with physics.

Instead of relying on pre-made simulations found online—which don’t always align with specific lesson content—students can now create their own custom simulations based on textbook images, merging hands-on experimentation with real learning materials.

Suzuki believes that “the future of education and learning should be more interactive and personalized.” He and his team designed their tool with input from both students and teachers. One educator noted how students could use it to perform "their own little experiments," encouraging self-driven exploration in a way static materials can’t.

The core technology behind Augmented Physics is Segment Anything, a model developed by Meta that allows users to click on an image and isolate particular objects. Using this, the tool can pick out components of a diagram, apply physics laws, and make objects move. It’s versatile, too—handling different types of diagrams, from optical prisms to electrical circuits. Students could simulate how light bends through a prism or observe electric currents running through a circuit.

The team even tested the tool on diagrams from Gunturu’s old physics textbook, bringing them to life with movement and energy. However, the technology isn’t perfect. Currently, the tool can only transform a new diagram into a fully functional simulation around 60% of the time. Suzuki’s team continues refining the program to improve accuracy.

In the long term, Suzuki and his colleagues hope to expand the tool’s scope beyond physics. They envision a future where any static diagram from any subject, be it biology or chemistry, could become a 3D experience. Students could see cell division unfold in biology or chemical reactions happen in real-time within chemistry textbooks.

“That’s our goal—to escape from static 2D into the interactive 3D world,” said Suzuki. Gunturu, meanwhile, knows that his mother is looking forward to the day he can design a similar tool for biology. “That would be her dream come true,” he said.

Interactive simulations, such as those enabled by Augmented Physics, have proven valuable for student understanding. Research has shown that engaging students in hands-on experiments with interactive tools can lead to a richer, more memorable learning experience. Studies have also found that such simulations help students understand complex concepts more deeply than textbooks or video lessons alone.

Unfortunately, creating these simulations typically demands time, programming skills, and resources most teachers and students lack. Many educators rely on pre-made simulations available online, but these don’t always match the specific needs of a classroom.

In a formative study conducted by Suzuki’s team, students reported difficulty finding simulations that accurately aligned with their coursework. They had to shift between their textbooks and external websites, which disrupted focus and limited learning.

This hurdle highlights the importance of Augmented Physics, a tool that offers personalized simulations based on the exact images in each student’s learning materials. By seamlessly integrating custom simulations into existing textbooks, the technology bypasses the hassle of generic online tools and creates a more immersive, cohesive learning experience.

Looking forward, the possibilities extend beyond individual classrooms. The tool could integrate with augmented reality (AR) technology to take immersion even further. Students might wear AR glasses to enter 3D versions of diagrams and simulations, making learning as close to real-world experience as possible. Suzuki’s vision is one where education transcends the limitations of paper and ink, merging digital interactivity with traditional learning materials.

Augmented Physics has begun to change how students learn physics, helping concepts like energy, motion, and electrical current come alive. As the tool develops, it promises to revolutionize how students engage with educational materials, turning passive reading into active discovery.

Note: Materials provided above by The Brighter Side of News. Content may be edited for style and length.

Like these kind of feel good stories? Get The Brighter Side of News' newsletter.