MIT Researchers Develop Software to Reveal 'Hidden Information" in Videos

By Peter Bernstein February 21, 2013

It turns out that just zooming in on a picture, or in this case a series of video frames, while good for looking at those hard to see aspects of a subject, are about to take a leap forward. As highlighted in a recent article in the MIT News, “Researchers amplify variations in video, making the invisible visible,” researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) will present new software that, “amplifies changes in successive frames of video that are too subtle for the naked eye” at this  summer's Siggraph — the premier computer-graphics conference.

The software can exaggerate tiny motions. Examples cited include: the ability to actually "see" someone's pulse as the skin reddens and pales with the flow of blood, making visible the vibrations of individual guitar strings, or the breathing of a swaddled infant in a neonatal intensive care unit. The medical implications alone make this exciting.

How it works

In explaining the principles on how the new software works, MIT draws the analogy to an equalizer in a stereo sound system which boosts some frequencies and cuts others. In this case, the pertinent frequency is the frequency of color changes in a sequence of video frames, not the frequency of an audio signal.

The prototype software allows the user to specify the frequency range of interest and the degree of amplification. The software works in real time and displays both the original video and the altered version of the video, with changes magnified. Although the technique is ideal for regularly occurring events, if the range of frequencies is wide enough it can amplify changes that occur only once. 

The success of unintended consequences

As the article points out, this is a case where the unintended consequences of finding a solution to one problem lead to an equally large discovery. Graduate student Michael Rubinstein, recent alumni Hao-Yu Wu '12, MNG '12 and Eugene Shih SM '01, PhD '10, and professors William Freeman, Fredo Durand and John Guttag, have set out to have the system amplify color changes. However, they found it amplified motion as well. "We started from amplifying color, and we noticed that we'd get this nice effect, that motion is also amplified," Rubinstein says. "So we went back, figured out exactly why that happens, studied it well, and saw how we can incorporate that to do better motion amplification."Again as the article notes, using the system to amplify motion rather than color requires a different kind of filtration. It works well if motions are relatively small which is why this holds so much promise for a variety of commercial applications. Those cited include "contactless monitoring" of hospital patients' vital signs, monitoring infants who are born prematurely or otherwise require early medical attention, and to augment home video baby monitors for the home.  

MIT has prepared a video on the research that gives a nice explanation of the system.

 

Maneesh Agrawala, an associate professor in the electrical engineering and computer science department at the University of California at Berkeley, and director of the department's Visualization Lab, noted that, “This approach is both simpler and allows you to see some things that you couldn't see with that old approach."  He added that, "The simplicity of the approach makes it something that has the possibility for application in a number of places. I think we'll see a lot of people implementing it because it's fairly straightforward."

The improved quality of cameras (even those on our smartphones), and the proliferation of HDTV, have made us almost spoiled in terms of our expectations when viewing still and moving pictures. High resolution does enhance the entertainment customer experience. However, the new MIT software is not about entertainment, it has practical uses that are going to enable individuals and institutions to be much more proactive about a host of things. How this gets realized commercially, and how fast will be interesting to watch. The ability to improve medical care alone makes this something to keep a close eye on. There is tremendous value in making the previously invisible readily viewable. 




Edited by Brooke Neuman
SHARE THIS ARTICLE
Related Articles

Microsoft Research Project Allows for Inexpensive 3D Scanning from a Smartphone

By: Christopher Mohr    8/27/2015

It is now possible to perform 3D scanning from a smartphone, without additional hardware or an Internet connection, thanks to a new Microsoft Research…

Read More

Amazon's Scaled Back Consumer Device Efforts, Dash Button, and More

By: Paula Bernier    8/27/2015

Word is that Amazon is scaling way back on its consumer devices efforts, having let go of dozens of Lab126 engineers who worked on its Fire phone, acc…

Read More

The 4K War is Brewing, but Don't Expect a Crowned Winner

By: Special Guest    8/27/2015

The hype around 4K Ultra HD video is growing and we're seeing it gain traction in real ways. From the NFL Network and CBS using 4K cameras to capture …

Read More

Wallet Wars Part 2: Thanks to EMV, the Force is with Mobile Wallets

By: Special Guest    8/26/2015

In December 2015, when "Star Wars: The Force Awakens" hits movie theatres across the U.S., a very different type of force will 'awaken' the mobile wal…

Read More

Major Automakers Forge Alliance to Combat Cyberattackers

By: Joe Rizzo    8/25/2015

If you take a few minutes to think about what hackers go after, you'll realize that it is anything that has an Internet connection. Thanks to the Inte…

Read More