By Chris Kocher

October 21, 2020

With video editing software becoming increasingly sophisticated, its sometimes difficult to believe our own eyes. Did that actor really appear in that movie? Did that politician really say that offensive thing?

Some so-called deepfakes are harmless fun, but others are made with a more sinister purpose. But how do we know when a video has been manipulated?

Researchers from Binghamton Universitys Thomas J. Watson College of Engineering and Applied Science have teamed up with Intel Corp. to develop a tool called FakeCatcher, which boasts an accuracy rate above 90%.

Lijun Yin is a professor in Watson Colleges Department of Computer Science. Image Credit: Jonathan Cohen.

FakeCatcher works by analyzing the subtle differences in skin color caused by the human heartbeat. Photoplethysmography (abbreviated as PPG) is the same technique used for a pulse oximeter put on the tip of your finger at a doctors office, as well as Apple Watches and wearable fitness tracking devices that measure your heartbeat during exercise.

We extract several PPG signals from different parts of the face and look at the spatial and temporal consistency of those signals, said Ilke Demir, a senior research scientist at Intel. In deepfakes, there is no consistency for heartbeats and there is no pulse information. For real videos, the blood flow in someones left cheek and right cheek to oversimplify it agree that they have the same pulse.

Working with Demir on the project is Umur A. Ciftci, a PhD student at Watson Colleges Department of Computer Science, under Professor Lijun Yins supervision at the Graphics and Image Computing Laboratory, part of the Seymour Kunis Media Core funded by donor Gary Kunis 73, LHD 02 It builds on Yins 15 years of work creating multiple 3D databases of human faces and emotional expressions. Hollywood filmmakers, video game creators and others have utilized the databases for their creative projects.

At Yins lab in the Innovative Technologies Complex, Ciftci has helped to build what may be the most advanced physiological capture setup setup in the United States, with its 18 cameras as well as in infrared. A device also is strapped around a subjects chest that monitors breathing and heartrate. So much data is acquired in a 30-minute session that it requires 12 hours of computer processing to render it.

Umur has done a lot of physiology data analysis, and signal processing research started with our first multimodal database, Yin said. We capture data not just with 2D and 3D visible images but also thermal cameras and physiology sensors. The idea of using the physiology as another signature to see if it is consistent with previous data is very helpful for detection.

Umur Ciftci, a PhD student in computer science, poses for a 3D scan in Professor Lijun Yins lab at the Innovative Technologies Complex. Ciftcis doctoral thesis will focus on detecting deepfake videos. Image Credit: Jonathan Cohen.

Deepfakes found in the wild are many steps below the kind of quality that Yins lab generates, but it means that manipulated videos can be much easier to spot.

Considering that we work with 3D using our own capture setup, we generate some of our own composites, which are basically fake videos, Ciftci said. The big difference is that we scan real people and use it, while deepfakes take data from other people and use it. Its not that different if you think about it that way.

Its like the police knowing what all the criminals do and how they do it. You understand how these deepfakes are being done. We learn the tricks and even use some of them in our own data creation.

Since the FakeCatcher findings were published, 27 researchers around the world have been using the algorithm and the dataset in their own analyses. Whenever these kinds of studies are made public, though, there are concerns about telling malicious deepfake makers how their videos have been shown to be false, allowing them to modify their work to be undetectable in the future.

Ciftci is not too worried about that, however: Its not going to be easy for someone who doesnt know much about the science behind it. They cant just use whats out there to make this happen without significant software changes.

Intels involvement in the FakeCatcher research is connected to its interests in volumetric capture and augmented/virtual reality experiences. Intel Studios operates what Demir calls the worlds largest volumetric capture stage: 100 cameras in a 10,000-square-foot geodesic dome that can handle about 30 people simultaneously even a few horses once.

Future plans include volumetric-capture technology to be included in mainstream television shows, sports and augmented-reality applications, where the audience can immerse in any scene. Films in 3D and VR also are in the works, with two VR projects recently premiering at the Venice Film Festival.

By compiling the FakeCatcher data and reverse-engineering it, Intel Studios hopes to make more realistic renderings that incorporate the kind of biological markers that humans with real heartbeats have.

Intels vision is changing from a chip-first company to putting AI, edge computing and data first, Demir said. We are making a transformation to AI-specific approaches in any way we can.

(Interesting to note: Intels CEO is Bob Swan, MBA 85, who last year told the School of Management magazine Reaching Higher that intellectual curiosity is a wonderful and powerful thing to help you grow and develop and evolve over time.)

Future research will seek to improve and refine the FakeCatcher technology, drilling further down into the data to determine how the deepfakes are made. That capability has many implications, including cybersecurity and telemedicine, and Yin also hopes for further collaborations with Intel.

Were still in the brainstorming stage, he said. We want to have an impact not only in academia but also to see if our research would have a role in industry.

More here:
Best way to detect deepfake videos? Check for the pulse - Binghamton University