The University of Rochester is one of the nation's leading private research universities, renowned for its commitment to innovation, interdisciplinary collaboration, and academic excellence. Guided by its motto, "Meliora"—Ever Better—the University fosters groundbreaking research and transformative education across its schools and centers.

University of Rochester

New Row

Contact expert @[Full Name](fullname), @[Job Title](job_title) at @[Organization](organization) for media, speaking, professional opportunities and more.

@[Full Name](fullname) at @[Organization](organization) | Contact Form

Expert Inquiry

Open Sans

ACM Future of Computing Academy (ACM FCA)

Association for the Advancement of Artificial Intelligence (AAAI)

Association for Computing Machinery (ACM)

Institute of Electrical and Electronics Engineers (IEEE)

American Association for the Advancement of Science (AAAS)

The Association for the Advancement ofAffective Commuting (AAAC)

Buildup of speaking skills in an online learning community: A network-analytic exploration

Automated Dydadic Data Recorder (ADDR) Framework and Analysis of Facial Cues in Deceptive Communication

CoCo: Collaboration Coach for Understanding Team Dynamics during Video Conferencing

The What, When, and Why of Facial Expressions: An Objective Analysis of Conversational Skills in Speed-Dating Videos

Say CHEESE: Common Human Emotional Expression Set Encoder Analysis of Smiles in Honest and Deceptive Communication

M. Ehsan Hoque directs the Rochester Human-Computer Interaction Lab. 

His research focuses on designing and implementing new algorithms to sense subtle human nonverbal behavior; enabling new behavior sensing and modelling for human-computer interaction; inventing new applications of emotion technology in high-impact social domains such as social skills training, public speaking; and assisting individuals who experience difficulties with social interactions.

Data Science

Human Nonverbal Behavior

Interactive Machine Learning

Human-Computer Interaction

Computer Vision

Experimental Psychology

Social Skills Training

Faculty Profile

Personal Website

Smart Speakers Like Alexa and Google Assistant Could Tell if You Have Parkinson's

Connections: Can AI help us become more fair as a society?

Federal award establishes Parkinson’s research center at URMC

How to Spot a Liar: Experts Uncover the Signs of Deception-Can you see them?

Can YOU spot the liar? Researchers develop online game to help AI crack down on racial biases by analyzing over a million faces

Facial software knows if you have something to hide

M. Ehsan Hoque develops digital helpers that teach social skills

Meet the Star of TED 2020: A Glass App That Coaches You As You Talk

Automated Dyadic Data Recorder (ADDR) Framework and Analysis of Facial Cues in Deceptive Communication

Say CHEESE: Common Human Emotional Expression Set EncoderAnalysis of Smiles in Honest and Deceptive Communication

How Emotional Trajectories Affect Audience Perception in Public Speaking

Asaro Biggar Family Fellow in Data Science, Assistant Professor of Computer Science, and faculty member in the Goergen Institute for Data Science and Artificial Intelligence

M. Ehsan Hoque, a 2004 graduate from Penn State Erie, The Behrend College, is an assistant professor of computer science at the University of Rochester

The Penn State Alumni Association honored nine outstanding graduates with the Alumni Achievement Award during an official ceremony on March 31, 2017. 

Alumni Achievement Award recipients are nominated by an academic college or campus and invited by the president of the University to return to campus to share their expertise with students and the Penn State community. Honorees demonstrate to students that Penn State alumni succeed in exceptional fashion at an early age.

For more information, visit alumni.psu.edu/awards/individual/aaa.

2017 Alumni Achievement Award: Ehsan Hoque ’04

Ehsan Hoque has created two computer systems that train people to excel in social settings.

MIT Technology Review: Ehsan Hoque: Innovator Under 35

Someone is fidgeting in a long line at an airport security gate. Is that person simply nervous about the wait?

Or is this a passenger who has something sinister to hide?

Even highly trained TSA (Transportation Security Administration) airport security officers still have a hard time telling whether someone is lying or telling the truth – despite the billions of dollars and years of study that have been devoted to the subject.

Now, University of Rochester researchers are using data science and an online crowdsourcing to create a screening system that can more accurately detect deception based on facial and verbal cues. 

They also hope to minimize instances of racial and ethnic profiling that TSA critics contend often occurs when passengers are pulled aside under the agency’s Screening of Passengers by Observation Techniques (SPOT) program.

“Basically, our system is like Skype on steroids,” says Tay Sen, a PhD student in the lab of Ehsan Hoque, an assistant professor of computer science. Sen is lead author of two new papers  accepted for major computing conferences hosted by IEEE (Institute of Electrical and Electrionics Engineers) and ACM (Association for Computing Machinery). The papers describe the framework the lab has used to create the largest video deception dataset so far -- and why some smiles are more deceitful than others.

“A lot of times people tend to look a certain way or show some kind of facial expression when they’re remembering things,” Sen said. “And when they are given a computational question, they have another kind of facial expression.”

An advantage of the crowdsourcing approach is that allows researchers to tap into a far larger pool of research participants, and gather data far more quickly, than would occur if participants had to be brought into a lab, Hoque says. So far, the researchers have gathered 1.3 million frames of facial expressions from 151 pairs of individuals (dyads) playing the game.

Data science is enabling the researchers to quickly analyze all that data in novel ways. For example, they used facial feature analysis software to identify which of 43 facial muscles were being used in a given frame, and to assign a numerical weight to each.

The researchers then fed the results into the University’s supercomputer, using a machine learning technique called clustering, which looks for patterns without being assigned any predetermined labels or categories.

“It told us there were basically five kinds of smile-related ‘faces’ that people made when responding to questions,” Sen said. The one most frequently associated with lying was a high intensity version of the so-called Duchenne smile involving both cheek/eye and mouth muscles. This is consistent with the “Duping Delight” theory that “when you’re fooling someone, you tend to take delight in it,” Sen explained.

More puzzling was the discovery that honest witnesses would often contract their eyes, but not smile at all with their mouths. “When we went back and replayed the videos, we found that this often happened when people were trying to remember [something]” Sen said. “This showed they were concentrating and trying to recall honestly.”

So will these findings tip off liars to simply change their facial expressions?

Not likely. The tell-tale strong Duchenne smile associated with lying involves “a cheek muscle you cannot control,” Hoque says.  “It is involuntary.”

“In the end, we still want humans to make the final decision,” Hoque says. “But as they are interrogating, it is important to provide them with some objective metrics that they could use to further inform their decisions.”
 
Md Kamrul Hasan, Zachary Teicher, Minh Tran, Matthew Levin, Yiming Yang, Kurt Haut, and Raiyan Baten – all students in the Hoque lab – also contributed to the research.

Students featured in the video (in order of appearance): Diana Maloku '21, Kurtis Haut '18, Shagun Bose '20, Manny Rodriguez '18, Ethan Wei '19, Melissa Kagaju '20.

Subscribe to the University of Rochester on YouTube: https://www.youtube.com/channel/UCZRLVZGCUZWYUEj2XQlFPyQ
Follow the University of Rochester on Twitter: https://twitter.com/UofR
Be sure to like the University of Rochester on our Facebook page: https://www.facebook.com/University.of.Rochester/

Can you tell when someone is lying to you?

Select Inquiry Type