Sebastian Scherer

Associate Research Professor Carnegie Mellon University

  • Pittsburgh PA

Contact

Carnegie Mellon University

View more experts managed by Carnegie Mellon University

Biography

Over the last decade in my lab, I have achieved resilient performance of robots by advancing the robustness, redundancy, and resourcefulness of the algorithms as well as systems. While careful engineering is part of resilient performance, I am particularly interested in answering the following fundamental questions:

-How does one design algorithms and systems that are robust in the face of large uncertainty and learning-based components?
-Where can we inject redundancy into the system without incurring excessive computation or weight penalties?
-How can we move beyond fixed behaviors, policies, or interpretations of the data and have a continuous improvement of our systems to achieve resiliency in the face of large uncertainty with little data?

For more than a decade, I have made fundamental contributions to this new area of “resilient robotics” to answer those key research questions for SLAM, perception, and planning, by demonstrating pioneering results, as well as by evaluating the resilience in the context of applications such as subterranean exploration, search & rescue, triage, wildfire, safety in shared airspace, autonomous offroad driving, autonomous full-scale helicopter flight, bridge inspection, and drone delivery.

I explore resilient robotics by “grounding” research problems in impactful applications. My efforts are not limited by what is perceived as too difficult or too laborious. I embrace the challenge and build as necessary and leverage what already exists if possible. A large part of my effort goes into formulating what the core research problem is and then “cleaning up” these problems. Often, I find that existing problem formulations addressed in prior work have a fundamental gap in their assumptions to be able to be effective for relevant applications which require advancements in core methods. I strive to validate our methods in the field in closed-loop experiments, beyond benchmarking on datasets. I test early and test often since I have seen that these experiences lead to richer feedback for the systems, and as I gather more data, algorithms keep improving. It is now an exciting time since I can go beyond relying on smart engineering of solutions and can start making stronger assertions using large-scale evaluations.

Areas of Expertise

Intelligent UAVs
Computer Vision
AI Reasoning for Robotics
Multisensor Data Fusion
Aerial Robotics
Motion Planning
Robotics Foundations
3-D Vision and Recognition

Media

Social

Education

Carnegie Mellon University

Ph.D.

Robotics

Carnegie Mellon University

M.S.

Robotics

Carnegie Mellon University

B.S.

Computer Science (Minor Robotics)

Patents

Vehicle operator workload estimation system and method

12504820

2025-12-23

An estimation system includes a plurality of sensors that generate a multimodal signal, where the multimodal signal indicates a state of a user. The estimation system also includes at least one processor that receives the multimodal signal from the plurality of sensors, and determines a workload experienced by the user based on the multimodal signal and a workload model, wherein the workload model relates multimodal signal data to an experienced workload.

View more

Articles

Resilient odometry via hierarchical adaptation

Science Robotics

Shibo Zhao, Sifan Zhou, Yuchen Zhang, Ji Zhang, Chen Wang, Wenshan Wang, Sebastian Scherer

2025-12-10

Resilient and robust odometry is crucial for autonomous systems operating in complex and dynamic environments. Existing odometry systems often struggle with severe sensory degradations and extreme conditions such as smoke, sandstorms, snow, or low-light conditions, threatening both the safety and functionality of robots. To address these challenges, we present Super Odometry, a sensor fusion framework that dynamically adapts to varying levels of environmental degradation.

View more

AnyThermal: Towards Learning Universal Representations for Thermal Perception

arxiv

Parv Maheshwari, Jay Karhade, Yogesh Chawla, Isaiah Adu, Florian Heisen, Andrew Porco, Andrew Jong, Yifei Liu, Santosh Pitla, Sebastian Scherer, Wenshan Wang

2026-02-05

We present AnyThermal, a thermal backbone that captures robust task-agnostic thermal features suitable for a variety of tasks such as cross-modal place recognition, thermal segmentation, and monocular depth estimation using thermal images. Existing thermal backbones that follow task-specific training from small-scale data result in utility limited to a specific environment and task.

View more

Sebastian Scherer

Google Scholar

Full list of publications.

View more