Graham Neubig
Associate Professor
- Pittsburgh PA UNITED STATES
Graham Neubig's research is concerned with language and its role in human communication.
Biography
Areas of Expertise
Media Appearances
AI agents get office tasks wrong around 70% of the time, and a lot of them aren't AI at all
The Register online
2025-06-29
Graham Neubig, an associate professor at CMU's Language Technologies Institute and one of the paper's co-authors, told The Register in a phone interview that the impetus for TheAgentCompany was a paper from researchers at OpenAI and the Wharton School of the University of Pennsylvania about all of the jobs that theoretically could be automated.
"Basically their methodology was that they asked ChatGPT whether the job could be automated," he explained. "They also asked people whether the job could be automated and then they said ChatGPT and people agreed some portion of the time."
Neubig, who also works at a startup building coding agents, said he was skeptical so he wanted to create a benchmark to test how well AI models handle knowledge work tasks. After around eight months of work, they released TheAgentCompany.
Initially, a software agent was able to completely finish about 24 percent of tasks that involved web browsing, coding, and related tasks.
"Recently, we tried a newer version of an agent and it got 34 percent," he said. "So it increased from like one quarter to one third. And that's after about six months. One thing that's been a little bit disappointing to me is this benchmark hasn't been picked up by the big frontier labs. Maybe it's too hard and it makes them look bad."
Neubig said he expects agents will become more capable in time but added that even imperfect agents can be useful, at least in the context of coding agents – a partial code suggestion can be filled out and improved.
For agents dealing with more general office tasks, the situation is different. "It's very easy to sandbox code and not have it affect anything outside of the sandbox," he said. "Whereas, if an agent is processing emails on your company email server… it could send the email to the wrong people."
That said, Neubig sees the adoption of the Model Context Protocol (MCP) as a positive development for agents because it makes more systems programmatically accessible.
AI isn't ready to do your job
Business Insider online
2025-04-22
AI Agents aren't ready to do your job. Researchers at CMU staffed a fake company with AI agents, and the results were disastrous. "While agents may be used to accelerate some portion of the tasks that human workers are doing, they are likely not a replacement for all tasks at the moment," said Graham Neubig (School of Computer Science).
Where DeepL Beats ChatGPT in Machine Translation with Graham Neubig
Slator online
2023-07-14
In this week’s SlatorPod, we are joined by Graham Neubig, Associate Professor of Computer Science at Carnegie Mellon University, to discuss his research on multilingual natural language processing (NLP) and machine translation (MT).
Angry Bing chatbot just mimicking humans, say experts
ARY News online
2023-02-18
“I think this is basically mimicking conversations that it’s seen online,” said Graham Neubig, an associate professor at Carnegie Mellon University’s language technologies institute.
The Latest in Translation Devices
The New York Times online
2019-11-07
These devices “bring us a bit closer to being able to travel to places in the world where people speak different languages and communicate smoothly with those who are living there,” said Graham Neubig, an assistant professor at the Language Technologies Institute of Carnegie Mellon University and an expert in machine learning and natural language processing.
Media
Social
Industry Expertise
Education
University of Illinois Urbana-Champaign
B.S.
Computer Science
2005
Kyoto University
M.S.
Informatics
2010
Kyoto University
Ph.D.
Informatics
2012
Articles
Scaling Evaluation-Time Compute with Reasoning Models as Evaluators
Findings of the Association for Computational Linguistics: ACL 2026Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Minkyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck
2026-07-02
Language model (LM) evaluators that generate chain-of-thought (CoT) reasoning are widely used for the assessment of LM responses. Simultaneously, increasing LMs’" thinking" time through scaling test-time compute has proven to be an effective technique for solving challenging problems in domains such as math and code. This raises a natural question: can an LM’s evaluation capability also be improved by scaling test-time compute? To answer this, we investigate employing reasoning models-LMs that natively generate long CoT reasoning-as evaluators. We explore scaling evaluation-time compute by using reasoning models to evaluate both the overall candidate response (ie, outcome evaluation) and the individual reasoning steps within it (ie, process evaluation). We observe that evaluator performance improves monotonically with the number of reasoning tokens generated, mirroring trends seen in LM reasoning. Furthermore, we use these more accurate evaluators to rerank multiple generations, and demonstrate that spending more compute at evaluation time can be as effective as increasing compute during generation for improving an LM’s problem-solving performance.
Openhands: An open platform for ai software developers as generalist agents
International Conference on Learning RepresentationsXingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, Graham Neubig
2024-04-24
Software is one of the most powerful tools that we humans have at our disposal; it allows a skilled programmer to interact with the world in complex and profound ways. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and effect change in their surrounding environments. In this paper, we introduce OpenHands, a platform for the development of powerful and flexible AI agents that interact with the world in similar ways to a human developer: by writing code, interacting with a command line, and browsing the web. We describe how the platform allows for the implementation of new agents, utilization of various LLMs, safe interaction with sandboxed environments for code execution, and incorporation of evaluation benchmarks. Based on our currently incorporated benchmarks, we perform an evaluation of agents over 13 challenging tasks, including software engineering (eg, SWE-Bench) and web browsing (eg, WebArena), amongst others. Released under the permissive MIT license, OpenHands is a community project spanning academia and industry with more than 2K contributions from over 186 contributors in less than six months of development, and will improve going forward.
Divergences between Language Models and Human Brains
Advances in Neural Information Processing Systems2024
Do machines and humans process language in similar ways? Recent research has hinted at the affirmative, showing that human neural activity can be effectively predicted using the internal representations of language models (LMs). Although such results are thought to reflect shared computational principles between LMs and human brains, there are also clear differences in how LMs and humans represent and use language. In this work, we systematically explore the divergences between human and machine language processing by examining the differences between LM representations and human brain responses to language as measured by Magnetoencephalography (MEG) across two datasets in which subjects read and listened to narrative stories. Using an LLM-based data-driven approach, we identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense.
Do llms exhibit human-like response biases? a case study in survey design
Transactions of the Association for Computational Linguistics2024
One widely cited barrier to the adoption of LLMs as proxies for humans in subjective tasks is their sensitivity to prompt wording—but interestingly, humans also display sensitivities to instruction changes in the form of response biases. We investigate the extent to which LLMs reflect human response biases, if at all. We look to survey design, where human response biases caused by changes in the wordings of “prompts” have been extensively explored in social psychology literature. Drawing from these works, we design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior, particularly in models that have undergone RLHF.
DIRE and its data: Neural decompiled variable renamings with respect to software class
ACM Transactions on Software Engineering and Methodology2023
The decompiler is one of the most common tools for examining executable binaries without the corresponding source code. It transforms binaries into high-level code, reversing the compilation process. Unfortunately, decompiler output is far from readable because the decompilation process is often incomplete. State-of-the-art techniques use machine learning to predict missing information like variable names. While these approaches are often able to suggest good variable names in context, no existing work examines how the selection of training data influences these machine learning models. We investigate how data provenance and the quality of training data affect performance, and how well, if at all, trained models generalize across software domains. We focus on the variable renaming problem using one such machine learning model, DIRE.
AmericasNLI: Machine translation and natural language inference systems for Indigenous languages of the Americas
Frontiers in Artificial Intelligence2022
Little attention has been paid to the development of human language technology for truly low-resource languages—i.e., languages with limited amounts of digitally available text data, such as Indigenous languages. However, it has been shown that pretrained multilingual models are able to perform crosslingual transfer in a zero-shot setting even for low-resource languages which are unseen during pretraining. Yet, prior work evaluating performance on unseen languages has largely been limited to shallow token-level tasks. It remains unclear if zero-shot learning of deeper semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, a natural language inference dataset covering 10 Indigenous languages of the Americas.
Can we automate scientific reviewing?
Journal of Artificial Intelligence Research2022
The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question “can we automate scientific reviewing?”, discussing the possibility of using natural language processing (NLP) models to generate peer reviews for scientific papers. Because it is non-trivial to define what a “good” review is in the first place, we first discuss possible evaluation metrics that could be used to judge success in this task.