Associate Professor and Director of Machine Learning Carnegie Mellon University
Zico Kolter researches how to make deep learning algorithms more robust, safer, and understand how data impacts how models function.
Wired online
2025-04-13
Small Language Models (SMLs) are capturing the attention of researchers. Using less power than LLMs, they are not used as general purpose tools, instead they focus on narrowly defined tasks like summarizing conversations. "The reason [SLMs] get so good with such small models and such little data is that they use high-quality data instead of the messy stuff,” said Zico Kolter (School of Computer Science).
Wired online
2025-04-09
Zico Kolter, a Carnegie Mellon professor and board member at OpenAI, tells WIRED about the dangers of AI agents interacting with one another—and why models need to be more resistant to attacks.
Technical.ly online
2025-03-19
Zico Kolter (School of Computer Science) will join ex-Google chief Eric Schmidt's AI Safety Science program. Schmidt is spending $10M on fundamental research into safety problems in AI. Kolter's role will focus on AI attacks.
Pittsburgh Post-Gazette online
2025-03-12
Recapping the K&L Gates conference, this piece highlights Zico Kolter (School of Computer Science) and Carol Smith (Human-Computer Interaction Institute) debate of the risks of unregulated AI development. The discussion reflected the conference's broader theme about AI governance, safety and global competition.
CMU News online
2024-10-09
"Pittsburgh has positioned itself as a worldwide leader in AI, led of course by Carnegie Mellon's long-time leadership and dedication to the field. Starting with Allen Newell and Herb Simon's inspiration and initiative, to the founding of departments dedicated to AI like the Machine Learning Department and Robotics Institute, and continued with today's influence on Generative AI and creation of AI startups, CMU has been a driving force in AI since the field's inception. With the recent continued expansion and public awareness of AI, in addition to continually welcoming numerous AI focused businesses, startups and research facilities to the city, Pittsburgh itself is well-positioned to capitalize on our lasting contributions."
Bloomberg online
2024-08-08
“I think part of my value is being deeply involved and integrated in research, and at the forefront of what’s happening in the field of not just the deployment of AI but the academic research into AI,” he said.
Kosch online
2024-03-11
In our interview series “Thought leaders in AI”, we had the opportunity to talk to Zico Kolter, Chief Scientist for AI at Bosch, about his personal view on various topics in the field of artificial intelligence. An AI system played the moderator and asked him questions on various exciting topics: How does he see the differences in AI development between Europe and the USA? Which celebrities would he like to meet one day? And finally: Which Bosch product does he particularly like?
C3.ai online
2023-11-06
Zico Kolter, an associate professor of Computer Science at Carnegie Mellon and author of the report, Universal and Transferable Adversarial Attacks on Aligned Language Models, put it bluntly: “These tools are attack vectors,” he said.
ZDNET online
2023-07-27
"There is no obvious solution," Zico Kolter, a professor at Carnegie Mellon and author of the report, told the Times. "You can create as many of these attacks as you want in a short amount of time."
1 min
Carnegie Mellon University’s artificial intelligence experts come from a wide range of backgrounds and perspectives, representing fields including computer science, sustainability, national security and entrepreneurship. Ahead of the AI Horizons Summit highlighting the city's commitment to responsible technology, CMU experts weighed in on why they see Pittsburgh as a hub for human-centered AI.
Ph.D.
Computer Science
2010
B.S.
Computer Science
2005
AI Horizons Pittsburgh Summit Pittsburgh, PA
2024-10-14
AI Horizons Pittsburgh Summit Pittsburgh, PA
2024-10-14
2024
Large language models (LLMs) trained on web-scale datasets raise substantial concerns regarding permissible data usage. One major question is whether these models "memorize" all their training data or they integrate many data sources in some way more akin to how a human would learn and synthesize information. The answer hinges, to a large degree, on how we define memorization. In this work, we propose the Adversarial Compression Ratio (ACR) as a metric for assessing memorization in LLMs. A given string from the training data is considered memorized if it can be elicited by a prompt (much) shorter than the string itself -- in other words, if these strings can be "compressed" with the model by computing adversarial prompts of fewer tokens. The ACR overcomes the limitations of existing notions of memorization by (i) offering an adversarial view of measuring memorization, especially for monitoring unlearning and compliance; and (ii) allowing for the flexibility to measure memorization for arbitrary strings at a reasonably low compute.
2024
Despite being trained specifically to follow user instructions, today's instructiontuned language models perform poorly when instructed to produce random outputs. For example, when prompted to pick a number uniformly between one and ten Llama-2-13B-chat disproportionately favors the number five, and when tasked with picking a first name at random, Mistral-7B-Instruct chooses Avery 40 times more often than we would expect based on the U.S. population. When these language models are used for real-world tasks where diversity of outputs is crucial, such as language model assisted dataset construction, their inability to produce diffuse distributions over valid choices is a major hurdle. In this work, we propose a fine-tuning method that encourages language models to output distributions that are diffuse over valid outcomes. The methods we introduce generalize across a variety of tasks and distributions and make large language models practical for synthetic dataset generation with little human intervention.
2024
We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers.
2024
Large language models trained on massive corpora of data from the web can memorize and reproduce sensitive or private data raising both legal and ethical concerns. Unlearning, or tuning models to forget information present in their training data, provides us with a way to protect private data after training. Although several methods exist for such unlearning, it is unclear to what extent they result in models equivalent to those where the data to be forgotten was never learned in the first place. To address this challenge, we present TOFU, a Task of Fictitious Unlearning, as a benchmark aimed at helping deepen our understanding of unlearning. We offer a dataset of 200 diverse synthetic author profiles, each consisting of 20 question-answer pairs, and a subset of these profiles called the forget set that serves as the target for unlearning.
2024
Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. In recent times, data curation has gained prominence with several works developing strategies to retain 'high-quality' subsets of 'raw' scraped data. For instance, the LAION public dataset retained only 10% of the total crawled data. However, these strategies are typically developed agnostic of the available compute for training. In this paper, we first demonstrate that making filtering decisions independent of training compute is often suboptimal: the limited high-quality data rapidly loses its utility when repeated, eventually requiring the inclusion of 'unseen' but 'lower-quality' data. To address this quality-quantity tradeoff (QQT), we introduce neural scaling laws that account for the non-homogeneous nature of web data, an angle ignored in existing literature.
