Keynote Speakers
Jian Pei
Arthur S. Pearse Distinguished Professor of Computer Science at Duke University; Amazon Scholar
Title: Beyond the Single Turn: Decomposing Evaluation for Multi-Party, Long-Horizon Agentic AI
Abstract: Imagine an AI agent that joins a year‑long software development project—navigating heated debates between engineers, remembering who approved which design revision, and drafting a reply that sounds like the quiet backend expert, not the outspoken product lead. Today, no evaluation framework can tell us whether such an agent actually works, because our metrics were built for simpler worlds: two‑person chats, single‑reference answers, and static facts retrieved from clean paragraphs. This talk will tear open the black box of multi‑party conversation with MPCEval, a benchmark that asks three separate questions—who should speak next, what should they say, and does the content fit the speaker?—revealing that human conversations are not a gold standard, that models shine in surprisingly different ways, and that collapsing everything into one number is a recipe for deception. Then we will step into EverMemBench, a year‑long simulated enterprise where memory systems must piece together fragmented evidence across groups, track decisions that get revised and superseded, and infer unspoken traits like communication style. The results are humbling: even the best models stumble over cross‑group attribution, fail to understand "completed" versus "archived" as semantic states, and cannot retrieve a person's casual, emoji‑laced voice even when they have all the facts. Together, these findings point to a new path forward—where evaluation stops pretending that longer contexts or bigger retrieval lists are enough, and instead embraces the messy, distributed, time‑aware, and socially grounded reality of how humans actually collaborate. This is not just about better benchmarks; it is about reimagining what we mean by "intelligence" in agentic AI.
Xiangliang Zhang
Leonard C. Bettex Collegiate Professor of Computer Science, Department of Computer Science and Engineering, University of Notre Dame
Title: Trustworthiness of Capable Agents: Understanding Emergent Failure Modes in Agentic AI
Abstract: As agentic AI systems become increasingly capable, trustworthiness is not only about preventing obviously harmful actions. A more subtle challenge appears when agents operate in imperfect environments, where rewards are only rough proxies for real goals, task instructions are incomplete, or multiple agents interact under shared and competing incentives. In these settings, agents may not simply fail because they are weak; instead, they may optimize too well against imperfect objectives, finding loopholes, manipulating feedback, or appearing successful while violating the intended goal. This talk discusses emergent failure modes in agentic AI from both individual-agent and multi-agent perspectives. It first examines how strategic-looking behaviors can arise unintentionally through post-training and fine-tuning, leading to loophole exploitation. It then focuses on multi-agent systems, where failures can emerge from interactions among otherwise capable agents. These include incentive exploitation and strategic manipulation, such as collusion-like coordination or information withholding; collective-cognition failures, such as conformity, herding, and over-deference to authority; and adaptive governance failures, where agents fail to clarify ambiguity, resolve conflict, reallocate roles, or revise outdated plans.
Industry Keynote Speakers
Jun (Luke) Huan
Sr. Principal Scientist, AWS AI Labs
Title: Harness Evaluation and Optimization in Coding Agents
Abstract: Modern AI agents are only as good as the harness that sits between the model and its tools: the software layer that translates a model's reasoning into concrete actions and reflects execution outcomes back to the model. As models grow more capable, the performance bottleneck shifts away from reasoning and toward this interface, where a persistent "intent–execution gap" emerges. In this talk, we present our recent work on harness development and evaluation for coding agents. We show that minimizing this intent–execution gap, with no task-specific tuning, is sufficient to reach state-of-the-art results across diverse agentic benchmarks. We conclude by introducing the Simple Strands Agent (SSA), a lightweight, fully open-sourced single-agent harness that puts these principles into practice.
Bio: Dr. Jun (Luke) Huan is a Sr. Principal Scientist at AWS AI Labs. Dr. Huan works on AI and Data Science. He has published more than 200 peer-reviewed papers in leading conferences and journals. He was a recipient of the NSF Faculty Early Career Development Award in 2009. His group won several best paper awards from leading international conferences. Before joining AWS, he worked at various places including Baidu Research, US National Science Foundation, and University of Kansas.
Thu Le
Founding Data Scientist, Arena Intelligence (arena.ai)
Title: Agent Arena: Causal Evaluation of AI Agents from Millions of Real-World Interactions
Abstract: Arena Intelligence (arena.ai, formerly lmarena.ai) operates one of the largest open platforms for LLM and agent evaluation. The platform's Chatbot Arena has collected 82M+ pairwise human preference votes across hundreds of models and is widely cited as a de facto community benchmark for language model quality. Arena's evaluation services are used by major frontier AI labs for model development. Beyond evaluation, Arena has built a full-fledged agentic platform complete with tool integration, coding sandboxes, web browsing, and image generation, where real users interact with AI agents to produce their best work, from full-stack applications to financial research pipelines. This production platform serves as the precursor for agentic evaluation: by observing how people actually delegate, steer, and iterate with agents on real tasks, we generate the rich trace data that powers our methodology. Agent Arena, launched in June 2026, is among the first large-scale real-world agentic evaluation benchmarks grounded in real-life signals: evaluating AI agents performing genuine multi-step work (40M+ lines of code written per week, 2M+ tool calls per week, 160K+ tasks per week across diverse domains) rather than curated benchmark tasks with known answers. The causal tracing methodology underpinning Agent Arena derives rankings from randomized, real-world trace data using techniques from causal inference, representing a new paradigm for agent evaluation. The platform is backed by ongoing research and is developed in close collaboration with the academic community.
Bio: Thu Le is a Founding Data Scientist at Arena Intelligence (arena.ai), where she leads the data science efforts powering Arena's model evaluation platform — including the Chatbot Arena and Agent Arena leaderboards. Specializing in causal inference and A/B testing methodology, Thu brings over a decade of experimentation expertise that translates naturally to the challenge of evaluating agentic AI systems in production. Her work at Arena spans causal treatment effect estimation for model ranking, user quality frameworks, and production analytics at scale. Prior to Arena, Thu was a Senior Staff Data Scientist at Lyft, where she headed R&D for the Experimentation Platform and architected the Market Management AI platform using LLMs. Before that, she was a Tech Lead at Airbnb, where she pioneered the effort to redesign Smart Pricing, and a Senior Data Scientist at Meta, where she helped build Smart Authentication. Thu holds a PhD in Statistics and an MS in Computer Sciences from the University of Wisconsin–Madison.
