与 Agent 一起规划：分割的世界、边界对象与更厚的界面

这场演讲讨论的是：与 Agent 协同规划——分割的世界、边界对象，以及更厚的界面。

但首先，让我们谈谈 Thomas Gladwin。他是一位人类学家，在 20 世纪 40 年代至 60 年代期间，曾生活在密克罗尼西亚的楚克人群体之中。

这些人生活在太平洋中央一组非常偏远的岛屿上。他研究的是：与欧洲航海者相比，他们如何在公海上航行。

欧洲航海者每次旅程都会先制定计划。他们根据地图和一般性的抽象原则规划航线，试图在出发前完成绝大多数思考。

一旦到了海上，他们会将每一步行动与计划对照，并努力保持在既定航线上。出现意外时，他们就不得不修改计划。

楚克航海者的做法不同：他们不会提前制定计划。

他们会先确定一个目标，例如抵达某一座岛屿，然后在没有固定路线的情况下出发。

相反，他们会随着情况出现而作出反应，阅读风、海浪、洋流、太阳和鸟类，并据此调整航向。他们持续思考如何抵达目标、即时作出决策，并不断根据情境即兴应对。

Suchman 是著名的人机交互（HCI）研究者和文化人类学家，曾在 20 世纪 80 年代和 90 年代任职于 Xerox PARC。她的著作讨论的正是如何与机器进行规划和沟通；尽管该书写于 1987 年，但对于解决我们当前的 Agent 问题依然高度相关。

我们总喜欢认为技术发展太快，历史对我们几乎没有意义。但实际上，我认为计算机科学领域往往没有吸取过去的教训；重新审视 20 世纪 80 年代和 90 年代的大量理论，或许能让我们受益。

总之，在书中 Suchman 借这个轶事讨论了计划能做什么、不能做什么。

这并不是在比较哪种航海方法更好。重点在于：即使你一开始制定了计划，一旦你开始在真实世界情境中采取行动，这个计划很快就会变得无关紧要。旅程中的每个人都在对环境作出反应、进行临时决策，不论他们是否有计划。

正如 Suchman 所说：“计划最好被视为临时行动的一种弱资源。”她还说：“无论我们中有些人多么像欧洲人那样说话，我们所有人行动起来都像楚克人。”

欧洲人的错误，在于假设一旦制定计划，就只需要执行它；假设地图就是疆域，现实会遵从你的预期。

换句话说，正如 Mike Tyson 所说：每个人都有计划，直到他们挨了一拳。

而在工程实践中，通常揍你的就是复杂性和棘手问题。

所以，这场演讲讨论的是：如何与 Agent 一起规划软件工程工作。这是我当前着迷的研究方向。

作为背景介绍，我是 GitHub Next 的设计师、工程师和研究员。GitHub Next 是 GitHub 的研发实验室，负责探索 GitHub 下一步该做什么，但关注的方向比常规组织所考虑的更具风险，也更为前瞻。

我目前研究的是，如何为人与 Agent 的协同规划打造更好的工具。

这里的“协同”指的是：人类和 Agent 处于同一语境中，所有参与者实时共同规划。

而现在，这件事做得实在很糟。

如今，当你或我开始与一个 Agent 规划时，我们会独自待在自己选择的 CLI 或桌面应用中，通常先进入某种规划模式，或者启用 /grill-me 技能，然后开始回答 40 个多项选择题。

大约答到第八题时，我们就累了，开始直接选择 A——也就是推荐选项。

可能是因为我们并不完全理解问题；也可能是因为所问的事情太复杂，我们无法通过这个界面进行推理；或者它超出了我们的知识范围。

最终，Agent 不再提问，给我一大堵 Markdown 文本让我阅读。然后它问：你是否批准这个计划，是还是否？

这里存在一些非常明显的问题。尤其是在 CLI 界面中，我无法直接编辑计划。尽管经历了那套连番盘问，计划中几乎总会留下开放问题，而我也没有回答这些问题的界面。

我无法在某个问题上暂停下来，说“这个问题我们需要深入一点”；也无法让一个 Agent 去研究它，或者帮助我对它进行原型验证。整个体验没有节奏。

这不是一个适合做出深思熟虑、知情决策的理想环境。而且，无论我做出什么决定，Agent 都会将其视为血书般不可更改。

它也诡异地私密。我的同事和我都在各自单独做计划，我们无法看见彼此的工作。我们无法征求彼此对这些决策的意见，甚至无法对此发表评论或展开辩论。

这不是一个成熟的流程。这不是一个好的流程。

那么，除去我对 CLI 的表层抱怨，这个系统真正的问题是什么？

第一个问题，是我们的欧洲航海者式方法。

我们是在为一段穿越代码库和运行时的想象旅程编写计划，试图在遭遇现实之前预测出最佳方案。

但这里还有一个不利的转折：制定计划的人和踏上旅程的人不是同一个存在。我们处于一种奇怪的境地——必须预先做出所有重要决策，因为 Agent 必须 独自踏上这段旅程。

它需要作出临时决策、采取情境化行动，但人类无法在场帮助它进行判断。

这很困难，因为如果要让结果变得良好、有用，人类的意图和愿望必须与 Agent 的情境化行动紧密一致。

第二个问题是，当前的规划媒介是一个设计糟糕的环境，不利于人类进行高质量思考。

现在的规划全是一堵堵文字。我们向 Agent 写一堵文字；它再向我们写回一堵文字。

人类并不是高效的阅读者。我们每分钟大约能读 240 个词，而一个 Agent 每分钟大约能写 4,500 个词，后者是我们的阅读速度的 19 倍。因此，Agent 生成的文本量与我们能够吸收的文本量之间存在巨大错配。

我们会疲惫、会略读，也会错过重要细节。这正是许多人报告在 Agent 式工作中出现倦怠和精疲力竭的原因之一。

他们整天都在被密集、难用的界面淹没的同时，试图做出艰难选择。这对于深度工作和复杂问题解决而言，是一种糟糕的媒介。

关于为什么这个问题难以解决，很大程度上可以归结为：人类与 Agent 是截然不同的存在，它们生活在非常不同的世界中，而且彼此只能部分理解对方。

让我们暂时回到 Suchman 的研究。

她做过一项著名研究：她录下两名博士研究人员尝试使用一台配有“智能专家帮助系统”的 Xerox PARC 复印机的过程。按照今天的理解，那可以算是原始 AI。

当然，他们花了很长时间试图按说明复印，却始终无法成功。

但这其实并不主要是一个界面设计失败的故事。它讲的是，人类与机器各自可获得什么信息，以及双方如何理解彼此的心智模型。

Suchman 指出，复印机无法接触到人类世界中大多数正在发生的信息，例如研究人员的犹豫、讨论、查看说明、做手势，以及他们对正在发生什么的预期表达。

它只能看到按键操作或他们打开纸盒。而同样地，人类只能看到屏幕上打印出来的消息，却看不到机器的内部状态或程序设计。

她说，这就好像双方都在透过一个钥匙孔看着对方。

对我来说，这与我们和 Agent 交互的方式并没有太大不同。

当然，自实体按钮和小屏幕的时代以来，我们的界面已经进步很大。我不想淡化那种体验与如今能够向 Agent 输入海量自然语言转录内容之间的差异。

但它依然像是通过一个输入框发送一条条简短文本消息。

仿佛我们正在这两个截然不同的世界之间传递纸条。

人类生活在一个由具身性、物理空间、丰富视觉、色彩、手势、情感、声音、表情、社会和文化构成的世界里。而这些恰恰都是 Agent 极不擅长理解的事物。

而 Agent 生活在一个由向量、权重、奖励、统计概率和梯度下降构成的世界里。

这两个世界各自都丰富而复杂，但没有任何一方能被另一方完全理解。我们 很难理解 Agent 系统内部正在发生什么；可解释性问题尚未解决。

Agent 可以 从我们这里获得不止文本。它们能理解和创建图像、音频和视频。但 Agent 中的 语言 模型部分依然是占主导地位的媒介。而我们知道，语言无法完整捕捉人类知识。

这就是为什么许多人正在尝试教 Agent 构建世界模型、解释物理现实。但在解决这个问题之前，挑战仍然是：如何将人类的意图、决策和专业知识传递给一个通过非常陌生、以语言为中心的视角看待世界的 Agent。

我认为，理解这一点的最好方式，是尝试想象一种超越当前界面能力的、由人和 Agent 共同参与的理想功能或产品规划方式。

这种愿望的一个小型版本是：我和一位同事在通话中，与 Agent 一起工作。

Agent 能够通过摄像头实时看见我们。

它们能看到我们的视线、面部表情和手势；能听到我和同事之间的对话，以及我们的语速、停顿和语音中的情绪；还能看到屏幕上的内容、我们的点击和悬停位置。

它们不必问我们自己的表现如何，或者我们是否喜欢结果。它们可以自行看到这些，并暂停、调整方向，或尝试修复问题。

这与我们当前拥有的能力相距甚远，但我认为在几年内仍是现实可行的。

但这个梦想甚至还不是真正的理想。它只是一个受限的版本，仍然在当前界面的可实现范围内。

如果不把自己严格束缚在当前限制下，我真正理想的问题解决环境是什么？

对我来说，那是在现实中与同事围坐在白板旁。我们交谈、指点、绘画、比划，观察困惑、兴奋、紧张和沉默；通过眼神接触引导房间中的注意力。多台笔记本电脑打开并运行原型。将屏幕投影到墙上，以便我们也能访问计算输出。

这近似于人类思考的理想情境。Agent 将无法理解其中的大部分内容，因为它没有接收到正确的输入。

一种解决方案是：摄像头、传感器、麦克风。让 Agent 能够看见和听见我们，观察我们的手，追踪我们的视线，并跟着我们在白板前移动。

这是一个非常泛在计算式的梦想，而我们没有实现它的理由有很多。

我刚才描绘的梦想，无论是小版本还是大版本，都充满了复杂挑战。Agent 需要实时处理、存储和理解的数据量极其庞大。这不仅包括物理输入，还包括社会情境、角色和背景。它充满了复杂问题。我不确定我们真的需要这些，才能实现我们想要的目标。

我们不需要将 Agent 完全带入我们的世界……

……也不需要完全理解它们的世界。

我们只需要在这两个世界之间建立有效的协作方式。

我们需要的是有效的 边界对象。

……也不需要完全理解它们的世界。

我们只需要在这两个世界之间建立有效的协作方式。

我们需要的是有效的 边界对象。

当博物馆创立时，它成功地将许多拥有不同文化、信念、训练背景和优先事项的社会群体聚集在一起，共同创建这座博物馆：生物学家、捕猎者、资助者和大学管理者。

所有人都能够通过边界对象进行有效协作。

标本、标签、田野笔记、绘图和地图——它们在每个社会世界中所承载的含义都略有不同。

它们对不同的人意味着不同的东西。生物学家和资助者对地图的解读不同，但他们对“加利福尼亚”这一概念有着共同理解。

这正是有效边界对象需要做到的事：适应每个群体的本地需求，为正确的人提供正确的信息，使他们能够共同构建一个共享现实。

我们当前与 Agent 之间的边界对象，是计划、提示词、技能和 AGENT.md 文件之类的东西。它们的效果还可以……

……但它们大多是为 Agent 看待世界的方式而设计的；它们优化的是 Agent 的表现。

目前，我们向人类和 Agent 展示完全相同的 Markdown 文件，并期待它对双方都同样有效。Agent 可以轻松处理大段文本、成堆的工具调用和调试日志。但对人类而言，阅读这些内容缓慢、疲惫且成本高昂。

我们是在让人类适应 Agent 理想的工作方式，而不是创造能够同时适应双方的对象。

换一种说法，我们在人类和 Agent 之间的界面非常薄。

人类所见与 Agent 所见之间，缺乏足够的转换。

我认为这些界面需要变得更厚：我们需要利用 Agent 相对廉价且近乎无限的劳动，构建服务于人类的丰富转换层。

This talk is about collaborative planning with agents: divided worlds, boundary objects, and thicker interfaces

But let’s first talk about Thomas Gladwin , an anthropologist who spent time between the 1940s and 1960s living among the Chuukese people in Micronesia.

These people live in a very remote set of islands in the middle of the Pacific Ocean. He was studying how they navigate the open seas compared with European navigators.

The European navigator would begin every journey by making a plan. They would plot a course from maps and general, abstract principles, trying to do most of their thinking in advance.

Once at sea, they would compare every move against the plan and work to stay on the course they had set. When something unexpected happened, they would have to revise the plan.

The Chuukese navigators work differently: they do not make plans ahead of time.

They instead begin with an objective, like reaching a certain island, and set off without a firm route.

Instead, they respond to conditions as they arise, reading the wind, waves, currents, sun, and birds and steering accordingly. They’re continuously thinking about how to reach the objective and making ad hoc decisions, constantly improvising based on the context.

Suchman is a famous HCI researcher and cultural anthropologist who worked at Xerox PARC in the 80s and 90s. Her book is all about planning and communicating with machines, and despite being written in 1987, it’s still very relevant to solving our current problems with agents.

We always like to think technology moves so fast that history is almost irrelevant to us. But actually I think we in computer science have often failed to learn the lessons of the past and could benefit from revisiting a lot of theory from the 80s and 90s.

Anyway, in the book Suchman points to this anecdote to talk about what plans can and can’t do.

It’s not about whose method of navigation is better. The point is that even when you make a plan up front, it quickly becomes irrelevant once you are taking situated action in a real-world context. Everyone on the journey is reacting to their environment and making ad hoc decisions, whether they had a plan or not.

As Suchman puts it, “Plans are best viewed as a weak resource for ad hoc activity.” She says, “we all act like the Chuukese, however much some of us may talk like Europeans.”

The European mistake is to assume that once you make a plan, you simply execute it. That the map is the territory, and reality will adhere to your expectations.

In other words, as Mike Tyson says, everyone has a plan until they get punched in the mouth.

And in engineering, you usually get punched by complexity and wicked problems.

So this talk is about planning software engineering work with agents. This is my current research obsession.

For context, I am a designer, engineer, and researcher at GitHub Next , which is the R&D lab arm of GitHub. We work on figuring out what GitHub should do next, but focus on things that are riskier and further out than what the usual organisation would think about.

And I’m currently researching how we make better tools for collaborative planning with agents.

And by collaborative, I mean humans and agents in the same context, all planning together in real time.

Right now, it’s really not good.

When you or I start planning with an agent today, we’re in our CLI or desktop app of choice, alone, and we usually begin in some kind of plan mode or with the /grill-me skill enabled, and then proceed to answer 40 multiple-choice questions.

We get tired somewhere around question eight and just start picking option A, the recommended option.

Maybe because we don’t fully understand the question, because we’re being asked about something so complex that we have no way to reason about it in this interface, or because it’s outside the scope of our knowledge.

Eventually, the agent stops asking questions and gives me a huge wall of Markdown to read. Then it asks: do you approve this plan, yes or no?

There are some very obvious problems here. Especially in a CLI interface, I can’t edit the plan directly. The plan almost always has open questions in it, despite the grilling process, and I don’t have an interface for answering them.

I can’t pause on any of the questions and say, we need to go deeper on this, or ask an agent to research this, or help me prototype this. There’s no pacing to the experience.

This is not an ideal environment for making thoughtful, informed decisions. And then anything I decide, the agent treats as if it’s as written in blood.

It’s also strangely private. My coworkers and I are all making plans separately, we can’t see each other’s work. We can’t ask for each other’s opinions on these decisions. We can’t even comment on them or debate them.

This is not a sophisticated process This is not a good process

So, aside from my surface-level gripes about CLIs, what’s really wrong with this system?

The first problem is our European navigator approach

We’re writing a plan for an imagined journey through a codebase and runtime. Trying to predict the best approach before encountering reality.

But we have the unhelpful twist that the planner and the traveller are not the same being. We’re in the odd position of needing to make all the important decisions up front because an agent has to go on the journey alone.

And it will need to make ad hoc decisions and take situated actions, but without the human there to help with judgement calls.

Which is hard because for the outcome to be good/useful, the human’s intent and desires need to closely align with the agent’s situated actions.

The second problem is that the current planning medium is a poorly designed environment for humans to do high-quality thinking in.

Planning right now is all walls of text. We write walls of text to the agent; it writes walls of text back.

Humans are not efficient readers. We can read about 240 words a minute, and an agent can write about 4,500, which is 19 times faster than we can read. So there’s a huge mismatch between the amount of text agents produce and the amount we can consume.

We get tired, we skim, and we miss important details. This is one reason why so many people report burnout and exhaustion with agentic work.

They’re trying to make hard choices all day while swamped in dense, hard-to-use interfaces. This is a poor medium for deep work and complex problem-solving.

A lot about why this is a difficult problem to solve boils down to humans and agents being very different kinds of beings, with very different worlds, that are only partially legible to one another.

Let’s go back to Suchman’s research for a second.

She did a famous study where she videotaped two PhD researchers trying to use a Xerox PARC copier that had an “intelligent expert help system” built into it. What we would now think of as primitive AI.

And they of course spend forever trying to make copies according to the instructions and fail to do so.

It’s not really a story about failed interface design, though. It’s a story about what information is available to the humans versus the machine, and their mental models of one another.

Suchman pointed out that the copier didn’t have access to most of the information about what was going on in the human world, such as the researchers hesitating, debating, looking at instructions, making gestures, and stating their expectations of what was happening.

It could only see button presses or them opening the paper tray. And in the same way, the humans could only see messages printed on the screen, but not the internal state of the machine or programme design.

She says it’s as if both sides were looking at each other through a keyhole.

This doesn’t feel all that different to me from how we interface with agents.

Of course our interfaces have come a very long way since the days of physical buttons and small screens. I don’t want to downplay the difference between that and being able to dump huge amounts of natural-language transcriptions into an agent.

But it does still feel like posting little text messages through an input box.

Like we’re passing notes between these two very different worlds.

Humans live in this world of embodiment and physical space and rich visuals, colour, gesture, emotion, voices, expressions, society, and cultures. All of which agents are terrible at understanding.

And agents live in a world of vectors and weights and rewards and statistical probability and gradient descent.

And both of these worlds are rich and complex in their own right, but neither of them is fully legible to the other. We struggle to understand what’s happening inside agentic systems. Interpretability is not solved.

Agents can get more than text from us They can understand and create images and audio and video But the language model part of agents is still the dominant medium. And we know language doesn’t fully capture human knowledge.

Which is why plenty of people are currently trying to teach agents to build world models and interpret physical reality. But until we figure that out, the challenge is to get human intent, decision-making, and expertise passed to an agent that sees the world through very alien, language-centric eyes.

I think this is best explained by trying to imagine your ideal way of planning a feature or product, with both people and agents, beyond what our current interfaces are capable of.

A small version of this desire would have me and a coworker on a call, working together with agents.

And the agents can see us using our webcams, in real time.

See our gaze, facial expression, and hand gestures. Hear what my coworker and I are saying to each other, as well as our pacing, pauses, and the emotional tone in our voices. As well as what’s on our screens, where we’re clicking and hovering.

They don’t have to ask us how they’re performing or whether we like the results. They can see it and pause, redirect, or try to fix the problem.

This feels relatively far from what we currently have, but still realistic within a couple of years.

But that dream isn’t even the actual ideal. It’s a constrained one that’s still plausible within our current interfaces.

What’s my true ideal environment for problem solving, without tying myself strictly to current constraints?

For me, it would be in person, around a whiteboard with coworkers. Talking, pointing, drawing, gesturing, noticing confusion, excitement, tension, and silence. Making eye contact to direct attention in the room. Multiple laptops open running prototypes. Projecting screens onto the walls so we can access computational outputs as well.

This is something like the ideal human context for thought. Agents would not understand most of it. It doesn’t have any of the right inputs.

One solution: cameras, sensors, microphones. Make agents able to see and hear us. Watch our hands, track our gaze, and follow us around a whiteboard.

This is a very ubicomp dream and there are plenty of reasons we don’t have it.

Both the small and large versions of the dream I’ve just painted are filled with complex challenges. The amount of data the agents would need to process in real time, to store, and to understand is immense. Not just the physical inputs, but the social context, roles, and backgrounds. It is awash in complex problems. I don’t know that we actually need it to achieve what we want.

We don’t need to fully bring agents into our world…

…or fully understand theirs

We just need effective ways to collaborate between the two worlds.

What we need is effective boundary objects .

…or fully understand theirs

We just need effective ways to collaborate between the two worlds.

What we need is effective boundary objects.

When the museum was being founded, it succeeded in bringing together lots of different social groups with different cultures, beliefs, training, and priorities to create it: biologists, trappers, funders, and university administrators.

All able to effectively collaborate through boundary objects.

Specimens, labels, field notes, drawings, and maps – they inhabited each of these social worlds slightly differently.

They meant different things to different people. The biologists and the funders read the maps differently, but had a shared understanding of the concept of California.

This is what effective boundary objects need to do: adapt to the local needs of each group and provide the right information to the right people, allowing them to create a shared reality.

Our current boundary objects with agents are things like plans, prompts, skills, and AGENT.md files. And they work okay…

…but they’re mostly designed for the agent’s way of seeing the world; they optimise the agent’s performance.

We currently show humans and agents the exact same Markdown files and expect it to work equally well for both of us. Agents can happily consume walls of text, stacks of tool calls, and debug logs. But for humans, reading all of that is slow, tiring, and expensive.

We are making humans adapt to the agent’s ideal ways of working, rather than making objects that adapt to both sides.

Another way to put this is that our interfaces between humans and agents are very thin.

There’s not a lot of translation between what humans see and what agents see.

I think they need to get thicker in the sense that we need to use the relatively cheap and infinite labour of agents to construct rich layers of translation that serve humans really well.

The agent is not the prioritised party in this interaction. We are.

The agent should be doing far more labour for human benefit.

So, what would thicker interfaces and better boundary objects look like?

First, they need to be optimised for human understanding and legibility. That means moving beyond text and making use of visual, spatial, interactive, and social ways of thinking.

Text is obviously brilliant. It’s precise, adaptable, and very good at expressing ideas. But it’s also a narrow channel for all the complex things we need to see and understand in software engineering.

Writing is only about 5,000 years old. But our complex visual systems go back more than 500 million years.

We’re evolutionarily designed to move through physical environments, track things on the horizon, notice colour and movement, judge distance, arrange objects in space, and compare shapes. Most of this happens quickly and automatically.

This is why visual representations work so well. We can map complex data to colours, shapes, and space to make things visible and tangible, showing us things we couldn’t see before.

It moves the work out of working memory and into the world.

And we have lots of visual languages for doing this.

Charts reveal comparisons. Maps show movement through space. Timelines make sequence and overlap visible.

Each representation reveals a different kind of structure. And software engineering has a moderate history of taking advantage of this.

For example, state machines

And architecture diagrams

And I’m seeing people explore novel, visual ways of showing agent activity, like Mindwalk , which traces an agent run, showing what files it touched in a codebase.

The best thing about this demo is it’s interactive. You can click around. Humans, of course, think best by doing. Point and gesture; we move things around to test what happens. We let feedback from the world update our understanding.

Ideally it would also be multiplayer! This should have lots of coworkers in here with us. We’re social animals and learn best when we’re developing ideas with other people, questioning each other, and building up shared understanding.

But most agent interfaces flatten all of this into a solo stream of text. So text is doing too much work while our visual, embodied, and social abilities sit mostly idle.

So better boundary objects should make relationships visible, let us manipulate the plan directly, and give people a shared thing to point at and discuss. That’s a concrete way to reduce the cognitive load of working with agents.

The second way to improve these objects is to solve our European navigator problem.

By the time we hand the plan over, it should already have made meaningful contact with the important parts of reality.

Luckily, this is not a nineteenth-century sailing expedition and we can explore the terrain ahead of time.

Planning agents already do this a bit: they search the codebase, read files, and trace dependencies. It gives them a useful aerial view and catches obvious issues.

But they currently stop at observation. They predict which implementation will work without making prototypes, running them, or comparing them based on outcomes.

They could instead do more upfront legwork by splitting the work among subagents on multiple Git branches, implementing many approaches at once, and then evaluating the results.

This brings reality into the planning process, while the human is in the loop and can help make informed decisions.

I’m sure you’re thinking “well, that’s going to get expensive.” Within Next , we have the philosophy that in order to invent the future you have to live in it. Pretend that agent labour is functionally free; eventually it’s going to become cheap and plentiful, so we should explore as if it is.

So what does this look like in practice? Here is a planning agent in a terminal, helping me develop a set of consistent shadows for my design system.

It’s offering me three options: do you want the shadows to be crisp, soft, or pronounced?

The way this decision is presented to me, in text in a CLI, makes it impossible to make an informed decision. I don’t know what the current shadows in my app look like, and I don’t know what each of these options it’s offering me is going to look like.

Obviously, for the shape of this problem, I need an interface more like this.

First, this is connected to my live app so it’s showing me the actual shadows currently in the system.

Then I can directly manipulate the qualities of these shadows and see the results live. Maybe I want to change the colour or intensity of the shadows. When I’m happy with them, I can save that decision back to the plan.

Similarly, let’s say I have these cards with a little animation when I hover over them. I should have a bespoke interface that helps me adjust the easing and speed of it.

Note that the number of options available to me in this interface is exponentially more than three.

These are the kinds of interfaces that meaningfully enable human decision-making. The interface should be grounded in reality (my codebase) and let me manipulate it to come to the right decision.

Of course, visual problems naturally lend themselves to interactive interfaces, so what about more abstract logic problems?

I was recently debugging some mess with a video player and a reducer and answering increasingly complex questions in the CLI. We clearly needed to rearchitect it. Again, this was not the right interface for the problem.

But a state machine would let me see and explore all the possible states of the component, see what happens when I add different transitions, and find out what’s happening with the buffering loop.

We already have good conventions for programming interfaces like state machines. We can now get agents to build this kind of exploration on the fly to help us solve a particular problem in the moment.

This brings the important parts of reality forward into the planning process, rather than letting the agent encounter them later, while it’s on the journey alone.

One last demo: I was working on some retry logic, and the agent asked me to choose between specific syntax like an options object, a wrapper function, or a client policy.

This is exactly the kind of thing that it’s hard to reason about the repercussions of from this interface.

This is a good example of when we can just have subagents split up onto separate branches and build all three options, evaluate the results, and report back to us.

So I had the agent try all three and then make an interactive report on how each would work at every call site and perform.

Option A covers all 60. Most changes are mechanical, but eight need a judgement call. Option B also covers all 60, but 16 silently change their behaviour. Option C initially seems promising because it’s only 14 changes and they’re all mechanical, but that’s only when you realise the other 46 call sites are unsupported.

It’s also given us some measurable metrics like the recovery P95 and the amplification that can help inform this decision. These will help us see that Option A is clearly the right choice.

The challenge between us and this more beautiful world of working in these rich interfaces is that agents aren’t good at making visual explanations right now. Certainly not without a lot of hand-holding from me and essentially designing all of these examples myself.

And secondly, we’d need to be strategic about inference time here so I’m not waiting ages for these explorations; we’d need to have agents work in parallel, five steps ahead, and make this while I’m answering other questions.

These kinds of rich, interactive plans are what I’m currently exploring in a project called Chopin . This is a multiplayer, real-time planning prototype that my coworker Krzysztof and I are making.

It’s currently a very basic proof of concept and exploration space.

You have your coworkers and agents writing plans together in a nice Markdown editor. But one that includes lots of interactive visuals. It’s built with MDX.

I didn’t touch on the social piece much in the demos, but a key part of having better decision-making interfaces is that your coworkers can see them too, debate them, and decide collectively.

You can see here I’m answering a decision, and then it records that I made this decision at this time and date.

If you want to read more about Chopin, we’ve published a writeup with our initial research and more demos on the GitHub Next site.

So let’s wrap up and review what we learnt:

• Plans orient action. They cannot predict the whole journey.
• Humans and agents do not need shared understanding. They need good boundary objects and thick interfaces.
• Make visual, interactive, and social boundary objects.
• Improve the conditions for humans to think within. The human is the preferential being. The agent can and should do 1000x more work to make the human more effective.

Things to ask yourself in whatever you’re building:

Thank you for reading! Look up GitHub Next for more of our research. And read our piece on GitHub Chopin to learn more about that project.