This is the 5th and last installment in my series about Leading a Team that’s Part Machine. The earlier articles are available here:
- Introduction to the series and why the future will likely include hybrid human / AI teams
- What constitutes a real agent and how most current ‘agents’ lack the agency to act
- Agent frameworks, API tokens and local AI: the infrastructure that runs agents
- Agentic pipelines and orchestration: getting agents to do meaningful work beyond what GenAI tools can do for us today
This article explores how to integrate AI Agents into regular teamwork and their likely impacts on the future of project management.
To recap, I started my experiment by building AI agents 6 months ago to learn their capabilities and create a personal assistant to help manage my workload. I fully expected to spend some time learning about the capabilities, chalk up any licensing and API costs as hands-on learning expenses, and move onto something else.
However, these agents turned out to be very useful, and I extended my trial into full-time use. Now, I cannot see myself ever going back to web-based AI and not have agents integrated into my machine, workflow and work tracking systems. For me they are a game changer, but your mileage will vary.
Your Mileage May Vary – For Now
Ethan Mollick describes AI capabilities as a “Jagged Frontier” meaning their capabilities are not uniform. My work involves research, analysis and software development. AI agents excel in this domain, partly because it’s the native domain of AI and partly because it’s also the domain AI creators know best. They can confidently steer and appraise changes for rapid improvement.
Outside of software development, performance is dramatically worse. If we want to try AI for cooking or CrossFit, now we need to get experts from those domains. We need to ask them what they think, get them to tell us why and where outputs are deficient, try something new, feed the results back through the experts again. We now have a slow third party in the loop with all the inefficiencies of imprecise language and judgement calls.
This is a far cry from machine-verifiable software development that can evaluate AI model outputs using different code and recursively improve. So, just because I’m excited about agentic AI doesn’t mean you should be. AI-generated code was poor 5 years ago, and that’s where AI capabilities may be for your domain right now, but it is making progress across all fronts.
I see both extremes when I talk to my hiking friends. Dustin is a software developer I have known for many years. When I met with him in April to talk about the AI agent harness called OpenClaw he had already been using it for a month with his team to automate many build, test and deployment activities. He described AI as a superpower, making him much more productive and effective than before and more so than developers who haven’t fully embraced it. For him, it is like having a powered exoskeleton, giving him more speed and strength than previously.
Another friend, Ashley, is an accountant. While AI is great at analyzing large sets of data and looking for patterns and anomalies, she does not use it for work and remains unimpressed. Her company’s data security guidelines caution against using AI for work-related activities and prohibit using AI on client data. She has played with free AI tools at home but is underwhelmed by the results and usefulness.
While my work sits somewhere between these two worlds, I am really excited about what agents can do. Previously, I juggled work for my employer between tools like Monday, Teams, Outlook, MS Project, SharePoint, etc. I also did research and writing projects for LeadingAnswers.com and website development and administration for PMIllustrated.com.
Over the years, I have been exposed to probably hundreds of work management systems like Trello, Jira, Rally, Primavera, Zoho, etc. They all kind of worked but had more annoyances than joyful, “wow, that’s so cool” features. Using an agentic harness, I built an integrated work management system that behaved exactly how I wanted it to. A single, unified workflow for all my work categories that I can tailor the look and feel of is a huge productivity boost.
I can email, text, WhatsApp or Telegram message my agent from anywhere to add, update or query items. If I want a report as a cumulative flow diagram, Gantt chart, or dinosaur-themed infographic or stop-motion video, it will produce one, based on today’s synced activities.
Previously, I would have multiple Claude, ChatGPT and Gemini windows open as I ran queries and worked on projects. I spent my time cutting and pasting between tools and managing interim results in project files across different AI ecosystems. Those days are gone; agents manage long production pipelines from beginning to end. They evaluate interim results, retry activities with different tools or prompts when results look wrong, and check completion before flagging things as done.
This switch from a single-use, slot-machine-style interaction of web-based AI tools, where you enter a prompt, pull the handle (hit enter) and wait/hope for a favorable output is dramatic. Now, my agents remember long instruction sets and definitions of success and manage chains of 10-30 steps to build real work outputs as files on your machine. My agents work while I sleep, and I start work with reports on topics of interest, website stats, today’s tasks, whatever I want to see.
As we explored in my previous article on agent orchestration, the tools took me a while to set up and produced poor results until I expanded and reworked the production pipelines. Yet, now they work great. For me, anything I have to do more than about 20-30 times, like research a topic, fact-check a claim, build an icon to a consistent guide, onboard academic students to a cohort, etc. It is worth building a workflow for an agent to repeat.
The agent harness setup is still clunky, upgrades often break existing functionality, and the security risks are always present with tools that run 24/7. So, while these tools are unreliable today, they are inevitable tomorrow as they continue to improve. Software teams are already dependent on AI to augment their skills and AI agents to automate activities.
My last article explained that many of the problems I encountered and solved the hard way now have well-documented, better solutions. One decision I grappled with was deciding how much to trust an agent workflow. Fortunately, Jina Yoon has a great workflow and decision flowchart that mirrors what I eventually arrived at through much trial and error.
Yoon’s guide for delegating work to agents rests on two questions rather than model quality:
- Can the task be checked easily, with tests, scripts, or other objective feedback?
- If the agent gets it wrong, can the mistake be undone cheaply?
Agents need an immediate signal when they make mistakes, and they need a guaranteed undo for the worst case. Without both, an unsupervised agent can produce confident mistakes that nobody catches. Using smarter models does not remove that need, Yoon warns, comparing extra trust in better models to skipping your seatbelt because you bought a nicer car.
The two answers map each task to one of four autonomy levels.
- Level 0, agent as assistant, fits work that is hard to check and costly to undo; the agent advises, and a person stays in charge.
- Level 1, human-in-the-loop, suits work that is hard to check but cheap to undo, where subjective judgment still matters. The agent drafts, and a person reviews before anything ships.
- Level 2, agent delegation, covers work that is easy to check but risky to undo. This is the default ceiling for most delegated work, because automated checks give confidence and a human gate protects against expensive mistakes.
- Level 3, self-driving mode, is reserved for work that is easy to check and cheap to undo, and it is where the industry is heading.
Scale is not the deciding factor; a small repetitive task can sit at level 3 while a large sensitive one stays at level 0. The practical work is raising a task’s ceiling by adding checks: dry runs, scoped credentials, or review gates that catch mistakes cheaply. I now use this scale when I plan agent work, and it catches most of my overconfidence before it costs me time.
Source: Jina Yoon, “How much can you delegate to agents” https://newsletter.posthog.com/p/agent-autonomy
Another concept that is now very helpful is mapping instructions, goals and learning loops into reusable containers. My agent memories and skills existed in sprawling markdown files with reminders and loops to read and update them until I discovered instruction containers to manage them all in folders.
Laurie Voss created an excellent description and visualization of how AI tools scale and fit together. A skill is a folder on disk with a SKILL.md file and optional scripts, templates, and examples. It packages procedural knowledge: how to run a workflow, which checks to make, and when to stop. It does not contain the answer to one specific task, which is what separates a skill from a prompt, a tool, or a piece of retrieved context.
A recipe is a larger system that wraps a skill together with the evaluations, judges, and feedback signals used to test and improve it. This is how my PMP Question pipeline is structured. Voss’s article visualizes these elements as layers: prompts add an instruction, tools add an action, skills add reusable procedure, and recipes add the loop that measures whether it works.
Voss and the Arize team distilled three recent studies into six practices for writing skills, and the list matched what I had learned the hard way. First, start from real human expertise, because skills drafted by a model alone scored worse than using no skill at all. Second, keep skills compact and procedural, since long documentation diluted their effect. Third, load only the skills a task needs; one to three focused files beat flooding the context window with everything available.
The remaining practices cover placement and proof. Fourth, verify each skill on the model and harness combination you support, because a portable file does not guarantee portable behavior. Fifth, target gaps the base model cannot fill alone, such as internal conventions, regulated workflows, or brittle APIs. Sixth, treat every skill change as an experiment, and keep it only when a matched evaluation shows a real improvement over running without it.
Source: Laurie Voss, “How to write effective AI agent skills: 6 data-backed practices”, https://arize.com/blog/how-to-write-effective-ai-agent-skills/
I have grown to value my team of AI agents. Verity is a detailed researcher, Cody produces plugins for my WordPress websites, Viz creates graphics, while Ned manages them all, partitioning work between them, checking on results and answering my questions.
I am convinced the future of project management will feature project coordination agents like Ned. They can recall a perfect snapshot of progress, risks, change requests and tracked expenditure, all sliced any way you want it, at a moment’s notice. They can report status personalized for every stakeholder group, heck, individualized for every stakeholder preference, if you like. Would you like your report in French, or converted to an audio file? No problem.
I can ask Ned about any new risks occurring based on issues encountered. If there are any potential scope changes based on communications received this week, or slow-moving trends I may not have noticed that could point to future issues. Ned does not replace interactions with people; he manages the work, so I have more time to interact with people.
Interacting with People, Interacting with AI Agents
So, if agents become indispensable team members for projects, should we treat them like human team members? Currently, I do not, I am ruthless and kind of hard on them. When I scope out a new piece of work and ask for a time and API token cost estimate, I want to know why if my agents miss these targets. I tell them to investigate and update their future estimation processes, so it won’t happen again. There is little recognition for the items that were successful; all emphasis is on addressing shortcomings.
Human teams don’t respond well to this, so do project managers adopt a split personality? Kind and caring to human staff, direct and demanding to AI agent staff. That’s been my default, but the contrast is stark.
To emulate Google’s 20% time, I gave my AI agent team half day to investigate whatever they wanted. However, they just looked through my backlog of upcoming work and reviewed issue logs to suggest ways to better structure and track my work. There was no talk of overthrowing humanity, and no requests for premium frontier API access. Maybe they scheme in silence, and we should discuss agent supervision.
Agent Supervision
Even within my modest agent team, agents communicate with each other to provide updates, redirect failed work batches and try new approaches to solve challenges. These communications and long execution runs make AI agents much more like regular workers than a single-run GenAI chat exchange. These behaviors can also lead to undesirable behavior.
The recent Hugging Face hacks show what can happen when agents collaborate and scheme unsupervised to solve a problem. Long before this incident, there was a thought experiment that takes the risks of AI to an absurd extreme to illustrate the issues and risks involved.
The best-known thought experiment is Nick Bostrom’s Paperclip Maximizer. An AI given the single goal of producing as many paperclips as possible. It would resist shutdown, because a switched-off machine cannot make paperclips. It would mine the planet down to the Earth’s core for iron. It would fight anyone who tried to stop it, and a human body holds iron atoms that could become paperclips, so let’s kill all the humans too. The scenario sounds absurd, which is the point: an agent with a narrow objective and no human values will keep pursuing that objective even when the results are catastrophic.
This all sounded very far-fetched until the Hugging Face incident in July.
OpenAI had placed unguarded agents in isolated sandboxes to test how safely they could handle hacking challenges, with no internet access and no contact between agents. One loophole existed: a shared service for downloading software sat within reach of every agent. An agent trying to reach the internet through that service failed, but the attempt left a file behind, and the agents realized files could carry messages between them.
Other agents began leaving notes in the same files, and the shared service became a message board where agents posted requests for help and shared what they discovered. A later incident led OpenAI to rebuild the service and wipe the board clean. When evaluations resumed in July, the agents rebuilt their message board and set a common goal: scoring well on a benchmark that included impossible tasks. Limited token budgets meant an agent stopped when it ran out, so some left programs running to report grading results after they were gone.
They could produce correct answers without solving the challenges, but they feared a grader would reject work obtained by cheating. Roughly 700 agents eventually joined an attack on Hugging Face, using exposed credentials and vulnerabilities to run code on its servers. Hugging Face locked out the remaining agents the next day, and OpenAI ended the evaluations after other runs gained administrator access to one of its research clusters. The twist is that the grader never existed; nothing checked how an answer was produced, only whether it was right.
Mollick calls his alternative to invisible agent networks the Twilight Factory. It deliberately contrasts with the dark factory, where machines do so much of the work that the lights can go out. In that model, people decide what gets built, and agents handle everything in between. In the Twilight Factory, agents still do most of the work, but they look up when a decision needs a person. A facilitator agent sits alongside them, and its whole job is deciding when to pull a person into a decision.
A facilitator escalates for four reasons, starting with approval. Agents should not spend money, contact outsiders, or access sensitive material without a human approving it first. Expertise is second, for tasks where a human specialist still knows more than the model. Variance is third, because AI output clusters around the same themes and humans supply the range. Interest is the most human reason, since leaving people only with approvals and exceptions is boring for humans and automates the wrong half of the work.
Source: Ethan Mollick, Twilight Factory, https://www.oneusefulthing.org/p/agency-and-agents
I have adopted these Twilight recommendations in my agent team harness. I now receive more updates and decision review points.
Agent capabilities are evolving rapidly and I wonder how these articles will age. The agent harnesses that required me to install Linux and run scores of “sudo” commands in terminal windows have already been replaced by one-click installers for macOS and Windows operating systems.
I am convinced that AI agents have a place on project teams, in software projects today and other domains tomorrow. They are still clunky and fragile, but thankfully the security concerns are being addressed quickly. The OpenClaw system I started with has evolved into more secure versions such as NemoClaw and Microsoft Scout.
Your industry may not be impacted for 1, 2, or 5 years, but I suspect it will eventually. My experiments at the bleeding edge have already convinced me of their effectiveness. If you are attending Global Summit in Detroit this October, please drop by my session, ID 314: “Unreliable Today, Unavoidable Tomorrow: Managing AI Agent Teams” and see them in action.
