Today we explore agentic pipelines, loops and recursive self improvement (RSI) via our old friends, PMP Questions. This is article 4 in my AI agents series. Start here for an introduction, see here for an explanation of what constitutes a real agent, and here for a discussion of how I use them and running AI locally. This article gets into how to plan and manage agent work; that’s what “agent orchestration” means. As a project manager, I suspect you already have the thinking skills required.
You already know why one person can’t do everything on a project. You staff a project with specialists, engage each of them in the correct piece of scope, and put reviews between the stages that matter. Nobody expects the same person to design the building, dig the foundation, and inspect safety, because quality suffers when one person handles every task.
AI agent orchestration is the same idea, just applied to AI work instead of human work. I introduced my own agent team in article 3 of this series. Ned is the main agent, overseeing four specialists: Cody for coding, Viz for images, Lexi for writing, and Verity for fact-checking and research. That article covered what the team does and what it costs to run. This one covers how the team works together on a single task, because “orchestration” remains a mystery until you watch it solve a real problem.
The task I’ll use is one I mentioned in article 3 in passing: writing PMP exam-style questions. It’s a good case study for three reasons. First, a single chat window cannot do this well, and the reasons why turn out to be a checklist of orchestration principles. Second, it’s a problem space where I can easily tell if the AI’s output is weak. Third, readers are likely familiar with them.
What Actually Makes a PMP Question Good
I should explain some background. I’ve been a volunteer PMP question writer for PMI for over twenty years. I’ve sat through the psychometric training that comes with the role often enough that I now help advise new volunteer writers. I’ve also worked alongside the exam designers who build the ECO blueprints these questions are written against. There’s a whole science behind writing multiple choice questions, defined in a book by Thomas Haladyna that I have read and presented about multiple times.
None of this experience makes me special; many other PMI question writers know all this information too. However, I’ve seen the same mistakes often enough to recognize them quickly, so I can review agents’ outputs effectively.
The short version of the theory: every good exam question targets a specific level on Bloom’s taxonomy, the classic ladder from remembering facts up through applying, analyzing, and evaluating. Most PMP questions sit at the application level, testing whether a candidate can use a concept in a new situation, not just recall it.
The guidelines for constructing fair, defensible multiple-choice items were largely synthesized by Haladyna and validated over decades of research, notably with Michael Rodriguez. The result is a set of rules covering everything from how you word the stem of a question to how you avoid accidentally telegraphing (suggesting) the answer. My rule file is my own working adaptation of that lineage, built for PMI’s specific content and format.
Simple Rules, Then the Rules Few People Think About
Some of the rules are easy to state and easy to see once you know where to look. Don’t make the correct answer the longest option, since length alone becomes an obvious giveaway. Keep every option grammatically and stylistically parallel with the others, since a distractor phrased as a deliverable next to three actions stands out for the wrong reason before anyone reads the content. These are the kind of rules a volunteer writer picks up in an afternoon.
The nuanced ones take longer to learn. For instance, a question set in an agile context might carry a distractor written in predictive-process language, like a detailed upfront baseline. A candidate eliminates it by pattern-matching the vocabulary alone. A test-taker doesn’t need to know anything about agile to spot the option that sounds like it wandered in from a different chapter. The question’s own vocabulary hands them the answer. A good distractor in an agile scenario must sound plausibly agile and still be wrong for another reason, which is a much harder thing to write than it looks.
Then there are the rules that exist purely because internet groups have become organized. PMP prep content on Reddit, YouTube, and Udemy has, over several years, distilled a set of test-taking hacks. These let candidates eliminate wrong answers without knowing the underlying material at all. For instance: Never pick the option where the PM escalates to the sponsor. Never pick the one where someone gets fired. Never pick “do nothing.” Never pick asking for more time or budget before trying anything else. Never pick skipping a formal process. Favor whichever option sounds the most collaborative and least confrontational. Assess first, act second, always.
Individually, these guidelines aren’t wrong; they usually track sound project management instinct. The problem is that they’ve become so well known that a candidate can answer plenty of questions correctly by pattern-matching the tone of the options rather than understanding the scenario. If a question writer isn’t careful, they end up writing the exact question these guidelines were trained to beat. Four options, where the “nice,” collaborative-sounding one is always correct, and the escalation option is always a trap.
Which is exactly why the antidote to these hacks is deliberate. Every so often, the right answer should be to escalate, bring in HR or remove someone from the team. Sometimes that is the correct professional response, and a candidate who’s been taught to auto-eliminate it needs to be caught out. Savvy question writers write those questions on purpose now. The shortcut has become so well publicized that not using it against the test-takers who rely on it would be a missed opportunity. That’s not a trick question in the sense the guidelines above warn against. It’s a fair question that happens to punish an unfair strategy.
The Jagged Frontier, Applied to This Job
AI capability isn’t a uniform, advancing front; it’s jagged, strong in some places and surprisingly weak in others close by. Coding is currently one of AI’s strongest domains, approaching or exceeding the capabilities of a competent professional in many areas. General writing is good but not expert, maybe around 80% of what a solid professional editor would produce unassisted. PMP question writing probably sits somewhere in a similar band: competent, structurally sound, occasionally excellent, but with recognizable weak patterns that repeat.
Here’s the way I think about why that gap exists. If you asked me to write a chicken soup recipe by myself, I’d probably achieve 20%-30% of what a professional chef would produce. Chicken, water, some vegetables, herbs, seasoning how hard can it be? With AI’s help, I could probably get up to 75% of the chef’s result, which is a huge jump for almost no effort. However, the chef still beats me + AI. The chef knows which cut gives the best flavor, which joints add body to the stock, and which bones to leave out because they turn the broth bitter. That knowledge isn’t written down anywhere I’d think to look. It’s often tacit, earned by doing the thing badly a few hundred times.
I think I’m like the chef on PMP questions, after twenty years of doing it badly before I did it well. Most people, including a capable AI model, can produce a question that reads cleanly, has one defensible correct answer, and three distractors that aren’t obviously weak. They look at that and say, ‘good enough’. I notice the ones that smell wrong even when items on the checklist seem to pass. The ones where the agile question is secretly given away by a word inflection, or the never-escalate heuristic has quietly crept back in unchallenged. That instinct is the product of twenty years of doing. It’s also exactly why a human checkpoint stays in this pipeline, no matter how good the model writing the first draft gets.
Why One Question Is Much Easier than a Batch of Sixteen
Ask a chat AI for a PMP-style exam question, and you’ll get something plausible in thirty seconds. A short scenario, a question, four options, an explanation. The trouble starts when you need sixteen of them, spread across industries and lifecycles. All defensible against exam-writing standards, none of them accidentally leaking the answer through sloppy wording.
My question-writing rule file has 64 named requirements across seven categories, covering the simple rules, the nuanced ones, and the exam-hack countermeasures all at once. There’s a target mix of question difficulty, a required balance between predictive and agile project scenarios, and a list of distractor types. Industry-matching rules make sure a construction scenario doesn’t wander into agile software vocabulary. Every one of those 64 rules exists because, at some point, a real question failed without it.
A single chat session might usefully hold all of that in mind for the first few questions. By question five or six, something gives. Wording starts repeating, or industries cluster instead of spreading out. The balance between question types drifts. We can patch this at the prompt level (my previous approach) or break the task into an agent pipeline.
Why the Model Drifts, Specifically
It’s worth explaining why errors occur, because “the AI got confused” isn’t a satisfying answer, and it isn’t quite accurate either.
Every AI model works within a context window, its working memory for the current conversation. Add the rule file, the industry matrix, the distractor taxonomy, some good example questions, and the growing pile of draft output into one conversation. Now you’re asking the model to hold all of it in attention at once, while also generating new text.
Models don’t weight everything in that window equally. Instructions given early tend to fade in influence as the conversation grows, particularly once new content keeps landing on top of them. By the time the model is drafting question seven, the rule about industry repetition it read at the start is losing the fight for attention. It’s competing against six questions’ worth of scenarios, options, and explanations that have piled up since.
This isn’t a flaw you can prompt your way around by asking nicely or repeating the rules more forcefully. I spent more time than I would like to admit “chastising” various AI models when familiar errors reoccurred in questions. I can assure you that shouting, restating and loading the rules as project files in AI chat sessions is not the solution. All this information competes for precious context and processing space, which models serving millions of customers tightly constrain.
It’s a structural property of how these models process a conversation. Ask one agent to be a scenario writer, distractor engineer, and auditor simultaneously, and it has to share across all three roles at once. It gets worse for all three. It is similar to how a project team member splitting attention across three unrelated deliverables produces weaker output on all three than focused work on one at a time would.
One thing that all the various frontier AI models do agree on is the solution. To break down the process into multiple steps (e.g. divide batch across life cycles, analyze ECO task and example enablers, choose appropriate scenarios, craft question at appropriate Bloom’s level, etc.) So, instead of trying to write a good question all at once, break the process into stages and run it as a pipeline.
How the Pipeline Got to Seven Steps
I didn’t design this pipeline in one sitting. It started as the classic three-step pipeline: a Planner, a Generator, and a Reviewer. That seemed reasonable on paper but failed in practice for specific, traceable reasons.
The single Planner and Generator steps were the first casualties. Asking one pass to write the scenario, the correct answer, the three distractors, and the explanation all at once reproduced the exact drift problem I was trying to escape. It just had fewer intermediate checkpoints to catch it. The writer/generator would nail the scenario and the correct answer, then visibly run out of attention by the time it reached the distractors. The wrong answers that resulted were either too obviously wrong or accidentally almost correct.
So, I split writing into more stages. Questions first, correct answer only, nothing else. Distractors second, as a separate pass that only has to think about plausible wrong answers. It draws from an ideas list: a partial solution, a solution to the wrong problem, a tooling misuse, a timing error, and a handful of others. Rationale last, since explaining why an answer is right or wrong is repetitive, template-driven writing that doesn’t need much judgment once the question itself is settled.
The single Evaluator step was the second casualty. One review pass tried to catch everything: rule violations, weak distractors, answer leakage, industry imbalance, exam-hack vulnerability, all in one read. It caught the obvious problems but missed the subtle ones because it was reading for too many types of failure simultaneously. Splitting evaluation from revision fixed this. Verity (my researcher agent) now runs the 64-point audit and only flags what’s wrong, with the specific rule broken. Lexi (the writer agent) then applies just the fixes Verity recommended. Questions that pass skip Lexi entirely. Neither agent tries to do the other’s job.
The planner also added a checkpoint it didn’t originally have. Early on, a bad scenario chosen at the planning stage would only get caught after it had already been written, given distractors, and explained. That meant the entire four-stage downstream chain had to be redone. Now Ned produces the plan alone, and I read it. Nothing gets drafted until I’ve confirmed the scenarios are sound, including whether any are quietly vulnerable to the escalate-never heuristic. That check happens before a single word of the actual question exists. Two minutes of my attention at the cheapest possible point in the process saves an entire batch of wasted work later.
That’s how three steps became seven, not by adding process for its own sake. One failure at a time, I found exactly where a single step was quietly doing two or three jobs badly instead of one job well.
Why Good Examples Matter as Much as Rules
The rule file alone wasn’t enough, even after the pipeline was split into narrow stages. A rule like “avoid overlapping distractors” is an abstraction. Telling a model what to avoid in the abstract is weaker than showing it a concrete example of what good looks like.
So, alongside the rule file, there’s a second file of gold-standard example questions Ned reads before drafting. When Ned needs an example, whether a good agile communication question or a fair escalation-is-correct question that doesn’t tip into a trick question, the example lives there. It’s not just described in a rule; it’s demonstrated in full. This is the difference between telling someone “write clearly” and handing them a well-written paragraph to model against. Both matter, but the example is often more helpful.
This distinction turned out to matter enough to shape how the feedback loop responds to failures. When Verity flags a recurring problem, the fix isn’t automatically a rule rewrite. If the rule itself was ambiguous or wrong, that’s a rule fix. But if the rule was perfectly clear and the model still got it wrong, the problem is usually different. It needed a concrete example to anchor the pattern, not a clearer sentence explaining it. That gets added to the exemplar file instead. Diagnosing which of those two problems you have, before you fix it, is half the skill in maintaining this system.
Write Loops, Not Prompts
There’s a phrase going around in AI circles right now that I think is exactly right: stop writing prompts, start writing loops. A prompt is a one-shot instruction, you ask, it answers, the interaction ends, and whatever happened dies with it. A loop is a system that reads its own output, judges it against a standard, and changes its own future behavior as a result. The distinction sounds subtle until you watch what it does to error rates over time.
The Recursive Self Improvement (RSI) element of this pipeline is the clearest example of a real loop I run. When Verity finds a failure, the fix, a rule rewrite or a new exemplar, gets committed to the file the relevant stage reads. The next batch reads the corrected file and behaves differently. The same mistake that showed up in batch six does not reappear in batch seven. The system that produced it was permanently changed, not just told to try harder next time.
It’s worth being precise about what counts as a loop here and what doesn’t. The seven drafting stages themselves are a pipeline, not a loop; work flows forward from plan to published question and doesn’t cycle back on itself within a single batch. The loop is the layer sitting above the pipeline: the mechanism that watches what the pipeline produces across many runs and edits the pipeline’s own instructions in response.
A prompt can never replicate a loop no matter how well written. Six months of conversations with a chat AI teaches you something, cumulatively, but the model itself learns nothing that survives the session. Chat sessions teach the model nothing that persists once each session ends. My agent team’s memory lives in files on disk that every future run reads, not in a transcript that only I remember.
Local and Cloud, Matched to the Job
There’s a staffing decision buried in all this that any PM will recognize, not every stage deserves your most expensive staff/equipment/API tokens.
Ned runs the five drafting stages locally on my own hardware, at no per-query cost. Verity and Lexi, the audit and revision stages, run on frontier models in the cloud, where the extra judgment earns its value. Running the whole pipeline on the cloud model throughout would cost five times as much. Splitting it this way is much cheaper because the expensive resource is spent only where its extra capability changes the outcome. The structured, repetitive drafting stages don’t need it. The audit, where subtle rule violations and exam-hack vulnerabilities may hide, does.
Guidance for Managing Agent Teams
You don’t need your own local hardware, twenty years of question-writing experience, or a 64-rule specification file to apply the underlying pattern to whatever AI-assisted work you’re doing. Instead, look for where quality drifts across a long single response. If item four in a batch is visibly weaker than item one, that’s not a prompt-wording problem. It’s a sign the task is asking one pass to hold too much in mind at once and needs to be split.
Separate drafting from reviewing, always. Don’t ask the same conversation that wrote something to also grade it fairly.
Put your checkpoint where a wrong turn is cheapest to catch. Mine sits right after planning, before a single word of the actual question gets written, because catching a bad scenario there costs two minutes instead of an entire batch.
My Main Learnings
The important lesson wasn’t that good examples are always better than rules. It was learning to diagnose which one a given failure needed. If a rule was ambiguous, fixing the rule solved it. If the rule was already clear and the model still got it wrong, no amount of rewording fixed it; only a good example did. That diagnosis, not the rule file or the exemplar file alone, was the skill.
Most importantly, design the loop that catches and fixes your own failures; don’t just write a better prompt and hope it holds up next time. A prompt that you improve helps the next output, but a loop that reads its own mistakes and rewrites its own instructions helps every output after that, forever. You never have to remember what went wrong or write it down yourself.
We spent a lot of time in this article discussing how to write PMP questions, but the same concepts apply to performing any real-world task with AI. The devil is always in the details, and things are typically more complicated than we think. We start optimistically (how hard can a chicken soup recipe be?) and quickly get partway there by adding some generic AI assistance. Many AI adoptions fail at this stage as they are just not ready for prime time. However, structures and approaches exist to fix them.
None of these ideas needs new technical skills. It needs the instinct you already have for staffing a predictive project: decompose the work, assign narrow scope, and checkpoint the expensive risks. Then build a loop that remembers what the team learned last time, so nobody has to relearn it the hard way. Mine just happens to be named Ned, Verity, and Lexi instead of people on an org chart.
My Outputs
This pipeline now reliably churns out balanced sets of PMP-type questions that pass the 64 review rules. I still review them all, I still sometimes make edits before moving them to my PMIllustrated site. However, I no longer have to correct the same issues; I no longer argue with an AI chat interface, frustrated that it now applies rules 54-64 but seems to have forgotten rules 1-10. The agentic pipeline and learning loop just work.
Article 5 – How Do We Direct/Lead Agents?
I planned this as a four-part series, but I see now that I need a fifth. Much of what I learned from building this pipeline, I learned the hard way, through failed batches and pipelines that grew from three steps to seven because nothing simpler worked. However, smart people have already worked out better frameworks for much of this. My next article outlines two of them: a way of thinking about what an agent needs to do a task well, and a simple decision tree for how much you should let it run without you watching.
I also want to discuss the communication styles used with agents. We know that to get the best from a human team, they need inspiration, motivation and leadership. If we micromanage and give orders to human staff, we tend to get poor performance. Yet, when we take an interest in personal development goals, find work aligned to these goals, pose work as questions to be solved, empower, encourage and provide autonomy then performance usually increases.
So, if the future is managing teams that are part machine, those teams break into three groups:
- People
- People using AI as accelerators
- Pure AI agents
Do project managers employ a split management style, servant leaders to people and tyrants to agents? Some studies claim that threatening to terminate AI agents improves their performance; I suspect it also makes them more likely to cheat and lie.
Does servant leadership and empowerment work with agents or is simple goal-instruction the best approach? Either way, it seems project managers of the future will need a dual strategy.
If we cannot manage machines like humans, how do we adapt our leadership? As project managers, we may have to trade the role of an empathetic boss for that of a systems architect when dealing with our digital workforce.
These ideas might sound far-fetched and fanciful now. I am the first to admit that getting my agents to do a good job in a domain that I understand has been difficult. However, now they are genuinely useful workers. The process of setting up agents is currently time-consuming, but it will get better, and teaching/steering them will also become easier. I will leave you with another image.
It shows how New York’s Fifth Avenue traffic went from horse-drawn carriages to cars in 13 years.
Now, admittedly, this was Easter Sunday when the rich would parade on Fifth Ave with their latest fancy possessions. It took the rest of the country another 5-10 years to make the transition from horses to cars. However, we are already in a similar position with software development.
I think we are about to see a transition to AI-agent team members in the software industry over the next two years. In software-dominated knowledge-work industries in the next five years and in electronics manufacturing, medicine and pharmaceuticals in the next 10 years.
- Instagram reached 100 million active users in 5 years
- TikTok reached 100 million active users in just 9 months
- ChatGPT hit 100 million monthly active users in 2 months
Things are not slowing down; they are speeding up. How will you prepare?
