Confidently Wrong - Post 2
In the first post about my Global Summit session “Confidently Wrong: Avoiding GenAI Pitfalls with SME-First Thinking” we summarized how GenAI works and how its training influences the results returned. It focused on how Large Language Models (LLMs) are pattern matchers that return likely and plausible results, but do not truly understand project management.
LLMs have been trained on vast datasets which make them great for suggesting project artifact templates, but usually poor at filling in the details for your unique project. Also, due to their training materials, they tend to suggest summarized content of popular ideas, unless specifically forced to think deeper and broader. Finally, while their outputs look plausible, we always need a human-in-the-loop to verify.
This article explores the main ideas of my Global Summit session, the neuroscience impacts of using AI individually and in a team setting. Often it is safe to do so, but in several areas we risk undermining empowerment and autonomy.
While this post contains cautions about using AI in team settings, I am not anti-AI. In fact, I now interact more with AI than my teammates. I have worked predominantly from home for the last 10 years. I have a three-monitor setup. My center screen is for what I am working on, maybe an article or presentation. Off to the left I have reference documents, old articles, etc. I used my right monitor for monitoring emails, MS Teams and Slack messages from co-workers.
This setup has worked well for me. I can get into a flow of work on my center and left monitor and check my right hand one between tasks, or when I want a break. However, in the last 18 months this has changed. My new right-hand collaborator is GenAI as a thinking partner.
I now interact with GenAI almost continuously as I work. It does small things like finding quotes, recalling research, creating mind maps from lists, etc. Yet, it has overtaken the interaction I have with my teammates. So, I am a fan, just with some cautions.
Effort builds Recall
My first experience of AI impacting memory retention happened about 5 years ago, before ChatGPT and the other GenAI models became popular. GenAI writing tools were available back then, but operated in a batch process. You submitted your request and were notified when it had been completed, and you could download it (a far cry from today’s conversational models.)
I often write silly poems for my wife, who is an English literature fan. So, for her birthday, I used one of these early AI writers to write her a poem in the style of one of her favorite poets. She liked it and a week later asked to hear it again. Not only could I not remember it, but I could also not remember a single line from it and was truly busted. Now we have had ChatGPT for three years and everyone is familiar with this phenomenon. No investment, no retention.
The Cornell University study “Your Brain on AI” researched this topic. Using EEG machines to test samples of volunteers as they created essays. One group worked unassisted, one group used a simple internet search to research topics then wrote the essays themselves, the final group used Large Language Models (LLMs) like ChatGPT to help write their essays.
The EEGs showed less brain activity during the essay writing process when using GenAI than people working unassisted. Fewer areas of the brain showed connections or activity, as people did not have to work so hard.
A side effect of this reduced effort was a matching inability to recall or answer questions about the essays that had been generated.
In a project management setting this can be significant. I am not so worried if we use GenAI to generate meeting minutes. (I have not manually created meeting minutes for 15 years anyway, but if someone thinks they might be useful, have at it.) Meeting minutes are usually a low consequence deliverable, if we use LLMs to make them and have difficulty remembering them, that’s not a big problem.
However, a Project Vision Statement is a very different animal. We use these to articulate the Why and the Where for our projects, they need to be memorable. To quote Kouzes and Posner (co-authors of The Leadership Challenge) a clear vision “Reveals a beckoning summit towards which others can chart their own course.” In other words, it guides people in their day-to-day decision making. If people cannot remember our vision statement, or project goals, what is guiding their choices?
Involvement and Investment
When we use AI to generate project plans, we get low commitment from the team.
If we do not engage the team in the planning process they have nothing invested in it and no intrinsic motivation. The same is true for estimates, risk response strategies and customer messaging. Low involvement leads to low commitment since we did not consult the people doing the work.
At best we hope they execute the work out of a sense of being a responsible employee. However, we are missing the ownership and drive that comes from autonomy and empowerment.
The graph above shows that as team’s involvement in the creation process of plans, estimates and decisions increases so does their commitment to them. These are “the team’s” plans and decisions, forged through debate and trade-off discussions. They have been vetted by what is practical and what we agree on.
The 11th Agile Principle
The 11th principle of the Agile Manifesto says:
“The best architectures, requirements, and designs emerge from self-organizing teams.”
This does not mean best technically. I am sure there are many clever architects that can devise better technical solutions than my team. Instead, it means best in terms of likelihood to succeed. Due to the fact they have the support and buy-in from those tasked with doing the work.
As I wrote, nearly 20 years ago, in the article Team Solving, when we engage the team we gain a set of Team Solving benefits:
- By asking the team for a solution we inherit consensus for the proposal
- Accessing a broader knowledge of the facts
- The solutions are practical
- When consulted, people work hard to generate good ideas
- Asking for help shows confidence, not weakness
- Seeking ideas models desired behavior
When we use GenAI to create plans we gain some speed at the expense of team engagement and these Team Solving benefits. Sometimes this is acceptable, few people care how a project manager generates project update communications if they are correct. However, when it comes to things that impact the team such as the approach, plans, estimates and decisions then people care a lot.
This is not rocket science. If something impacts the team significantly, we need to involve the team significantly. Yet, we see AI guidance espousing the benefits of applying AI accelerators everywhere. What do you think will happen to the visible artifacts we see project managers deliver, if they lack the underlying collaboration, analysis and coordination required to support them?
They sink. They fail as soon as project reality hits. Yes, GenAI can create vision statements, charters, plans, estimates, budgets and schedules in minutes that would normally take a project manager weeks to create. However, they are worthless, if we did not engage the right people appropriately in their creation.
During my talk I used the Eisenhower quote “Plans are worthless, but planning is everything” to illustrate this point.
It is the process of planning, working with stakeholders, understanding their perspectives, and learning what is practical for our capabilities and constraints that adds value. The plan is always “today’s best guess” that will be wrong tomorrow, but as long as we know how to plan collectively, this not a problem.
Again, I am not here to knock AI, instead help people use it effectively. Going fast without the support of your team is not progress it is just frustrating and you redo, replan and regenerate plans every time they get proved wrong by project realities and the people omitted from the decision making process. Worse, each redo reduces your credibility as another go-round gets promoted that will inevitably be redone yet again.
Save yourself the frustration and loss of trust. Engage the involved stakeholders when the outcome impacts others. During the session we examined Sam Kaner’s Diamond Model of participation. It describes the Diverge / Converge phases of team ideation, discussion and agreement.
After introducing the problem, the team enters a Divergent phase where they generate many different perspectives. Some will be impractical or maybe even impossible, that is OK, we are ideation phase, no silly ideas. This often leads to the “Groan Zone” a phase where we have many options but no clear consensus on how to proceed.
Good facilitators manage teams through this transition from the Groan Zone to convergence through techniques like grouping and dot-voting. The team consolidates the options and refines them. Emerging with a solution that has been collectively vetted and assessed as the best practical option.
We examined Wicked Problem Solving as a practical implementation of the Kaner Diamond model. We also looked at the Ringelmann Effect that describes the tendency for people to work hardest individually and slack off sometimes when suggesting ideas in groups.
Wicked Problem Solving (WPS) is a visual collaboration toolkit. Project managers can use it to run pre-built “plays” which are facilitated sessions, but it boils down to group facilitation using sticky notes and dot-voting. Anyone who has participated or facilitated group sessions using physical or electronic sticky notes can use these approaches.
Before getting into the recommended techniques, we discussed a couple of additional psychological teamwork phenomena including:
- Anchoring – Being overly stuck to an initial set of suggestions and not moving well beyond them
- The Halo Effect (also called the HIPPO effect for Highest Paid Person’s Opinion) – Not wanting to contradict a senior contributor with your alternative ideas
Using AI Effectively in Teams
The largest component of the session addressed the ADVANCE mnemonic as a suggested way of effectively using GenAI in team settings.
A = Alone first – Use AI as your personal thinking tool and collaborator. Double check and refine your own suggestions so you are the best individual contributor you can be.
Then, as a project manager, when doing work that impacts the team or involves the team, facilitate their involvement. Encourage your team members to contribute ideas individually first to avoid the Ringelmann effect, anchoring and the halo effect. This is the Diverge portion of Kaner’s Diamond model.
D = Discuss in Groups – Bring insights, share ideas, and discuss constraints. This group process increases our shared understanding by adding the collective knowledge of our local SMEs.
V = Vet Suggestions – Here the team discusses what is practical, what can we do with the skills, tools and materials available. We eliminate the unlikely and ground options to those we can afford as we start to converge on viable alternatives.
A = Align Views – Dot-voting is the most common, team-consensus way of aligning on the chosen solution. In some environments experts may have the final decision, but to build true alignment and commitment we really want this to be a team decision so everyone is invested.
N = Name an Owner – As GenAI improves, having a human in the loop to verify the results is more important than ever. We need to check for bias, our local constraints and regulatory implications beyond just basic hallucinations and mistakes. We cannot assume someone else has checked an artifact. It is becoming more important than ever that we name the person-in-the-loop to ensure this critical step happens and the owner validates any updates.
C = Check with GenAI – To many people this seems awfully late in the process, but for me, this is where we engage AI after local SME input. Ask GenAI to look for flaws, gaps, risks and ways to improve our collective-made artifacts. Nudge it to think creatively from adjacent domains, provide examples and list sources for additional research.
E = Express Graphically – Visuals boost recall and help reduce confusion. If we have a common model to point at, it is less likely I will be imagining thing A when someone is talking about thing B. GenAI is great at creating visuals. Challenge the team to flex their AI skills to create the best artifact visuals.
In my session, we did some Slido surveys to understand how participants are currently using GenAI in team settings, then walked through the SME-First ADVANCE process example for a fictitious team building a smart home automation system.
The left most collection of blue stickies represent team members contributing individually first. WPS talks about the Silent Sort where we encourage individual contributions first and the book Collaboration Explained calls for Silent Grouping
Then the team members use dot-voting to indicate their preferences. This vetting process builds consensus as the team converges on the best options.
Once the team aligns on the priorities, it’s time to name the human-in-the-loop. This person is accountable for checking for bias, errors and omissions, both for the initial version and subsequent updates. When we flag our artifacts this way, no readers need to wonder “Did anyone check this?” or “Can I trust it?”
Now we use GenAI to check our work, think of new ideas, look for improvements, risks and novel suggestions from adjacent fields. However, GenAI does not know our domain, organization, or project specifics like the team does. So, many suggestions may be irrelevant or impractical. Just like we did for human generated content, engage the team in grouping, sorting and dot-voting the best ones.
The last step in the ADVANCE process is to express your final artifact visually. This could be work on a Kanban board as down here, in a WBS, or an infographic. Visuals help identify gaps in understanding. Techniques such as Design The Product Box are as much about surfacing issues early, as they are about practicing prioritization and timeboxing.
Too little, Too Late?
After my session, I had some good discussions with conference attendees including GenAI consultants and trainers. They agreed with my points, but thought reserving GenAI use until after SME convergence might be too little, too late. To some extent I agree and need to stress this is only for certain classes of artifact where a lack of team engagement would undermine buy-in from the team.
Many classes of project management problems may be best started with GenAI attempts. Creating a Stakeholder Register or Communications Plan? The team cares less about these deliverables. Designing a Project Dashboard or Intranet Site? Have at it with as many AI tools as you like, then ask the team to review it afterwards. Yet, if it involves the team in how they work, think or behave, then please involve the team.
This means vision statements, charters, approaches, plans, schedules, estimates, risks and decision making. Dan Pink’s Drive book teaches us that individual motivation comes from having mastery, autonomy, and purpose over how we work. Using GenAI to define any of these team impacting artifacts robs teams of their mastery and autonomy making motivation afterwards almost impossible.
SME Core, AI Wrapper
There are many other uses for GenAI once we have our SME developed core content, and have improved it then rounded it out with GenAI ideas. In my presentation I used blue text or boxes for SME generated content, and red text or boxes for AI generated content. So, I recommend starting with a blue, SME created base and then rounding it out with red, AI generated supporting materials.
GenAI is valuable for creating introductory text and suggesting standard sections for artifacts. We can also use it to explain industry specific terms for less familiar stakeholders (think IT jargon to business speak, or chemical engineering terms to environmental groups).
I frequently use GenAI to create summaries, glossaries and FAQs. It can reformat and repurpose documents into audio summaries, infographics and videos for websites and video feeds. GenAI is also invaluable at regenerating and rebaselining artifacts after changes have been approved.
GenAI really is a productivity accelerator for nearly all knowledge workers. However, as project managers, we need to be careful to use it without undermining team empowerment.
The ADVANCE mnemonic gives us a process for when we are in team-zone activities. It brings the following benefits:
People:
- Stronger collective memory – forged through debate
- Authentic buy-in – co-created and agreed to
- Clear accountability – human named owner and reviewer
Product:
- Increased product relevance – local domain expertise incorporated
- Enhanced quality – organizational practicality vetted
- Greater delivery speed – less rework and retrofitting
Strategy:
- Better governance – local regulations and compliance
- Durable competitive edge – your organizations R&D insights
- Resilience to change – planning creates resilience
This post summarizes the key points covered. We also looked at Ethan Mollick’s 15×5 grid that summarizes 15 areas to leverage AI and 5 areas to avoid. It supports the general recommendations here, using it for low consequence content, pattern matching, sense making and format converting. It recommends against use for high stakes endeavors (estimates, budgets, contracts – a lot of project management work) or where struggle is the point to learn and build group consensus.
I wrapped up by showing my three-monitor view again. AI is not going away from my workflow, and I am increasingly using agents for background research and task automation. However, for now, that central screen for new content remains my work, supported by AI.
When my activities impact other people, I will continue to engage them first before presenting AI suggestions. This basic psychology step enhances engagement and empowerment. Meaning the plan we jointly arrive at might take a little longer to create but is more likely to be embraced and executed effectively. Yes, co-created artifacts are slower to produce than AI generated ones, but in many team settings, the consensus and alignment built is well worth the extra effort.
