Introduction 

Large Language Models (LLMs) and Generative AI (Gen AI) have generated great interest from not-for-profits looking to accomplish more, with finite resources.  As organizations with limited staff consider using tools like these, they are asking a simple question: how does one evaluate new technologies, and determine whether and how they will fit their needs?   

Luckily, teams tasked with accomplishing this objective have been formed many times in the past and have used a well-known approach. The approach described in this article is intended to help you, your colleagues, and others in your organization, should you be asked to vet and begin using a novel technology in your work.  

Components of Improvement Teams to Vet Technology 

Cross-Functional Teams for Improvement 

The term “cross-functional team” generally refers to a team composed of members from different areas of the organization whose objective is to solve a specific problem that requires expertise not contained within a single department.  This usually involves intractable problems that the organization has struggled to solve.  For our purposes, we use the term "improvement team” as a synonym for cross-functional teams. 

Many improvement teams have solved significant problems by meeting once a week for about 60 to 90 minutes, along with preparatory work before and after the meeting.  If a systematic problem-solving process is used, an individual on the improvement team may therefore commit to about two hours of extra time each week on average. 

Improvisation and Adaptation 

The prior section does not mean to suggest that an existing team template should be used as-is.  Each organization is unique, and forming an improvement team is an excellent starting point; however, these templates are general in nature, and it may be necessary to modify them to an organization's size and culture. 

The American Society for Quality is a good resource: "What is a Team?" Types of Teams & Processes | ASQ.   

Further details will be provided in Part Two of this series. 

The Steering Committee 

The question of whether to adopt LLM/Gen AI technologies is significant for an organization.  There can be multiple reasons, such as the technology's maturity, which functionality provides the desired benefits, how well it delivers the desired functionality or performance, and exactly where in the organization’s process flow the technology delivers the greatest benefits. 

In large organizations, a steering committee would be formed to pursue the question.  For small organizations, the senior leadership team is often the de facto steering committee. 

The Senior Leader Champion 

The champion represents the needs of senior leaders to members of the improvement team.  The champion may be selected by senior leadership, or the champion may be the person who determines that improvement is needed and volunteers to form the improvement team. 

The champion must be both a critic and a proponent of the technology. If the champion is too much of a proponent, the improvement team's ideas and suggestions will not undergo the scrutiny required to ensure a good solution.  This can happen when the champion has already decided that LLM/Gen AI technologies are of value and does not want to be convinced otherwise.  The converse is also true: the champion is so critical of LLM/Gen AI technologies that they are convinced no solution is workable. 

The Charter 

The charter is a document that defines the deliverables, responsibilities, authority, boundaries, resource availability, and composition of the improvement team.  Although not required, it is good practice to have one, and moreover, have it approved by senior leadership.   

While a charter may seem like a bureaucratic step, it is crucial for the establishment of the following. 

Clarity: The charter helps ensure clear expectations, including the team’s goals and responsibilities.  This clarity applies to both the team and senior leadership. 

Boundaries: Boundaries, or decision-making rights, are essential for setting expectations about the team's authority to make independent decisions to reach team goals. 

Resources: This section of the charter outlines the resources available to the team to achieve its goals.  This helps avoid questions about the team's authority to spend resources, such as their training budget, access to subject-matter experts, or budget for prototyping potential solutions. 

Consensus: The charter's primary goal is to reach consensus on the team's structure, objectives, responsibilities, and authority.  Depending on the situation, it may be helpful to use a “catch-ball” approach, in which the team reviews a draft charter to get the group’s buy-in before team formation and charter approval. 

Team Members 

Many teams comprise front-line workers in addition to managers.  The benefit of having front-line workers on the team is that they will implement and use new LLM and Gen AI technologies and therefore understand their future impact on daily operations. 

In addition to knowledge in how the organization conducts daily work, a mix of skill levels is essential for team dynamics.  Like the discussion of the senior-leader champion's stance, team members may be biased toward or against LLMs and Gen AI.  A mix of skill levels helps ensure the team will evaluate new technologies with a balanced, objective viewpoint. 

What Consensus Really Means 

Consensus does not mean 100% agreement on all decisions or recommendations.  What it really means is that decisions or recommendations are something all team members can live with. 

Since adopting new technologies likely means changes to daily work, one or more areas of the organization must change how they operate.  Any change can be difficult and uncomfortable and will likely involve some apprehension; therefore, holding out for 100% agreement on all details is unlikely to lead to a solution.   

The Importance of a Subject Matter Expert (SME) 

As Mark Twain said, “experience is the name we give to our mistakes.”  The ideal SME is experienced in using technology. If the organization investigating a new LLM/Gen AI technology lacks deep technical expertise in this area, it may need to seek external support.   

The primary role of an SME is to burst the bubble of the notion that AI is a magic solution.  While doing so, the SME should also educate the team on risks to watch for and how to mitigate them. For example, Gen AI has a history of hallucinating nonsensical answers to questions. How can an organization minimize the risk?  SMEs should be a resource for answering these kinds of questions. 

Implicit Recognition – the Final Presentation 

The team's final presentation of recommendations can serve as an implicit recognition of all team members' efforts. Ideally, it is a presentation to the senior leadership team.  This provides an opportunity for senior leaders to understand and appreciate every team member’s work.  Typically, this is done via a PowerPoint presentation, during which all team members can discuss their contributions. 

The Value of Necessity 

Investigating the use of LLMs and Gen AI necessitates team involvement. 

While the first use of such a team will feel uncomfortable and clunky, over time, as more teams are used to addressing complex problems, the practice of using teams will become second nature to the organization.  Skills in team-based problem-solving will grow across the organization’s staff. 

Summary 

Improvement teams have been developed and refined for decades, providing an excellent starting point for creating teams to investigate LLM/Gen AI technologies.  Moreover, once team-based problem-solving becomes a core competency for an organization, it can develop solutions to persistent problems that the organization has difficulty addressing. 

This is Part One of a two-part series. Part Two will provide realistic scenarios and will be published in the next few weeks.