An Autonomous Agent for Agile Planning

Malware News
1

Introduction

I am a Software Engineer. A big part of my job, though, is not coding, but planning. I like coding, and I can code well (hope that is true) but planning is not something I am crazy about.

I can understand that planning is essential for a big tech org like Walmart, given our team size and geographical spread. and this is something inevitable exercise.

But hey… LLMs these days can do pretty much anything (playful exaggeration), So can they help me with Agile Planning?
The short answer is of course… Yes.

Now, for a slightly longer and more informative answer, Here goes the rest of the article.
To demonstrate how we (me and my team) created an autonomous agent for Agile planning, let me break this conversation into four parts.

We will start with a problem statement. Then we will discuss the thought process behind building this agent. We will talk briefly about LLMs and retrieval augmented generation (RAG) and how these concepts can used as reasoning Agents to accomplish a task. Towards the end, we will discuss how we have implemented this agent .
Building this agent has been a great learning for me. And I am excited to share it with you, so let’s get started.

What are we solving?

Before I explain the problem, let me briefly explain the Agile process — for those happy folks who never had to deal with it.

It starts with quarterly Programme Increment (PI) planning. The business and the product managers sit together to chalk out the product goals for the next quarter. The engineering managers use these goals and distil them into a list of software features. A PI is typically divided into 4 or 6 sprints. Once the features are finalized, the next task is to plan these sprints.

Sprint planning is quite a laborious process. Here is a brief outline:

First, we divide each goal into a list of tasks, spanning from design to production.

For each task, also called stories:

1. Understanding the requirements for each story.

2. Add detailed description for each story.

3. Add crisp and demo-able items as acceptance criteria.

4. Identify dependency flags, and dependencies on upstream and downstream teams.

5. Estimate the level of effort (LoE) and the story points.

6. Finally, we create the timeline for each goal.

All this is essential because, once we know the level of effort (LoE), we can negotiate with the product management, our commitments per sprint, and get down to work.

But here lies the problem.

We, the coders, don’t write well in non-programming languages, English being one. So, a good sprint plan, often, remains a lucid dream. We find it difficult to write crisp task descriptions, and even more so, testable acceptance criteria.

Is there a way we can code our way out of this problem?

There is…

How It Started

To solve this problem, we decided to observe ‘the human’ way of solving this first. So, to our team members’ annoyance, we observed and interviewed them while they planned for their sprints. Apart from a common feedback — ‘Go away and let me work in peace’, — we got some helpful insights as well. We converted it into the following flow chart.

Flow Chart — Agile strory creation process

The flowchart looks complicated, but simply put, here is what it says.

  • Fetch Goal and Feature Details.
    Using: existing documents, meetings, notes, and emails.
  • Break the Feature into a Series of Tasks.
    Using: previous examples, experience (memory), and the company’s technical guidelines.
  • For each task, create its description and a list of acceptance criteria.
    Using: company guidelines and experience.
  • Estimate the Level of Effort (LoE) for each story.
    Using: company guidelines and experience.
  • Review the plan with the team and iterate.
    Using: the team’s feedback
  • Add this to the sprint board.

Note that at each step, we use a specific set of documents and our experience. Our experiences come from the previous PI planning exercises. So essentially all the information we needed to create sprint, is readily available in written format in our internal documents. Isn’t that awesome!?

Implementation

Once we jotted this process down on paper, we started implementing it using an LLM — GPT instace in our case.

Here is a simpler diagram of our very own Agile Hemingway Version 0.

Agile Hemingway — Design Flow

Every blue box here is a call to LLM. Before we dig into each of them one by one, let’s take a slight detour to understand the tech stack we are using.

Tech Stacks

We are using the following tools.

Data Sources

We need four data sources for the various steps we chalked out earlier.

1. The Goal: This is the PI Goal which is generally a part of the product Canvas in the Agile framework.

2. The Product Requirement Document (PRD): This is a document that the product writes in partnership with the Engineering team, describing the product requirements in detail.

3. Corporate Guidelines: Every company has its implementation of Agile Processes. These guidelines specify the overall structure of the process and define concepts like how a feature, story, acceptance criteria, etc, should look like.

4. The stories we created in previous sprints: These examples act as substitutes for our experience.

We curated all this information and stored it in the PgVector store as text embeddings.

Our LLM Models

Now let’s briefly discuss the three blue boxes in our flowchart.

The sprint planner: This is a Prompt that plans the timeline. It divides a given goal into a series of tasks. It uses the guidelines data as well as borrows inspiration from our standard processes.

The story groomer: This LLM call generates a story headline, description, and acceptance criteria for each task generated by The Sprint Planner. The prompt written for this LLM takes the following inputs-

  1. task details created by the story planner,
  2. all supportive information present in vectorDB for given task and
  3. guideline information that needs to be followed during story creation.

The story sizer: estimates the level of effort (LoE) based on acceptance criteria generated by the story groomer and information available for similar stories created in the past which is stored in PgVector. The prompt created for this LLM takes two kinds of information –

  1. story details like description and acceptance criteria created in the previous step by the story groomer.
  2. and story details created in the past which are semantically similar.

Time to test out our Agent.

For illustration purposes, let’s use this openly available PRD from Google called the “Multi-spectral Imaging Device Project”.

PRD: https://www.hajim.rochester.edu/optics/undergraduate/senior-design/pdf/fall15/color_prd_final.pdf

We ran the Agent, and it created about 12 stories to satisfy the PRD. Here are a few sample stories.

Agile Planning Agent — Output

Future

The Agent works. It writes stories in minutes that we spend days to write. We have coded our way out of our problem :)

However, this is the version 0. We still have some way to go, to make it an Agile software engineer’s best friend.

In the next version, here are a few enhancements we plan to implement:

Validation

  • Curation of good and bad examples :
    Build bucket of good and bad stories which will be used by model to learn and improve result.
  • Validation :
    Build framework to validate model outcome on various parameters like story relevancy, LoE etc.
  • Human Feedback :
    Develop a way to intake human feedback on model outcome as well as reassessment of story based user comment and ratings on various story details parameters like story name, description, acceptance criteria or level of effort.
  • Automatic acceptance tests :
    Build mechanism to test whether story’s acceptance criteria are relevant and demo able

About Me

I am a senior software engineer at Walmart, having more than 7+ years of industry experiences. I am an AI enthusiast. I have keen interest in system design, competitive programming and exploring about latest tech stacks.

Thank you for reading this article, hope you find this useful. Happy Learning!!

Special Thanks

A special thanks to my mentor Rahul Nayak for who guided throughout this journey and thanks to my teammates — Kishan Jha, Chandra Mouli and Ritika Rana who put efforts round the clock after office hours as well. Let’s continue learning together.

An Autonomous Agent for Agile Planning was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Article Link: An Autonomous Agent for Agile Planning | by vishal singh | Walmart Global Tech Blog | Dec, 2023 | Medium