(By: Kamal Upadhyay Mohit Gupta)
Introduction:
The emergence of Generative AI (GenAI) has heralded a new era of possibilities for Technical Program Management. By reducing non-value-added tasks and enhancing the productivity of core activities, GenAI offers a paradigm shift in the way we approach the technical program value chain. This blog will explore the challenges currently faced by this sector and how GenAI can provide solutions. We will also examine the benefits, sustainability, scalability, and the overarching design of the proposed system.
Understanding the Technical Program Value Chain:
Before delving into the specifics of the Technical Program Value Chain, it is essential to first grasp what a value chain is. A value chain is a framework or system that outlines the activities a team performs to create and deliver value to its customers or users through a product or service. In the same vein, the Technical Program Value Chain is a set of activities aimed at creating value as part of a technical program. This chain comprises three primary segments: Planning, Execution, and Reporting.
Technical Program Value ChainAn Introduction to Generative AI (GenAI):
GenAI is an innovative AI technology that creates and provides on-demand information, content, and ideas by continuously learning and improving, mimicking human cognitive processes. Unlike traditional AI, which is primarily used for predictive analysis, GenAI takes it a step further, generating unique content based on its analysis.
Exploring the Role of GenAI in Technical Program Management:
To understand the potential of GenAI, let us consider an example program designed to build a seller support system for third-party sellers of an E-Commerce marketplace. The primary objective of this program is to reduce the number of seller contacts. Using this example, we will highlight the challenges faced in each segment of the program value chain and how GenAI can address them.
Identifying Failure Modes & Potential Risks in the Program Value Chain:
Below are some of the top failure modes across Value Chain
Planning
- Incomplete and poor-quality requirements without business value, acceptance criteria, detailed problem statement. This Impacts Intake prioritisation, quality, and output of execution.
- Missing high level design and dependency, leading to risk in planning and commitment.
- Subjective sizing estimates, missing nuanced capacity details. For Example — Available capacity is 200 PW; Backend Capacity 180 PW; Front End capacity is 20 PW. Work committed,180 PW, within available capacity but Front-End commit is risk, which was not considered.
Execution
- Problems occur when work breakdowns happen in terms of tasks instead of small iterations of shippable products, leading to a gap in story points recorded versus actual delivery
- Burn rate, velocity, and developer performance may not necessarily correlate directly with actual delivery, thereby affecting the commitment to deliver. For instance, discrepancies may exist between recorded story points and the actual delivery accomplished.
- Excessive production and testing bugs, coupled with prolonged testing cycles, can detrimentally impact product quality. This could be a result of unclear or incomplete requirements and the adoption of a mini waterfall approach, both of which potentially lead to increased defects and extended testing and delivery cycle times.
- The lack of clear release plans for smaller, iterative deliveries/releases further complicates project management, particularly for complex programs. This ambiguity can hinder efficient project execution, underscoring the need for enhanced clarity in release scheduling.
Reporting
- Risk identification and the recommendation of mitigation plans tend to be subjective, relying heavily on manual processes. This approach can pose a challenge in terms of objectivity and efficiency.
- Regular updates are crucial to maintain consistency in daily and weekly reporting to leaders. However, systematic, and consistent approach is a challenge due to manual dependency on TPMs.
- On-demand real-time reports and insights currently necessitate manual effort, which are tedious and time-consuming.
How GenAI Can Address These Challenges:
Planning :
GenAI-based automations can help to mitigate these challenges and potential risks. During the planning phase, GenAI can assist in scoring functional requirements, generating non-functional requirements, and estimating workloads
- Gen AI can help in functional requirement scoring of requirement based on the factors like problem statement, business value, solution of the problem, user persona and acceptance criteria, using contextual product data. Allowing to move in process further only if score is beyond the agreed threshold. For example — get the score using Gen AI and only allow further id score is greater than 8 on scale of 10.
Prompt to Gen AI: Score the below business/product requirement on scale of 1–10 on factors Problem Statement, Details, Business Impact, Acceptance Criteria “Feature to reduce the seller contacts for selling restricted materials, by providing a feature in chatbot to answer the non transactional query”.
Response : Here’s a score breakdown for the requirement:
Problem Statement: 7/10 — The problem is identified (reducing seller contacts), but could benefit from more context about why this is a problem (e.g., frequency of queries, seller frustration).
Details: 6/10 — The description is somewhat clear, but it lacks specifics about how the chatbot will operate, what types of non-transactional queries it will address, and integration points.
Business Impact: 8/10- Reducing seller contacts could lead to improved efficiency and cost savings, but the exact impact on sales or seller satisfaction isn’t quantified.
Acceptance Criteria: 5/10 — No acceptance criteria are provided. Criteria should specify what success looks like (e.g., reduction in contacts by a certain percentage, user satisfaction metrics).
Overall Average Score: 6.5/10 — To improve the scores, adding more detail, quantifying the impact, and defining clear acceptance criteria would be beneficial.
- Gen AI can help in nonfunctional requirement generation based on user and system details.
- Gen AI can use high level design and based on HLD dependency mapping can automatically be added and communicated.
- Gen AI can help in estimating the work based on PRD, HLD and Tech stack, at overall feature and at component level. This will help in correct sizing estimation and better planning.
- Gen AI can help in nuanced capacity planning based on the existing capability, available head account of the teams.
Execution :
In the execution phase, GenAI can break down features into smaller, manageable tasks, generate automated Test Cases (TCs), and formulate weekly release plans.
- Breaking down the Features into smaller — Stories, Tasks deliverable at sprint level
Prompt to Gen AI: Generate the user stories with acceptance criteria for below business/product requirement “Feature to reduce the seller contacts for selling restricted materials, by providing a feature in chatbot to answer the non transactional query”
Response : Here are some user stories with acceptance criteria for the chatbot feature aimed at reducing seller contacts for selling restricted materials:
### User Story 1: As a Seller, I want to ask the chatbot questions about restricted materials so that I can get instant answers without contacting support
**Acceptance Criteria:**
The chatbot can handle at least 50 predefined non-transactional queries related to restricted materials.
Responses to queries must be provided within 2 seconds.
The chatbot offers clarifying questions if the seller’s query is ambiguous.
A feedback option is available for users to rate the response quality.
### User Story 2: As a Seller, I want to receive clear information about the restrictions on certain materials so that I can ensure compliance.
**Acceptance Criteria:**
The chatbot provides detailed explanations of restrictions for at least 10 common restricted materials.
Information must include the reason for the restriction and any applicable regulations.
The chatbot should be able to provide links to official documentation or resources for further reading.
The response should be in clear and understandable language.
### User Story 3: As a Seller, I want the chatbot to remember my previous queries so that I don’t have to repeat myself.
**Acceptance Criteria:**
The chatbot stores previous queries for a session and can reference them if the user asks a related question.
The chatbot must notify the seller when retrieving previous interactions for context.
The session history should be cleared automatically after 30 minutes of inactivity.
- Gen AI can help in generating automated burn rate, velocity, dev performance and delivery and its contribution on program goals
- Gen AI can help in generating automated test cases generation of negative, Edge cases, Nonfunctional and Chaos scenarios
- Gen AI can help in generating automated weekly release plan, release notes and playbook generation
Reporting :
- Gen AI can efficiently generate event-based reports by identifying major risks. Take for instance, the system can promptly issue alerts surrounding potential delivery risks due to a high number of unresolved issues or extended bug resolution time. Furthermore, it is adept at providing potential recommendations to mitigate these risks.
- Gen AI is capable of creating on-demand reports, responding to specific requests for example the number of Open P0 bugs or the current sprint or quarterly program velocity or burn rate.
- Gen AI can be programmed to generate scheduled reports, providing daily and weekly updates without fail. It streamlines the reporting process, ensuring that data-driven insights are readily available for informed decision-making.
Gen AI based automated solution for E2E Technical Program Value Chain ExcellenceHigh-Level Design Details:
The proposed system includes four primary services: Core Services, Data Services, Contextual Data Services, and Reporting Services. These services work in harmony to facilitate seamless operation, enhance planning and execution, improve reporting, and provide real-time, on-demand reports. This system’s main sequence involves Jira users triggering changes, Core Services interpreting these changes, GenAI responding with information, and Core Services using this information to make decisions and log data.
Core Service :
Service which gets the details from Jira, sends the request (Prompt & Data ) to Contextual Gen AI.
- It uses the response/data from contextual Gen AI, based on rules and logic solves the problem.
- It modifies the Jira status, details etc and logs the response data
Data Service :
Service to work on logged data and keep data ready for
- Planning Commit recommendations
- Dev Utilisation, Velocity, Outcomes
- Early risk identifications
- On Demand Reports
- Scheduled Reports
Contextual Data Service :
Service which provides the context of the platform. For example if the platform is marketplace, data and context details would be of marketplace.
- Reporting Service — Reporting Service to provide reports on scheduled and on demand for
- Planning Commit
- Dev Utilisation, Velocity, Outcomes,
- Risks
- Weekly & Daily Execution reports.
High Level Design of Gen AI based solution for TPM Value chain transformationAssessing Business Impact, Scalability, and Sustainability:
Benefits
- Improved Delivery Cycle Time
- Improved Product Quality — Reduced Testing bugs & GAPs
- Improved Site stability due to reduction in incidents
- Improved Reporting & E2E visibility
- TPMs, EMs and Product Leads –NVA cut downs via this More strategic engagement
Scalability
- Highly Scalable to Program, Portfolio and Pillar due to design of solution
- Changes only in - Context engine — Provides the context [Program, Portfolio Or Pillar]
- Other components no change
- Core Services
- Data Processing Services
Sustainability
- Relevant problems across all Programs
- Product provide — E2E Solution across the Value chain
- Limited manual Intervention
- High Accuracy and possibility to improve accuracy by feeding more contextual data
Conclusion & Way-forward :
The rise of GenAI offers a significant opportunity to enhance the Technical Program Management value chain. It can effectively address various failure modes and risks in planning, execution, and reporting stages, streamlining processes, and improving productivity. The proposed system’s high-level design ensures seamless operation and offers numerous benefits. Moreover, the solution is highly scalable and sustainable, making the integration of GenAI in Technical Program Management an effective way to enhance the efficiency and effectiveness of the value chain. At Walmart, we have conducted successful POCs on requirement scoring and functional test cases generation, with extremely promising results that have begun aiding in the development of good requirements. As we continue to feed more contextual data to GenAI, we can anticipate even greater accuracy and an enriched value chain in Technical Program Management.
We will be extremely delighted to hear from you. If you wish to share your feedback and wish to collaborate, please do send an email to [email protected] /[email protected]
Technical Program Value Chain Transformation using GenAI E2E Automated solution was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.