Decoding Causal Incrementality in E-Commerce: Leveraging Bayesian Structural Time Series Model with a Real-World Example
In the realm of data-driven decision-making, understanding causal relationships is crucial. Traditional statistical methods often fall short when it comes to capturing the nuanced effects of interventions or changes in dynamic environments. This is where Bayesian Structural Time Series (BSTS) emerges as a powerful tool for measuring causal impact.
Why Causal Impact Measurement?
Imagine you run an e-commerce platform and are planning a major redesign of your website. Before rolling out changes, it’s essential to gauge how these alterations will impact user behavior, sales, and overall performance. Simple before-and-after comparisons might not accurately isolate the impact of the redesign due to various external factors. Causal impact analysis helps in attributing changes directly to interventions, offering a clearer understanding of their effects.
Why BSTS?
Bayesian Structural Time Series is a statistical approach combining Bayesian Inference and Time Series Forecasting. It aims to estimate the underlying structure of the time series using components like trend, seasonality and other external factors. BSTS uses Bayesian inference, which allows for incorporating prior knowledge and generating probability distributions around the estimated causal effect, providing a more nuanced understanding of the uncertainty involved.
Limitations of Traditional Causal Impact Approaches in Dynamic Settings
In many scenarios, traditional approaches to causal impact analysis face significant challenges. Methods such as Difference-in-Differences (DiD) or simple A/B testing often assume static conditions and may not account for the evolving nature of data over time. Most DiD analyses only consider two time points: before and after the intervention. In practice, the way an effect evolves over time, especially its onset and decay structure, is a key question. Dynamic environments, characterized by seasonality, trends, and external factors, require more sophisticated models to accurately isolate the effects of interventions.
Causal Impact using Bayesian Structural Time Series & Synthetic Controls
We estimate the causal impact by leveraging Bayesian statistics and time series modeling to estimate the causal effect of an intervention or event. Unlike traditional methods, BSTS considers the dynamic nature of time series data and incorporates uncertainty into its predictions, making it robust for real-world applications. BSTS utilizes a structural time series model that decomposes observed data into various components, allowing for a more nuanced understanding of causal impacts. The model in simple terms can be represented as follows:
where:
- y_t is the outcome metric at time t
- μ_t is the local level term. The local level term defines the unobserved trend in the data.
- γ_t due to seasonality in S seasons. We can think of it as a hyperparameter to the model.
- x_t are the observed controls, and we assume they contribute to through β^T x_t, a regression component.
- The components of admit a spike-and-slab prior. The spike-and-slab model produces inclusion probabilities for each of the possible controls.
- ∈_t is the noise.
The posterior gives us a predictive distribution for the counterfactual. Through that, we get a distribution of pointwise impact (observed minus counterfactual).
Application: Measuring Causal Impact Using BSTS and Synthetic Controls
Let’s delve into a practical example to illustrate this in action. Consider a merchant is deliberating on how to create / change the ice cream browsing shelves on their e-commerce website. They hypothesize that designing popsicles & bars, ice cream sandwiches, tubs and cones as the main shelves will lead to optimal customer browsing experience measured by metrics like browse views and sales. Once the change is launched, using BSTS, they can measure the causal impact of this taxonomy change on browse views and sales performance by following the below steps:
Step 1: Collecting Observational Data Over Time
For the metric of interest, you first collect the observational time series data of the ice cream category on say, a weekly basis. For example, collect weekly sales of ice cream n weeks before and n weeks after the launch of the new taxonomy. n, typically, would be anywhere between 8–10 weeks pre- and 5–10 weeks post-launch depending on how far in time you want to go.
Step 2: Selecting Synthetic Controls
BSTS allows the merchant to create synthetic controls, mimicking what sales might have looked like in the absence of the taxonomy change. This process involves:
- Identifying Similar Units: Finding comparable categories that were not affected by the taxonomy change.
- Weighting Variables: Adjusting weights for these control units to minimize differences with the treated unit (the category that underwent the taxonomy change).
- Evaluating Closeness: Assessing the similarity between the treated unit and synthetic controls based on relevant metrics (e.g., historical sales trends)
For example, the potential synthetic controls for the ice cream category can be other similar categories like frozen pizzas, frozen yogurts, frozen vegetables, sunscreens (as both are sold the most in summer), etc. We don’t have to worry about the accuracy of these potential controls because of the spike and slab prior technique used in BSTS to handle variable selection. By evaluating multiple potential control time series, the model can select a subset that best explains the variation in the treated series without being affected by the intervention. This selection process is crucial for accurately estimating the counterfactual scenario, which is the expected outcome had the intervention not occurred.
Next, collect pre- and post-launch time series data on these controls.
Step 3: Apply BSTS to Predict the Counterfactual & Measure Impact
Now, arrange your data in the required format and fit the BSTS model. You can also provide a custom model with additional parameters.
By comparing actual sales data post-change with the synthetic control, BSTS provides a clear estimation of the causal impact of the taxonomy restructuring. This method accounts for seasonality, trends, and other variables that could influence sales independently of the intervention. As one of the major differentiators compared to other causal methods, it accounts for uncertainty in the impact measurement. As shown in the figure below, as we move far along in time, we see that the confidence interval (shaded blue region) expands which means we can only be so sure that the impact shown is due to the taxonomy change. Another differentiator is that it lets us know exactly when the impact is visible in the post-period and how it evolves with time.
Demo Example: Weeks vs. Total Sales — Ice Cream category — Predicted vs. Actual ; green arrow: Incremental salesAdvantages of BSTS in Decision Making
The application of BSTS offers several advantages in decision-making processes:
- Precision: BSTS provides a precise estimation of causal effects, helping businesses make informed decisions based on reliable data.
- Flexibility: It can handle various types of time series data, accommodating complex scenarios typical in business environments.
- Uncertainty Quantification: By incorporating Bayesian inference, BSTS not only estimates causal impact but also quantifies uncertainty around these estimates, enhancing decision confidence.
- Real-time Analysis: BSTS can be adapted for real-time analysis, allowing businesses to monitor and adjust strategies promptly based on evolving data.
In conclusion, Bayesian Structural Time Series (BSTS) represents a significant advancement in causal impact measurement, particularly in dynamic settings. By accurately assessing the effects of interventions or changes, BSTS empowers businesses to optimize strategies and drive meaningful outcomes.
References:
Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015). Inferring causal impact using Bayesian structural time-series models.
Decoding Causal Incrementality in E-Commerce: Leveraging Bayesian Structural Time Series Model… was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.