Retail Demand Forecasting

Executive Overview

This project demonstrates a complete cloud data engineering and MLOps workflow for large-scale retail demand forecasting. Raw Walmart M5 files are stored in Amazon S3, processed through Databricks and Delta Lake using a Bronze-Silver-Gold architecture, transformed into model-ready features, evaluated under strict temporal and leakage controls, and materialized as production forecast outputs.

The project goes beyond model training. It includes data-quality validation, experiment tracking, model governance, production forecast generation, monitoring tables, inventory-oriented analytics, a four-page Power BI report, a documented FastAPI service, Docker containerization, and automated API tests.

Business Problem

Retailers must decide how much inventory to position across thousands of products and stores before future demand is known. Under-forecasting creates stockouts and lost sales; over-forecasting creates excess inventory and working-capital cost.

• Forecast demand consistently across product-store combinations.
• Reduce stockout and overstock risk.
• Support safety-stock and reorder-point decisions.
• Provide traceable, governed model outputs.
• Monitor quality failures before forecasts reach downstream consumers.

End-to-End Architecture

The platform separates storage, processing, modelling, serving, monitoring, and business intelligence into clear layers. This makes the workflow reproducible and easier to audit.

Dataset and Processing Scale

The project uses the Walmart M5 Forecasting dataset, which combines daily unit sales, calendar events, SNAP indicators, and historical sell prices across multiple stores and product categories.

Data Engineering Pipeline

1. Ingest raw M5 calendar, sales, and price files from Amazon S3.
2. Validate schemas, source coverage, row counts, and required columns.
3. Build clean Silver tables with standardized identifiers and dates.
4. Join demand, calendar, event, SNAP, and price information at scale.
5. Create Gold datasets for modelling, evaluation, inventory planning, and monitoring.
6. Persist critical outputs in Delta tables for reproducibility and downstream consumption.

Data Quality and Production Controls

Production eligibility is based on explicit controls rather than on model accuracy alone.

Leakage Audit and Horizon-Safe Design

A central engineering lesson was that the model with the lowest apparent error is not automatically the best production model. An earlier hybrid system achieved a lower WAPE, but the final audit identified leakage concerns in its feature construction and selection path.

The governed model therefore uses a direct 28-day horizon-safe baseline. For each target date, it only uses historical observations that would genuinely have been available at forecast origin—approximately t-55 through t-28. The final test period was not used for tuning or champion selection.

Model Comparison and Governance Decision

Model promotion required both predictive quality and governance eligibility. A challenger had to provide at least 0.50% relative improvement without introducing worse operational behavior.

Governed Production Model

Final Test Performance

WAPE is the total absolute forecast error divided by total actual demand. The score reflects the difficulty of sparse and intermittent retail demand across a very large hierarchy.

Production Forecast Run

The final governed run generated a complete 28-day forecast snapshot with operational quality checks and monitoring readiness.

Inventory and Operational Analytics

Forecast outputs are converted into decision-support views for inventory planning. The platform is designed to support expected demand, safety stock, reorder points, stockout-risk prioritization, excess-inventory identification, and category/store drill-downs.

MLOps and Governance

• MLflow experiment tracking and run comparison.
• Champion-model metadata and versioning.
• Validation/test separation and test-use audit.
• Forecast-run IDs and reproducible output snapshots.
• Monitoring tables for WAPE, MAE, RMSE, bias, coverage, nulls, negatives, duplicates, and alerts.
• Explicit production eligibility and promotion gates.

Power BI Forecast Intelligence

A four-page Power BI report communicates executive KPIs, forecast performance, category-level behavior, production volumes, and model-governance status. The exported report is stored in the GitHub repository.

FastAPI Serving Layer

A modular FastAPI service exposes governed model metadata, production forecast summaries, service health, and representative 28-day item-store forecasts. Swagger and ReDoc documentation are generated automatically.

Docker and Reproducible Execution

The API is packaged in a lightweight Python 3.10 Linux image. This eliminates machine-specific dependency differences and provides a consistent runtime for local demonstration or future deployment.

Automated Testing

Pytest and the FastAPI test client validate service availability, model metadata, forecast-summary values, 28-day series structure, non-negative predictions, data-source transparency, and correct 404 behavior.

Selected Project Evidence

Screenshots are loaded automatically when matching files are available in the website image folder.

End-to-end retail forecasting and MLOps lakehouse architecture.

Power BI executive overview with production forecast KPIs.

Model governance and leakage-audit dashboard.

Forecast monitoring, quality controls, and alert readiness.

FastAPI OpenAPI documentation and forecast endpoints.

Containerized API running with Docker Desktop.

Technology Stack

Current Limitations

• The item-store API endpoint uses an explicit local demo fixture rather than a live Databricks adapter.
• The API is containerized but not publicly hosted in the cloud.
• The Power BI report is shared as an exported PDF rather than a public interactive workspace.
• Intermittent retail demand remains difficult and limits achievable point-forecast accuracy.
• Automated CI/CD and scheduled retraining are future extensions.

Key Engineering Lessons

• Temporal validity matters more than an artificially strong benchmark.
• Model governance must combine accuracy, leakage safety, bias, reproducibility, and operational checks.
• Production forecasting requires reliable data contracts and monitoring—not only notebooks.
• Clear documentation of limitations increases technical credibility.
• Serving, containerization, tests, and BI turn a modelling exercise into an end-to-end engineering project.

Project Outcome

This is one of my strongest portfolio projects for Data Engineering, Cloud Data Engineering, Analytics Engineering, and MLOps roles. It demonstrates large-scale Spark processing, lakehouse architecture, temporal modelling discipline, model governance, API design, containerization, automated testing, and executive reporting in one coherent system.

The final result is not only a forecast model, but a traceable production-oriented data product with documented engineering trade-offs and clear boundaries between implemented functionality and future extensions.