The challenge
An international development organization collects operational data across multiple countries and programme areas. Field teams submit forms; leadership needs consistent views of activities, indicators and cross-cutting KPIs.
Without a shared platform, each geography and programme tends to grow its own extracts, spreadsheets and report logic. Definitions drift. Historic files and live submissions disagree. Quality issues surface late. Adding another country or indicator set risks another independent reporting stack.
Why it was complex
- Multiple programme domains with different forms, rules and KPIs
- Multiple geographies served from one estate, with controlled distribution
- Live field collection plus historic files that must converge
- Identity and keying when natural identifiers are incomplete across domains
- Shared organizational KPI and monitoring logic that must not be redefined per report
- Operational data quality with rule catalogues and operator feedback
- Separated environments and a path to production change
Our approach
We treated the work as an enterprise analytics platform, not a set of dashboards:
- Stabilize sources into a governed ingest path
- Medallion warehouse (Bronze → Silver → Gold) for conformed history and consumption marts
- Domain-oriented delivery with centralized platform standards (data mesh patterns where they fit)
- Reusable orchestration so new programmes follow the same pattern
- Shared semantic definitions for monitoring and organizational KPIs
- Audience-based distribution for country, operations, global and quality consumers
- Environment strategy, Git-integrated delivery and operational handover
The solution
Data foundation
Lakehouse landing for files and payloads, plus an analytical warehouse with Bronze, Silver, Gold and infrastructure schemas for keys, watermarks and staging.
Data engineering
Fabric pipelines and notebooks orchestrate ingest, transform and mart builds. Domain schedulers coordinate programme and KPI paths so the estate grows by extension, not by cloning.
Data quality
Rule catalogues drive validation into error outputs, a dedicated quality report surface and an overnight notification path for operators.
Semantic layer and analytics
Power BI semantic models and reports cover programme monitoring, matrices and organizational KPI themes. Shared models prevent indicator logic from being reinvented per workbook.
Governance and distribution
Row-level security patterns on key models and an organizational app with multiple audiences control what each group sees.
Deployment and operations
DEV / TEST / PROD separation, environment configuration, Fabric Git delivery and a structured handover pack for day-2 operations.
Architecture
Platform flow
- Field data collection and historic files
- Microsoft Fabric ingestion
- Bronze → Silver → Gold warehouse
- Orchestration schedules + data-quality feedback
- Semantic models
- Power BI reports
- Audience-based distribution
Platform at a glance
Engineering scale from the as-built estate (approximate where inventories support ranges):
- 3Environments (DEV / TEST / PROD)
- ~160Production pipelines
- ~150Production notebooks
- 30+Power BI reports
These figures describe platform depth. They are not a substitute for confirmed business impact metrics.
Key engineering decisions
Warehouse medallion as the analytical system of record
Problem
Mixed API payloads, files and evolving form schemas.
Decision
Bronze / Silver / Gold in a Fabric warehouse, with lakehouse landing.
Why
SQL-governed marts, surrogate keys and history fit programme monitoring.
Result
New domains extend the same layers instead of creating new ad-hoc datasets.
Shared semantic model for monitoring and KPIs
Problem
Indicator definitions duplicated per report become inconsistent.
Decision
Consolidated models feeding multiple monitoring and organizational KPI surfaces.
Why
One place to maintain mappings and grain.
Result
Cross-programme comparability without cloning business logic.
Surrogate-key and mapping infrastructure
Problem
Natural identifiers from field tools are incomplete across domains.
Decision
Infrastructure key registry and mapping dictionaries feeding Silver and Gold.
Why
Durable joins when source forms change.
Result
Analytics stay stable as collection tools evolve.
Explicit data-quality layer
Problem
Invalid or empty critical fields silently corrupt management numbers.
Decision
Rule catalogue, error outputs, quality reporting and operator notification.
Why
Quality is an operational process, not a one-off cleanse.
Result
Issues surface before they harden into KPIs.
Environment-separated Fabric estate
Problem
Changing production while developing new programmes is unsafe.
Decision
DEV / TEST / PROD with environment configuration and promotion helpers.
Why
Serious platforms need a change path.
Result
Credible production ownership and handover.
Outcomes
Before
Programme and country reporting logic distributed across independent processes. Historic and live data hard to reconcile. Quality issues discovered late. New geography or programme risked another silo.
After
Shared warehouse and semantic definitions. Controlled ingest into one medallion model. Rule-based data quality with operator-facing feedback. Reusable pipeline patterns and a multi-environment estate ready for handover.
Measured efficiency or adoption percentages are not published here until the organization confirms them for marketing use.
Technology
Full stack used on this engagement (hero tags above highlight the buyer-facing themes; detail sits here):
Platform
Microsoft Fabric, OneLake, data lakehouse, Warehouse, Data Factory, Org App, Delta Lake
Engineering
SQL, Python / PySpark, Fabric Pipelines, Notebooks, Dataflows, medallion (Bronze → Silver → Gold), domain-oriented / data mesh patterns, data quality rules
Analytics & governance
Power BI, semantic models, DAX, row-level security, Entra ID, Microsoft Purview
Delivery
Azure DevOps, Fabric Git / CI-CD, DEV / TEST / PROD environment configuration, operational monitoring and quality feedback
Our role
We designed the architecture, implemented data engineering and semantic modelling, built data-quality and distribution patterns, established environment strategy and prepared operational handover documentation. The engagement demonstrates end-to-end platform delivery, not dashboard-only work.