We modernised the fund’s legacy analytics platform with a secure, scalable cloud-based data architecture built on Databricks, Data Factory, dbt, and Microsoft Fabric. The new platform improves reliability, transparency, and time-to-insight while enabling the small data team to operate with greater agility and consistency.
The Problem
A superannuation provider was relying on an on-premise SQL Server analytics environment that had become increasingly difficult to operate as data volumes and reporting expectations grew. Core challenges included long processing times, limited visibility of business logic, and poor repairability when issues occurred—all of which slowed down the delivery of insights to internal teams.
With a small development and maintenance team, the provider needed to improve agility, standardise processes, and modernise their analytics platform to support timely reporting, member insights, operational oversight, and regulatory obligations typical within the superannuation sector.
The Solution
We delivered a modern cloud-based analytics architecture built on Azure Databricks, Azure Data Factory, dbt Core, and Microsoft Fabric. Azure Databricks became the central compute and data engineering platform, while Azure Data Factory and Databricks Autoloader modernised data sourcing, enabling real-time, near real-time, and batch ingestion.
All ingestion and transformation workloads were standardised through metadata-driven, pattern-based frameworks, replacing bespoke stored procedures and custom processes. A dbt-powered medallion lakehouse was implemented – including a full-history bronze layer for time travel, lineage, and simplified downstream SCD modelling – supported by Unity Catalog integration for consistent governance.
Microsoft Fabric provided the semantic and reporting layer, enabling both interactive and paginated reporting. The entire solution was secured through enforced private connectivity across Databricks, Data Factory, and Fabric, aligned with best-practice cloud security patterns.
Business Benefit
The modernised platform significantly reduced processing delays and improved the reliability of data delivery across the fund. By shifting to metadata-driven, pattern-based ingestion and modelling, the organisation lowered its maintenance burden and enabled faster development of new data products, even with a small team.
Data trust and transparency improved through clear lineage, full-history storage, and self-documenting dbt models, while Autoloader and dbt added strong resilience and repairability.
The solution also enhanced scalability for growing data volumes and reporting demands, strengthened auditability, and aligned the fund more closely with contemporary governance and security expectations.
Download our full case study here.