The Problem
Our Oil & Gas client implemented Snowflake as their preferred data warehousing platform in conjunction with Matillion as their ETL toolset in an AWS environment.
They had a number of concerns relating to performance, scalability and cost. Generally, they accepted the way things worked as just “the way things work”.
There were ongoing challenges updating ETL logic, and adjusting data when business rules changed, which sometimes took months.
Each data source they ingested took around 2 weeks to create per source – that is, an individual table/view/file.
The platform was not effectively monitored, leading to high running costs and unacceptable performance.
The Solution
Exposé was brought in to challenge the end-to-end architecture, tools and practices, and then to define a new Data Service paradigm that aligned with the technology capabilities and business demands.
At no stage did we recommend changing their platform, just use what they have more effectively.
- We focused on improving the velocity of service delivery to the business with reduced support and operational costs whilst maintaining quality and security.
- We adjusted their data structures to align them more suitably with the business capabilities and data governance practices.
- We worked with Snowflake to tune the platform and train their support team to proactively monitor the platform.
Business Benefit
The benefits to the client were significant in terms of cost, performance and time:
- 2-week delivery cycles reduced to just hours
- Snowflake operational costs dropped to around 10% of their original operational costs
- Additional agencies were enabled to deliver side by side without contention
- Staff capabilities were improved across the board
- The Snowflake-based solution is now capable of meeting the scale and shape of all of the client’s data assets in a cost-effective and repeatable manner
- Security and visibility of data assets is controlled and easy to consume for business users
Overall, the business now has a clearly defined Data Service and an understanding of how much effort it takes to deliver timely business outcomes.
The client is now ready to take advantage of Snowflake’s capabilities like time-travel and sandboxing in a managed method.
Download our full case study here.