A few of our Purple Peeps attended the Data + AI World Tour organised by Databricks in Melbourne on the 31st of August, 2023. What an honour to be surrounded by so many local and like-minded peers who support the transformative future of data, analytics and AI.
Below are some of the takeaways and highlights from our Purple Peeps that we would like to share.
It was an amazing turn out for the Melbourne Data & AI World Tour by Databricks. Databrick’s displayed such a fantastic event that covered both key product launches and customer case studies. Their message of how democratised Generative AI is, that NOW is the time to capitalise on their data environments to improve efficiency and drive better outcomes for them and their customers.
The Databricks ecosystem is a robust and feature-rich platform that they have really focused on ensuring customers can structure their data to maximise the outcomes. The Lakehouse platform unifies customer’s data warehousing and AI use cases onto a single platform. By utilising their Unity Catalog (governance) and Delta Lake (storage layer), customers can securely manage and view data whilst getting the best insights from their data.
Top 2 highlights:
- I loved how successful the use cases were from the presentations. Their customers have seen such positive outcomes from working with Databricks and their partners that they are advocating and sharing their stories for everyone to learn from.
- The depth of the Databricks platform is really strong, and the engagement with partners and customers will see more amazing use cases in the market.
It’s clear that Databricks have positioned themselves as a market leader covering Data, AI and Governance to manage their environments securely. As one of our key Vendors, we are looking forward to partnering with them and utilising their products to deliver fantastic outcomes to our customers.
At the Melbourne Databricks World Tour Data + AI session at the Glasshouse, I was interested in seeing how Databricks has been used at some customer organisations and seeing what new features are coming soon.
Some of the features discussed by the partner organisations were in Private preview only but showed Databricks is making a concerted effort to build their platform into one where you can (if you choose) do everything required of a Data Analytics and ML Platform from Ingestion, Transformation, Processing and Serving.
The features I liked the most were: Delta Live Tables:
Delta Live Tables is a declarative system that streamlines building dependable, sustainable, and verifiable data processing pipelines for batch and streaming data, handling task coordination, cluster control, monitoring, data integrity, and error management.
Unity Catalog is Databricks’s unified governance solution for data and AI assets on the Lakehouse, which seems to be continuing to evolve and improve its capability from its initial introduction. New features of UC include system tables which allow you to integrate your account’s operational data (including audit, billing and lineage data).
It’s no surprise that it’s the human stories that captured my attention on the day. Databricks wove customer outcomes – and customer presenters- seamlessly into presentations throughout their Data & AI World Tour. These tales of organisational enablement and the outcomes that had been unlocked all came with the same, sensible caveat – making sure your organisation is getting the data basics right. Clean data – and, as Nathan Steiner eloquently put it – avoid just slapping an LLM assistant onto your problems.
Highlights of day
- Continued emphasis on getting the fundamentals right to solve your organisation’s business problems – “Don’t just slap an LLM assistant onto your problems”
- Democratisation of technology enabling innovation – and a diverse, equitable and inclusive technology landscape
Databricks is developing with two kinds of data engineers in mind, one that comes from a software background and prefers IDEs and Python, and one that comes from an analyst background and prefers SQL and GUI. This allows for more people to use what they’re familiar with to achieve the engineering goal.
Databricks has seen 150% year-on-year growth in streaming jobs. Delta live tables hide away the complexity of Spark and allow engineers to query live streams with the same logic as regular delta tables.
Unity Catalog has increased in scope. It now catalogues lineage for everything within Databricks automatically. Data is now searchable; delta sharing can be in place; and definitions are applied to the data itself and is not solely available from within Unity Catalog. Access controls on data can be added using SQL or within the GUI, adding flexibility for admins.
Added Systems Tables for queryable billing and metadata, making administration of the platform simpler.