Our Purple Peeps attended the Data + AI World Tour 2023, organised by Databricks in Brisbane. We were absolutely thrilled to immerse ourselves in the world of data, analytics and AI alongside many local visionaries who share our passion for the data-driven future.
Below are some of the takeaways and highlights from our Purple Peeps that we would like to share.
Ken’s Takeaways
Databricks has been evolving rapidly over the past 4 years, and the current lakehouse architecture is a testament to its abilities. By simplifying the traditional complexities of data ingestion using Delta Lake, it can provide the scale of a data lake while giving the benefits of a data warehouse.
Many lessons were learned from companies presenting their meta-driven ELT frameworks, allowing for easy configuration and management of a large data lakehouse. If you want to try out these capabilities, Databricks labs have their own framework available DLT-meta.
Sallie’s Takeaways
Attending the seminar was enlightening, especially during the companies’ presentations, which showcased their successful implementation of Databricks. Witnessing how much progress they’ve achieved in just a few years was fascinating. Beyond the tech highlights, I also emphasised the importance of investing in people’s skills and refining processes for maintaining data quality. This experience motivated me to deepen my knowledge of Databricks, particularly focusing on Delta Live Tables. The discussions on Large Language Models (LLMs) and other emerging technologies added an extra layer of interest and excitement about the future of data and technology.
Attending the Databricks 2023 Data+AI World Tour was eye-opening, demonstrating how far the company has grown since its infancy in 2013. The event emphasised the importance of data democratisation for all employees and AI democratisation in all projects. Databricks discussed important product introductions such as their Unity Catalog, Databricks SQL and their major release: “Delta Lake 3.0” there were engaging customer success stories, emphasising the notion that now is the ideal moment to harness data ecosystems for increased productivity and improved outcomes.
Jaysee’s Takeaways
- What is the Unity Catalog?
-
- Unity Catalog is a unified governance and AI layer for data within the Lake House platform
- Unity Catalog features include unified visibility, a single permission model, AI-powered monitoring, and open data sharing
- Unity Catalog provides a comprehensive governance lens for data within the Lake House platform
- Unity Catalog has added an Apache Hive API that allows any engine that understands Hive to talk to it
-
- What is Databricks SQL?
-
- Databricks SQL is a new SQL analytics engine designed for the lake house architecture
- Databricks SQL is designed to handle a variety of data types and use cases, including text, images, files, and generative AI
- Databricks SQL is fully compatible with existing BI tools and data warehousing solutions to ensure a smooth transition
- Databricks SQL allows customers to query data using SQL seamlessly, making it easier for organisations to integrate machine learning and analytics into their workflow
- Databricks SQL offers a powerful, scalable platform with a serverless option and dynamic caching, allowing customers to query data without worrying about the underlying infrastructure or data size
- Databricks SQL is fully compatible with existing BI tools and data warehousing solutions to ensure a smooth transition
-
- What is LLMOPS?
-
- The Lakehouse AI bot uses a retrieval-augmented generation LLM pattern
- The system involves splitting documentation into logical chunks, embedding them with an LLM and putting them into a vector database
- Data collection preparation is a data problem that needs to be tied to the Unity catalog
- Hugging Face has a list of 240,000 curated AI models and automl for LLM training is critical
- Lake House monitoring is essential for monitoring LLM models
-
Overall, the Databricks 2023 Data+AI World Tour offered a comprehensive overview of the evolving landscape of data and AI. Databricks’ emphasis on democratisation necessitates data professionals (like myself) to integrate these principles into their BAU processes. This involves ensuring that data is accessible to all employees and for them to consider how AI can seamlessly integrate into projects as a support assistant. The customer success stories provided practical examples for implementation, lessons learned when utilising Databricks products, and the platform’s value on the organisation. Data professionals should stay vigilant to emerging updates, ensuring alignment with organisational objectives and addressing any gaps that may impede successful implementation. I’m excited to assist my clients on their next Databricks journey.
Igor’s Takeaways
A key takeaway for me from attending the Databricks conference is the remarkable journey the platform has undergone in less than a decade. Reflecting on the days of manual Spark cluster configuration, the evolution of the platform now truly open to virtually any data use case regardless of the Big Data 5 V’s (velocity, volume, value, variety and veracity) is evident. The recent acquisition of MosaicML and the public preview of Lakeview dashboards underscore Databricks’ commitment to expanding its comprehensive offerings further in the ever-growing data and AI market.
My key platform feature insight was a demo of Delta Live Tables, a declarative ETL framework to simplify streaming and batch data engineering workloads efficiently and cost-effectively.
