Data Engineering at Data and AI Summit 2022
What role do data pipelines and workflows play in modern data engineering? What is the easiest way to deploy them? How about streaming data and ensuring data quality? And can you throw in some ML in under 15 minutes?
Data Engineering in 2022
Organizations realize the value data plays as a strategic asset for growing revenues, improving the customer experience, operating efficiently or improving a product or service. Data is really the driver of all these initiatives.
Nowadays, data is often streamed and ingested from hundreds of different data sources, sometimes acquired from a data exchange, cleaned in various ways with different orchestrated steps, versioned, and shared for analytics and AI. And increasingly, data is being monetized.
Data teams rely on getting the right data at the right time for analytics, data science and machine learning, but often are faced with challenges meeting the needs of their initiatives for data engineering.
The role of the Data Engineer
The challenging goal of the data engineer is to build and run the machinery that creates this high-fidelity data product all the way from ingestion to monetization.
At the core of all this are data pipelines with Delta Live Tables and Databricks Workflows.
The Data Engineering Session at Data and AI Summit 2022
I felt really honored to present the data eng session at Data and AI Summit 2022 with a colleague of mine Paul Lappas. The session was recorded (video below), my Twitter Stream, Delta Live Tables, Hugging Face demo starts at 12:15.
GitHub Repo
If you are adventurous and you like to replicate the demo that I was showing you can grab it from GitHub.
