Data Processing

Official references for this objective

• Databricks — Deep Dive Into Streaming on Databricks | Databricks

  Zerobus Ingest is a Kafka-free ingestion capability that lets applications and devices write event data directly into Delta Lake, no message broker, no compute overhead

• Databricks — Intelligent Document Processing with Lakeflow: Transforming Unstructured Data for Agents and Analytics | Databricks

  how to build intelligent document processing pipelines with Lakeflow and AI Functions purpose-built for document understanding

• Databricks — Apache Iceberg™ Interoperability: First-Class Open Support in Databricks OpenSharing | Databricks

  Shift from closed sharing models to a transparent protocol that supports multiple clouds, formats, and platforms simultaneously

• Databricks — Building a Production: Scale Dimensional Data Mart With Lakeflow Spark Declarative Pipelines and AUTO CDC | Databricks

  including patterns for CDC, schema evolution, and hybrid batch/streaming workloads

• Databricks — Unlocking Video Data at Scale: VLM Batch Inference with Ray on Databricks | Databricks

  code integrating Ray, VLLM, and Unity CatalogPrompt engineering techniques for video inputsCost and performance considerations for VLM workloads

• Databricks — Beyond Medallion: Architecting Disney’s DATOS for Complex, Real-Time Data Streams With Databricks | Databricks

  DATOS reduces operational overhead, decouples producers and consumers, accelerates onboarding, and delivers a context-aware source of truth that powers real-time analytics

• Databricks — From Factory Floor to AI-Ready: Building Industrial Data Pipelines for Databricks | Databricks

  recipes.Learn how to unlock the value of your industrial data with Databricks.

• Databricks — Lakeflow Connect: Cutting Enterprise Ingestion Costs by up to 90% on Databricks | Databricks

  ingestion tools, and governance best practices with Unity Catalog lineage

• Databricks — Migrate to Lakeflow Jobs: Modernizing Data Orchestration on Databricks | Databricks

  What real customer migrations look like in practice, including key challenges, tradeoffs, and outcomes