Databricks — 5 min read

How do data scientists combine Kedro and Databricks?

In recent research, we learned that Databricks is the dominant machine-learning platform used by Kedro users. We investigated aspects of our users' data science workflows to prioritise seamless Kedro usage, and we are collaborating with Databricks on our findings.

14 Apr 2023 (last updated 20 Nov 2023)
small square mint knot

In recent research, we found that Databricks is the dominant machine-learning platform used by Kedro users.

The purpose of the research was to identify any barriers to using Kedro with Databricks; we are collaborating with the Databricks team to create a prioritized list of opportunities to facilitate integration. For example, Kedro is best used with an IDE, but IDE support on Databricks is still evolving, so we are keen to understand the pain points that Kedro users face when combining it with Databricks.

Our research took qualitative data from 16 interviews, and quantitative data from a poll (140 participants) and a survey (46 participants) across the McKinsey and open-source Kedro user bases. We analysed two user journeys.

How to ensure a Kedro pipeline is available in a Databricks workspace

The first user journey we considered is how a user ensures the latest version of their pipeline codebase is available within the Databricks workspace. The most common workflow is to use Git, but almost a third of the users in our research set said there were a lot of steps to follow. The alternative workflow, which is to use dbx sync to Databricks repos, was used by less than 10% of the users we researched, indicating that awareness of this option is low.

databricks-slide-1

How to ensure the latest version of a Kedro pipeline runs on Databricks

How to run Kedro pipelines using a Databricks cluster

The second user journey is how users run Kedro pipelines using a Databricks cluster. The most popular method, used by over 80% of participants in our research, is to use a Databricks notebook, which serves as an entry point to run Kedro pipelines. We discovered that many users were unaware of the IPython extension that significantly reduces the amount of code required to run Kedro pipelines in Databricks notebooks.

We also found that some users run their Kedro pipelines by packaging them and running the resulting Python package on Databricks. However, Kedro did not support the packaging of configurations until version 18.5, which has caused problems. The final option some users select is to use Databricks Connect, but this is not recommended since it is soon to be sunsetted by Databricks.

databricks-slide-2

How to run a Kedro pipeline on a Databricks cluster.

Watch the discussion about Kedro and Databricks from a recent Kedro update meeting

The output of our research

To make it easier to pair Kedro and Databricks, we are updating Kedro’s documentation to cover the latest Databricks features and tools, particularly the development and deployment workflows for Kedro on Databricks with DBx. The goal is to help Kedro users take advantage of the benefits of working locally in an IDE and still deploy to Databricks with ease.

You can expect this new documentation to be released in the next one to two weeks.

We will also be creating a Kedro Databricks plugin or starter project template to automate the recommended steps in the documentation.

Coming soon…

We have a managed Delta table dataset available in our Kedro datasets repo, which will be available for public consumption soon. We are also planning to support managed MLflow on Databricks.

We have set up a milestone on GitHub so you can check in on our progress and contribute if you want to. To suggest features to us, report bugs, or just see what we’re working on right now, visit the Kedro projects on GitHub. We welcome every contribution, large or small.

Find out more about Kedro

There are many ways to learn more about Kedro:


On this page:

Photo of Jo Stichbury
Jo Stichbury
Technical Writer, QuantumBlack
Share post:
Mastodon logoLinkedIn logo

All blog posts

cover image alt

GenAI — 10 min read

Building a GenAI-powered chatbot using Kedro and LangChain

This post shows how to use Kedro to build and organize GenAI applications with a real-world example: a Retrieval-Augmented Generation (RAG) chatbot trained on Kedro Slack conversations. You'll learn how to structure your pipeline, manage LLMs and prompts, and apply practical Kedro tricks to streamline GenAI workflows - plus see why RAG outperforms plain LLMs in real use cases.

Elena Khaustova

25 Apr 2025

cover image alt

Success stories — 10 min read

Building Robust Data Science Pipelines at TomTom with Kedro

In this guest blog post, Toni Almagro, Senior Staff Data Scientist at TomTom, shares the transformative journey of Map Quality & Insights as the team transitioned from using Databricks notebooks to the Kedro framework for building data science pipelines. Initially prioritizing speed, the team faced challenges with technical debt, code repetition, and version control issues, which made their workflows unsustainable.

Toni Almagro

21 Apr 2025

cover image alt

News — 5 min read

Deprecating Experiment Tracking in Kedro Viz

Kedro-Viz will phase out its Experiment Tracking feature in the upcoming release of Kedro-Viz 11.0, with complete removal in version 12.0 due to low user adoption and the availability of robust alternatives like MLflow. This blog post includes detailed guidance on migrating to kedro-mlflow, a plugin that seamlessly integrates Kedro with MLflow.

cover image alt

Feature highlight — 5 min read

Top 10 features added to the Kedro ecosystem in 2024

This blog post highlights ten of the most notable enhancements and improvements to the Kedro ecosystem in the recent releases.

Merel Theisen

7 Oct 2024

cover image alt

Kedro newsletter — 5 min read

In the pipeline: October 2024

From the latest news to upcoming events and interesting topics, “In the Pipeline” is overflowing with updates for the Kedro community.

Jo Stichbury

2 Oct 2024