This document explains the steps needed to prepare your Databricks platform for a dScribe connection.
Prerequisites
Before you get started, make sure you have access to a user with:
administrator access to your Databricks platform
administrator access to your Azure Entra ID environment
compute up and running on your Databricks platform
Step-by-step guide
Steps in Azure
Go to Azure Entra ID
a. Go to https://portal.azure.com/#home
b. Select Azure Entra ID (previously called Azure Active Directory):
Create an app registration
TIP 1: remember the name of your application, you will need it laterTIP 2: (important!) Make sure there are no Databricks admin-consent-required permissions set on this application.
Navigate back to your application via Azure Entra ID > App Registrations > Select your newly created App and copy the Application (client) ID somewhere safe. You will need it later.
In your application, navigate to 'Certificates & secrets' and create a new client secret:
Copy the client secret value (not the Secret ID).
Steps in Databricks
Add a service principal in Databricks
Make sure to choose 'Microsoft Entra ID managed' and copy the Entra application ID and provide a Service Principle name.
In Databricks we will need 'USE CATALOG', 'USE SCHEMA' & 'BROWSE' rights on the catalogs you want to access. Next to that we also query the system.acess.table_lineage table to get all lineage out of Unity catalog.
Steps in dScribe
Using the Workspace URL (Databricks URL string ending with .net), Tenant ID, Application (client) ID and secret value, create a new Source in dScribe.