MainTargetsDatabricks
Databricks Setup
Prerequisites
Before you begin, ensure you have:
- Supported Databricks Implementations:
- Unity Catalog: Unity Catalog enabled on your Databricks workspace.
- SQL Warehouse: Serverless SQL Warehouse.
Setup
Configure Authentication
Create a Service Principal
Follow the databricks documentation to create a Service Principal.
Connection Details
Note down the Client ID and Client Secret.
Create a Personal Access Token
Follow the databricks documentation to create a PAT.
Connection Details
Note down the personal access token.
Create a SQL Warehouse
Login to Databricks console to create or use an existing SQL Warehouse.
- Go to SQL > SQL warehouses > Create SQL warehouse
- Fill in and select the required fields like
- Name
- Warehouse Size (2X Small)
- Warehouse Type (Serverless)
- Auto Stop (10 minutes)
- Scaling Min & Max (1)
- Unity Catalog (Enabled)
- etc.
- Click Create
- Once created, click on the Connection Details Tab
Connection Details
Note down the following details:
- Server Hostname (your-workspace.cloud.databricks.com)
- Warehouse ID (0123456789abcdef)
Configure a Catalog
Login to Databricks console to choose a Catalog (or set a new one from databricks documentation).
- From Databricks workspace console, navigate to Data
- Choose a Catalog (
my_catalog)
Create a Volume
Supermetal uses the configured volume as a temporary stage
Follow the steps from the databricks documentation.
- From Databricks workspace console, navigate to Catalog
- Choose the Catalog from above step (
my_catalog) - Search or browse for the schema that you want to add the volume to and select it.
- Click on Create Volume and specify a Name.
- Click Create
CREATE VOLUME my_catalog.my_schema.my_volume;Connection Details
Note down the following details:
- Catalog Name (my_catalog)
- Volume Path (/Volumes/my_catalog/my_schema/my_volume)
Last updated on