Databricks Setup
Prerequisites
- Databricks on AWS, Azure, or GCP, or a serverless workspace.
- Unity Catalog enabled on the workspace.
- A serverless SQL warehouse.
Setup
Configure Authentication
Create a Service Principal
Follow the Databricks documentation to create a Service Principal.
Connection details
Note down the Client ID and Client Secret.
Create a Personal Access Token
Follow the Databricks documentation to create a PAT.
Connection details
Note down the personal access token.
Create a SQL Warehouse
Log in to the Databricks console to create or reuse a SQL warehouse.
- Go to SQL > SQL warehouses > Create SQL warehouse
- Fill in the fields
- Name
- Warehouse Size (2X Small)
- Warehouse Type (Serverless)
- Auto Stop (10 minutes)
- Scaling Min & Max (1)
- Unity Catalog (Enabled)
- Click Create
- Once created, click on the Connection Details tab
Connection details
Note down the following:
- Server Hostname (your-workspace.cloud.databricks.com)
- Warehouse ID (0123456789abcdef)
Configure a Catalog
Choose a catalog, or create one following the Databricks documentation.
- From the Databricks workspace console, navigate to Data
- Choose a catalog (
my_catalog)
Create a Volume
Supermetal uses the configured volume as a temporary stage.
Follow the steps from the Databricks documentation.
- From the Databricks workspace console, navigate to Catalog
- Choose the catalog from the previous step (
my_catalog) - Search or browse for the schema to add the volume to and select it
- Click Create Volume and specify a name
- Click Create
CREATE VOLUME my_catalog.my_schema.my_volume;Connection details
Note down the following:
- Catalog Name (my_catalog)
- Volume Path (/Volumes/my_catalog/my_schema/my_volume)
Storage Credential
By default, data stages through the configured volume. A Databricks storage credential with access to the object store buffer lets Databricks read the files directly from the buffer instead.
Last updated on