Announcing General Availability of Azure Private Link and Azure Storage firewall support for Databricks SQL Serverless

By neub9
4 Min Read

We’re thrilled to announce the upcoming general availability of Azure Private Link support for Databricks SQL (DBSQL) Serverless in April 2024. There will be no additional charges for use. Additionally, Azure Storage firewall support with stable VNet subnet IDs is now generally available for DBSQL Serverless. This blog will provide details on both of these features, as well as best practices for securely accessing data from your Azure Storage account within Databricks serverless.

Maximize Performance and Security with Azure Databricks Serverless Network Connectivity Features

The Databricks Data Intelligence Platform provides robust security through multi-layered isolation and best practices, as outlined in our Trust Center. You can leverage data stored in existing Azure Storage accounts while enhancing security with the following options:

  1. Configure Azure Storage firewall to allow access based on stable VNet subnet IDs
  2. Configure Private Endpoints to use Private Link to your Storage account

The diagram below illustrates the high-level connections into and out of your Azure Databricks account for serverless. In this blog, we’ll focus on securing your connection between DBSQL Serverless workloads and your Azure Storage.

DBSQL Serverless workloads

Private Link connections from Databricks SQL Serverless workloads to Storage accounts will be available at no additional charge upon general availability. This substantial improvement boosts the TCO for DBSQL Serverless on Azure Databricks and sets the stage for support of additional Azure Databricks serverless products and Azure resource types down the line.

“Azure Databricks’ advanced networking features offer security and simplicity in managing serverless data transformations and analytics at scale.”

— Jonas Kardell, Data Science Lead, SJ AB

Azure Storage Firewall Support with Stable VNet Subnet IDs

Azure Storage firewall allows you to restrict access to your Storage account to only authorized workloads running on authorized networks. With stable VNet subnet IDs, you can configure Databricks to use a set list of subnets within Azure VNets to access your Storage. You can manage access by adding these subnets to your Azure Storage firewall rules, along with Azure Managed Identities for a layered protection approach.

Streamlined Serverless Network Connectivity Across Workspaces

The Network Connectivity Configuration (NCC) allows you to centrally manage network connectivity across multiple Workspaces, reducing the number of private endpoints you need to manage. NCC continues to be the single point of managing connectivity across all serverless products.

Network Connectivity Configuration

Getting Started with Serverless Network Connectivity on Azure Databricks

Azure Storage firewall support and Azure Private Link are available on the Premium Tier version of Azure Databricks. Refer to our documentation for step-by-step instructions. While Azure Private Link is in gated public preview, contact your Azure Databricks account team for more information on how to enroll. Azure Private Link support for Azure Databricks serverless will be generally available in April 2024.

For more information about Databricks’ security best practices and features, visit our Security and Trust Center.

Share This Article
Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *