Skip to main content

Databricks

Squid can connect to a Databricks SQL warehouse to serve as a data source for your Squid application.​

With the Databricks connector, you can query tables in your Databricks catalog through the Squid Client SDK, run native SQL queries, and let AI agents answer questions using your lakehouse data.

note

The Databricks connector is read-only. Queries, including native SQL and AI queries, are supported, while inserts, updates, and deletes are not.

Databricks-side prerequisites​

  • A running SQL warehouse. You'll need its HTTP path (found under the warehouse's Connection details, in the form /sql/1.0/warehouses/<warehouse-id>).
  • A personal access token for authentication.
  • The token's user needs SELECT on the tables you want to expose, plus read access to the system.information_schema views (tables, columns, table_constraints, constraint_column_usage), which Squid uses for schema discovery.

Adding the Databricks connector​

  1. Open the Squid Console and select your application to use with Databricks.
  2. Click the Connectors tab, and then click Available connectors.
  3. Select the Databricks connector.
  4. Provide the following configuration information:
  • Connector ID - Choose an ID that is brief and helps identify the connector.
  • Database Host - Your workspace host. For example, dbc-yourId.cloud.databricks.com.
  • Database Access Token - Your personal access token. To keep it private, store it in Squid Secrets.
  • Warehouse Path - The SQL warehouse's HTTP path. For example, /sql/1.0/warehouses/warehouse-id-here.
  • Catalog - The catalog to connect to.
  • Catalog Schemas - A comma-separated list of schemas within the catalog to expose.
  • Tables to Process - Optional. A comma-separated list of tables to include in schema discovery. Use forward slashes to specify a regex pattern (e.g. /^temp_.*/). Leave empty to include all tables.

Querying your data​

Collections are named schema.table. Query them from the Client SDK the same way as any other Squid collection, passing the connector ID:

Client code
const orders = await squid
  .collection<Order>('sales.orders', 'YOUR_DATABRICKS_CONNECTOR_ID')
  .query()
  .gte('total', 1000)
  .snapshot();

You can also run native SQL queries:

Client code
const result = await squid.executeNativeRelationalQuery(
  'YOUR_DATABRICKS_CONNECTOR_ID',
  'SELECT region, SUM(total) AS revenue FROM sales.orders GROUP BY region',
);

Native and AI-generated queries are validated to be read-only before execution.

Securing your database​

By default, security rules are required, and in a production environment, it's essential to never allow access to the connector without them. Learn more about securing your database data in the security rules documentation.

To learn how to work with your data in Squid, view the Client SDK documentation.