Databricks Workspace Integration

Databricks Workspace Integration

LakeFusion seamlessly integrates with Databricks, leveraging its robust features for data analytics and governance. Key aspects include:

a. Authentication and Access

  • Databricks OIDC: Facilitates secure communication between LakeFusion and Databricks.

  • SSO Integration: User authentication flows are unified through SSO, using providers like Okta or Azure AD.

b. Databricks Services Utilized

LakeFusion relies on the following Databricks services:

  • Workflows: To automate and orchestrate data pipelines.

  • SQL Warehouse: For executing queries and accessing large-scale datasets.

  • Model Serving: To deploy and manage machine learning models.

  • Unity Catalog: To ensure consistent data governance and compliance across the platform.

c. Communication via Python SDK

The LakeFusion microservices use the Databricks Python SDK to interact programmatically with Databricks resources. This enables:

  • Automated job execution and scheduling.

  • Real-time data processing and analytics.

  • Integration of advanced models for predictive analytics.

    • Related Articles

    • Integration Hub

      Integration Task creation Navigate to Integration Hub post-Match Maven completion Configure new pipeline with required parameters: Task Name designation Entity selection Model specification Execute task creation Access workflow configuration via ...

    • Technical Stack

      Front End: React, single-spa. Backend Microservices: Python, FastAPI. Database: MySQL, MS SQL, PostgreSQL. Cloud Platform: Databricks. Authentication: OIDC, SSO (Okta, Azure AD). Integration: Databricks Python SDK.Containerization and Orchestration: ...

    • Customer Environment

      The Customer Environment represents the user's access point to the LakeFusion platform. The key steps include: User Access: Customers access LakeFusion via their web browser. Authentication: Authentication is handled using Databricks OpenID Connect ...

    • LakeFusion Deployment Guide via Azure Marketplace

      Prerequisite: Register Required Azure Resource Providers Before deploying LakeFusion from the Azure Marketplace, ensure that the necessary resource providers are registered for your Azure subscription. Follow the steps below: Navigate to ...

    • Troubleshooting Guide

      This section outlines known issues encountered during setup or usage of LakeFusion, along with steps to resolve them. 1. Error Accessing the Match Maven Screen Issue: Users encounter an error when attempting to access the Match Maven screen within ...