Databricks Workspace Integration

Databricks Workspace Integration

LakeFusion seamlessly integrates with Databricks, leveraging its robust features for data analytics and governance. Key aspects include:

a. Authentication and Access

  • Databricks OIDC: Facilitates secure communication between LakeFusion and Databricks.

  • SSO Integration: User authentication flows are unified through SSO, using providers like Okta or Azure AD.

b. Databricks Services Utilized

LakeFusion relies on the following Databricks services:

  • Workflows: To automate and orchestrate data pipelines.

  • SQL Warehouse: For executing queries and accessing large-scale datasets.

  • Model Serving: To deploy and manage machine learning models.

  • Unity Catalog: To ensure consistent data governance and compliance across the platform.

c. Communication via Python SDK

The LakeFusion microservices use the Databricks Python SDK to interact programmatically with Databricks resources. This enables:

  • Automated job execution and scheduling.

  • Real-time data processing and analytics.

  • Integration of advanced models for predictive analytics.

    • Related Articles

    • Integration Hub

      Integration Task creation Navigate to Integration Hub post-Match Maven completion Configure new pipeline with required parameters: Task Name designation Entity selection Model specification Execute task creation Access workflow configuration via ...

    • Technical Stack

      Front End: React, single-spa. Backend Microservices: Python, FastAPI. Database: MySQL, MS SQL, PostgreSQL. Cloud Platform: Databricks. Authentication: OIDC, SSO (Okta, Azure AD). Integration: Databricks Python SDK.Containerization and Orchestration: ...

    • Dataset Creation

      In the this configuration, the dataset’s location within Databricks is specified, and it is set up for subsequent tasks in LakeFusion. Locate and select the Datasets card on the home screen Initialize the dataset creation process with the “Create ...

    • Customer Environment

      The Customer Environment represents the user's access point to the LakeFusion platform. The key steps include: User Access: Customers access LakeFusion via their web browser. Authentication: Authentication is handled using Databricks OpenID Connect ...

    • LakeFusion Deployment Guide via Azure Marketplace

      Prerequisite: Register Required Azure Resource Providers Before deploying LakeFusion from the Azure Marketplace, ensure that the necessary resource providers are registered for your Azure subscription. Follow the steps below: Navigate to ...