End-to-End Flow

End-to-End Flow

Here’s how the components of LakeFusion interact in a typical user session:

  1. User Login:
    The user accesses LakeFusion via the web browser. Their credentials are authenticated using Databricks OIDC and SSO through Okta or Azure AD.

  2. Portal Navigation:
    After login, users interact with one of the portals (Root, Main, or Utility), depending on their tasks.

  3. Backend Orchestration:
    The front-end portals send requests to the relevant microservices. For example:

    • The Databricks Service handles data queries or workflow orchestration.

    • The AI/ML Service performs analytics tasks, such as deduplication or anomaly detection.

  4. Data Processing in Databricks:
    Requests are forwarded to Databricks for processing. Services such as Workflows, SQL Warehouse, or Model Serving are utilized to complete the task.

  5. Results Display:
    Processed data or analytics outputs are sent back to the front end for the user to view, interact with, or export.

    • Related Articles

    • Data Flow in LakeFusion

      This section provides a structured overview of the LakeFusion Data Flow, outlining the key stages and enabling technologies that support seamless data ingestion, preprocessing, and Master Data Management (MDM). Each stage ensures data is unified, ...

    • Technical Stack

      Front End: React, single-spa. Backend Microservices: Python, FastAPI. Database: MySQL, MS SQL, PostgreSQL. Cloud Platform: Databricks. Authentication: OIDC, SSO (Okta, Azure AD). Integration: Databricks Python SDK.Containerization and Orchestration: ...

    • Integration Hub

      Integration Task creation Navigate to Integration Hub post-Match Maven completion Configure new pipeline with required parameters: Task Name designation Entity selection Model specification Execute task creation Access workflow configuration via ...

    • Data Quality Diagramming Configuration

      This section walks you through the steps to create, configure, run, and review a data quality task using Diagramming Configuration in LakeFusion. The process is designed to ensure your data meets the required standards for accuracy, consistency, and ...

    • LakeFusion Environment

      The LakeFusion Environment is the core of the platform and consists of two main layers: Micro Frontends: The user interface components. Microservices: The backend logic and processing components. a. Micro Frontends The LakeFusion Micro Frontends are ...