Data Quality Notebook Configuration

Data Quality Notebook Configuration

This section walks you through the steps to create, configure, run, and review a data quality task using Notebook Configuration in LakeFusion. The process is designed to ensure your data meets the required standards for accuracy, consistency, and reliability.

Step 1: Creating a Data Quality Task

  • Access the Data Quality card (either from Home or from the left navigation pane) 

  • Click on Create Quality Task

You have two options to define your task:

  1. Notebook Configuration: Use a predefined notebook template to configure your data quality tasks.

  1. Diagramming: Use a visual interface to create and link data quality tasks in a flow.

This section walks you through the steps to create, configure, run, and review a data quality task using Notebook Configuration in LakeFusion.      
            
            Provide the following required information:

  • Quality Task Name (following naming standards)

  • Detailed task description

  • Choose a task type

  • Target dataset selection

  • Task execution frequency (if required)

Step 2: Configuration

  • After choosing a task type, the task will be in draft status and change to configured after this step.




Step 3: Notebook Configuration


  • Download the Template Notebook containing predefined quality tasks.


  • Modify the notebook as needed and upload it back by clicking on Update Notebook and selecting your notebook and click on confirm.




Step 4: Run task
  1. Execute the data quality task by clicking on run now in the details tab. The system will apply the configured rules and logic to the selected dataset.       

Step 5: View Results

  • After configuration, a job is created, and job details can be found in the Details Tab.

  • View task execution results in the results tab.

  • Monitor the task execution status (e.g., In Progress or Completed) in the Runs Tab.


    • Related Articles

    • Data Quality Diagramming Configuration

      This section walks you through the steps to create, configure, run, and review a data quality task using Diagramming Configuration in LakeFusion. The process is designed to ensure your data meets the required standards for accuracy, consistency, and ...

    • Data Profiling Configuration

      This section walks you through the Data Profiling process in LakeFusion, which analyzes datasets to generate key metrics that reveal data structure, assess quality, and identify anomalies for informed decision-making and improved data management. ...

    • Data Flow in LakeFusion

      This section provides a structured overview of the LakeFusion Data Flow, outlining the key stages and enabling technologies that support seamless data ingestion, preprocessing, and Master Data Management (MDM). Each stage ensures data is unified, ...

    • Overview

      LakeFusion is an AI-powered Master Data Management (MDM) solution, purpose-built for the Databricks Lakehouse Platform. It delivers a single source of truth by unifying fragmented data across systems using advanced entity resolution and deduplication ...

    • Integration Hub

      Integration Task creation Navigate to Integration Hub post-Match Maven completion Configure new pipeline with required parameters: Task Name designation Entity selection Model specification Execute task creation Access workflow configuration via ...