---
description: "Set up no-code metadata ingestion in DataHub through the UI to automatically pull metadata from your data sources on a schedule."
---

import FeatureAvailability from '@site/src/components/FeatureAvailability';

# Metadata Ingestion

<FeatureAvailability/>

DataHub helps you discover and understand your organization's data by automatically collecting information about your data sources. This process is called **metadata ingestion**, allowing DataHub to automatically pull in:

- **Table and column names** from your databases
- **Asset Lineage** showing how information flows between systems
- **Usage statistics** revealing which datasets are most popular
- **Data quality information** including freshness and completeness
- **Business context** like ownership and documentation

This makes it simple to connect to popular platforms like Snowflake, BigQuery, dbt, and more, schedule automatic updates, and manage credentials securely.

## Prerequisites and Permissions

To manage metadata ingestion in DataHub, you need appropriate permissions.

:::note Ask DataHub for Ingestion (Public Beta - DataHub Cloud)
**Ask DataHub** (Public Beta) is available within the ingestion creation and troubleshooting workflow for DataHub Cloud deployments. Get AI-powered assistance with configuration, filtering, troubleshooting, and best practices—right in your workflow.
:::

### Option 1: Admin-Level Access

Users can be granted the following privileges for full administrative access to all ingestion sources:

- **`Manage Metadata Ingestion`** - Provides complete access to create, edit, run, and delete all ingestion sources
- **`Manage Secrets`** - Allows creation and management of encrypted credentials used in ingestion configurations

These privileges can be granted in two ways:

1. **Admin Role Assignment** - Users assigned to the **Admin Role** receive these privileges by default
2. **Custom Policy with Platform Privileges** - Create a [Custom Policy](authorization/policies.md) that grants the `Manage Metadata Ingestion` and `Manage Secrets` platform privileges to specific users or groups

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion-privileges.png"/>
</p>

### Option 2: Resource-Specific Policies

For more granular control, administrators can create [Custom Policies](authorization/policies.md) that apply specifically to **Ingestion Sources**, allowing different users to have different levels of access:

- **View** - View ingestion source configurations and run history
- **Edit** - Modify ingestion source configurations
- **Delete** - Remove ingestion sources
- **Execute** - Run ingestion sources on-demand

**Prerequisites:**

- **DataHub Core**: Enable the `VIEW_INGESTION_SOURCE_PRIVILEGES_ENABLED` feature flag
- **DataHub Cloud**: Work with your customer success team to get the feature enabled

:::caution
**Important**: Once this feature flag is enabled, any policies that apply to "All" resource types will now include Ingestion Sources, including the default read-only policies. This will make the Ingestion tab visible and potentially actionable depending on the applied privileges. Implement this with care if you have view-only policies that should not expose the Data Sources page.
:::

## Creating an Ingestion Source

Once you have the appropriate privileges, navigate to the **Ingestion** tab in DataHub.

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion-tab.png"/>
</p>

On this page, you'll see a list of active **Ingestion Sources**. An Ingestion Source represents a configured connection to an external data system from which DataHub extracts metadata.

If you're just getting started, you won't have any sources configured. The following sections will guide you through creating your first ingestion source.

### Step 1: Select a Data Source

Begin by clicking **+ Create source** to start the ingestion source creation process.

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/create-new-ingestion-source-button.png"/>
</p>

Next, select the type of data source you want to connect. DataHub provides pre-built templates for popular platforms including:

- **Data Warehouses**: Snowflake, BigQuery, Redshift, Databricks
- **Databases**: MySQL, PostgreSQL, SQL Server, Oracle
- **Business Intelligence**: Looker, Tableau, PowerBI
- **Streaming**: Kafka, Pulsar
- **And many more...**

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/select-platform-template.png"/>
</p>

Select the template that matches your data source. If your specific platform isn't listed, you can choose **Custom** to configure a source manually, though this requires more technical knowledge.

### Step 2: Configure Connection Details

After selecting your data source template, you'll configure how DataHub connects to and extracts metadata from your source.

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/ingestion-connection-ask-datahub.png"/>
</p>

_Ask DataHub (Public Beta - Cloud only) provides contextual assistance throughout the ingestion configuration process_

**Name and Owners**: First, provide a descriptive name for your ingestion source that will help you and your team identify it later. You can also assign **Users** and/or **Groups** as owners of this ingestion source. By default, you (the creator) will be assigned as an owner, but you can add additional owners or change this at any time after creation.

**Connection Information**: Next, you'll configure the connection details using a user-friendly form. The exact fields will vary depending on your chosen platform, but typically include:

- Host/server address and port
- Database or project names
- Authentication credentials

**Asset Filters**: Configure what metadata to extract:

- Which databases, schemas, or tables to include
- Filtering options to exclude certain data

**Ingestion Settings**: Configure ingestion behavior including profiling, stale metadata handling, and other operational settings. The defaults represent best practices for most use cases.

:::note Ask DataHub for Configuration Help (Public Beta)
**Ask DataHub** (Public Beta) can help you understand the behavior and options of each configuration setting. Get tailored recommendations for your data source and use case.
:::

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/ingestion-configuration-ask-datahub-2.png"/>
</p>

_Ask DataHub (Public Beta - Cloud only) helps you understand configuration options and provides tailored recommendations for your data source_

#### Managing Sensitive Information with Secrets

For production environments, sensitive information like passwords and API keys should be stored securely using DataHub's **Secrets** functionality.

To create a secret:

1. Navigate to the **Secrets** tab in the Ingestion interface
2. Click **Create new secret**
3. Provide a descriptive name (e.g., `BIGQUERY_PRIVATE_KEY`)
4. Enter the sensitive value
5. Optionally add a description
6. Click **Create**

Once created, secrets can be referenced in your ingestion configuration forms using the dropdown menus provided for credential fields.

:::caution Security Note
Users with the `Manage Secrets` privilege can retrieve plaintext secret values through DataHub's GraphQL API. Ensure secrets are only accessible to trusted administrators.
:::

#### Test Your Connection

Before proceeding, it's important to verify that DataHub can successfully connect to your data source. Most ingestion source forms include a **Test Connection** button that validates:

- Network connectivity to your data source
- Authentication credentials
- Required permissions for metadata extraction

<p align="center">
  <img width="70%" alt="Test BigQuery connection" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/bigquery/bigquery-test-connection.png"/>
</p>

If the connection test fails, review your configuration and ensure that:

- Network access is available between DataHub and your data source
- Credentials are correct and have sufficient permissions
- Any firewall rules allow the connection

#### Advanced Settings

For users who need additional control, DataHub provides advanced configuration options accessible in the Advanced Settings section:

- **CLI Version:** Specify a particular version of the DataHub CLI for ingestion execution
- **Environment Variables:** Set custom environment variables for the ingestion process
- **Executor ID:** Configure remote execution if needed
- **Debug Mode:** Enable detailed logging for troubleshooting

### Step 3: Sync Schedule

Configure how often DataHub should sync metadata from your source. You can enable or disable scheduled execution using the toggle (recommended: enabled). This ensures your metadata stays up-to-date without manual intervention.

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/schedule-ingestion.png"/>
</p>

If you prefer to run ingestion manually or on an ad-hoc basis, you can skip the scheduling step entirely.

### Step 4: Review and Save

Review your configuration to ensure all settings are correct. When you're ready, you have two options:

- **Save**: Save the ingestion source configuration without executing it immediately
- **Save and Run**: Save and immediately execute your first ingestion run

Once you're happy with your configurations, click your preferred save option to finalize your source.

## Running and Monitoring Ingestion

### Executing an Ingestion Source

Once you've created your Ingestion Source, you can run it by clicking the 'Play' button. Shortly after, you should see the 'Last Status' column of the ingestion source change to `Running`, indicating that DataHub has successfully queued the ingestion job.

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/running.png"/>
</p>

When ingestion completes successfully, the status will show as `Success` in green.

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/success-run.png"/>
</p>

### Viewing Run History

The **Run History** tab shows you a complete history of all your ingestion runs. Here you can:

- **See all runs**: View every ingestion execution across all your sources
- **Check recent activity**: Runs are listed with the most recent at the top
- **Filter by source**: Use the dropdown to see runs from a specific ingestion source
- **Access from Sources tab**: Click on any source's **Last Run** status or select **View Run History** from the source menu

This makes it easy to track your ingestion performance and troubleshoot any issues over time.

<p align="center">
  <img width="80%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/run-history-tab.png"/>
</p>

### Viewing Ingestion Results

After successful ingestion, you can view detailed information about what was extracted:

1. Click the **Success** status button on a completed ingestion run
2. Select **View All** to see the list of ingested entities
3. Click on individual entities to validate the extracted metadata

<p align="center">
  <img width="75%" alt="ingestion_details_view_all" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/ingestion-run-summary.png"/>
</p>

### Cancelling Running Ingestion

If an ingestion run is taking too long or appears to be stuck, you can cancel it by clicking the 'Stop' button on the running job.

<p align="center">
  <img width="80%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/cancelled-run.png"/>
</p>

This is useful when encountering issues like:

- Network timeouts
- Ingestion source bugs
- Resource constraints

## Troubleshooting Failed Ingestion

### Common Failure Reasons

When an ingestion run fails, you'll see a failed status indicator in your sources list.

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/failed-source.png"/>
</p>

Common causes of ingestion failures include:

1. **Configuration Errors**: Incorrect connection details, missing required fields, or invalid parameter values
2. **Authentication Issues**: Wrong credentials, expired tokens, or insufficient permissions
3. **Network Connectivity**: DNS resolution failures, firewall blocks, or unreachable data sources
4. **Secret Resolution Problems**: Referenced secrets that don't exist or have incorrect names
5. **Resource Constraints**: Memory limits, timeouts, or processing capacity issues

### Viewing Detailed Logs

To diagnose ingestion failures, click on a run history status (Failed, Aborted) value to view and download comprehensive ingestion run logs.

<p align="center">
  <img width="70%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion/ingestion-run-log.png"/>
</p>

The logs provide detailed information about:

- Connection attempts and errors
- Authentication failures
- Data extraction progress
- Error messages and stack traces

### Authentication for Secured DataHub Instances

If your DataHub instance has [Metadata Service Authentication](authentication/introducing-metadata-service-authentication.md) enabled, you'll need to provide a Personal Access Token in your configuration.

<p align="center">
  <img width="80%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/ingestion-with-token.png"/>
</p>

## Advanced Configuration with YAML

While the UI-based forms handle most common ingestion scenarios, advanced users may need direct access to YAML configuration for:

- Custom ingestion sources not available in the UI
- Complex transformation pipelines
- Advanced filtering and processing logic
- Integration with external systems

For these advanced use cases, DataHub supports direct YAML recipe configuration. For detailed information about YAML-based configuration, including syntax and examples, see the [Recipe Overview Guide](metadata-ingestion/recipe_overview.md).

### Deploying Recipes (CLI)

You can deploy recipes using the CLI as mentioned in the [CLI documentation for uploading ingestion recipes](./cli.md#ingest-deploy).

```bash
datahub ingest deploy --name "My Test Ingestion Source" --schedule "5 * * * *" --time-zone "UTC" -c recipe.yaml
```

### Deploying Recipes (GraphQL)

Create ingestion sources using [DataHub's GraphQL API](./api/graphql/overview.md) using the **createIngestionSource** mutation endpoint.

```graphql
mutation {
  createIngestionSource(
    input: {
      name: "My Test Ingestion Source"
      type: "mysql"
      description: "My ingestion source description"
      schedule: { interval: "*/5 * * * *", timezone: "UTC" }
      config: {
        recipe: "{\"source\":{\"type\":\"mysql\",\"config\":{\"include_tables\":true,\"database\":null,\"password\":\"${MYSQL_PASSWORD}\",\"profiling\":{\"enabled\":false},\"host_port\":null,\"include_views\":true,\"username\":\"${MYSQL_USERNAME}\"}},\"pipeline_name\":\"urn:li:dataHubIngestionSource:f38bd060-4ea8-459c-8f24-a773286a2927\"}"
        version: "0.8.18"
        executorId: "mytestexecutor"
      }
    }
  )
}
```

**Note**: Recipe must be double quotes escaped when using GraphQL

## Frequently Asked Questions

### Why does ingestion fail with 'Failed to Connect' errors in Docker environments?

If you're running DataHub using `datahub docker quickstart` and experiencing connection failures, this may be due to network configuration issues. The ingestion executor might be unable to reach DataHub's backend services.

Try updating your ingestion configuration to use the Docker internal DNS name:

<p align="center">
  <img width="80%"  src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/quickstart-ingestion-config.png"/>
</p>

### What does a dash mark (-) status mean and how do I fix it?

If your ingestion source shows a dash mark (-) status and never changes to 'Running', this could mean:

1. **The source has never been triggered to run** - Try clicking the "Play" button to execute the source
2. **The DataHub actions executor is not running or healthy** (DataHub Core users only)

If clicking "Play" doesn't resolve the issue, DataHub Core users should diagnose their actions container:

1. Check container status with `docker ps`
2. View executor logs with `docker logs <container-id>`
3. Restart the actions container if necessary

### When should I use CLI/YAML instead of UI ingestion?

Consider using CLI-based ingestion when:

- Your data sources aren't reachable from DataHub's network (use [remote executors](managed-datahub/operator-guide/setting-up-remote-ingestion-executor.md) for DataHub Cloud)
- You need custom ingestion logic not available in UI templates
- Your ingestion requires local file system access
- You want to distribute ingestion across multiple environments
- You need complex transformations or custom metadata processing

## Additional Resources

- **Demo Video**: [Watch a complete UI ingestion walkthrough](https://www.youtube.com/watch?v=EyMyLcaw_74)
- **Quick Start Guides**: Step-by-step setup instructions for popular data sources
- **Recipe Documentation**: [Comprehensive YAML configuration reference](metadata-ingestion/recipe_overview.md)
- **Integration Catalog**: [Browse all supported data sources and their features](https://docs.datahub.com/integrations)
