Maia Flex connector

This page describes how to configure the Maia connector component as part of a data pipeline within Designer. The Maia component uses the Connect and Configure parameters to create a table of Maia data, which is then stored in your chosen data warehouse or cloud storage location. You do not need to use the Create Table component when using this connector, as the Maia component will create a new table or replace an existing table for you using the Destination parameters you define.

The Maia connector is a Flex connector. Out of the box, you can use it to:

• List and retrieve details for projects, environments, published pipelines, and connector profiles.
• Execute published pipelines, and retrieve execution and step status details.
• Create, retrieve, and manage schedules, artifacts, secret references, and runners.
• Retrieve consumption, audit event, and data lineage information.
• Trigger test executions, and retrieve their status.

In Maia, Flex connectors let you connect to a curated set of endpoints to load data.

You can use Maia connector in its preconfigured state, or you can edit the connector by adding or amending available Maia endpoints as per your use case. You can edit Flex connectors in the Custom Connector user interface.

For detailed information about authentication, specific endpoint parameters, pagination, and more aspects of the Maia API, read the Maia API documentation.

If your account uses IP restrictions and you’re running this connector on a Hybrid SaaS Maia runner, you must add your Maia runner‘s outbound IP address or range to the Maia IP allow list. For more information, read IP allow list.

---

Video example

Watch our video about using Maia Flex connector: YouTube.

---

Properties

Reference material is provided below for the Connect, Configure, Destination, and Advanced Settings properties.

Name · string · required

A human-readable name for the component.

Connect

Data Source · drop-down · required

The data source to load data from in this pipeline. The drop-down menu lists the Maia API endpoints available in the connector. For detailed information about specific endpoints, read the Maia API documentation.

Endpoint	Method	Reference
Projects
List All Projects	GET	Retrieves a list of projects
Environments
List All Environments	GET	Retrieves a list of environments
Pipelines
List All Published Pipelines	GET	Retrieves a list of published pipelines
Pipeline Execution
Execute Published Pipeline	POST	Execute a pipeline
Pipeline Executions	GET	Retrieves a list of summarized pipeline executions
Pipeline Execution Details	GET	Retrieves the status of an executed pipeline
Pipeline Steps Status	GET	Retrieves a single execution of a pipeline triggered from a step of another pipeline
Consumption
Get Flat-Rated Product Consumption	GET	Retrieves a breakdown of your credit usage for flat-rated products. This data refreshes every three hours to reflect recent credit usage.
Get ETL Users Consumption	GET	Retrieves a breakdown of your credit usage for Matillion ETL users. This data refreshes every three hours to reflect recent credit usage.
Schedules
List All Schedules	GET	Retrieves all schedules for a project
Create Schedule	POST	Create a new schedule for a specific project, pipeline and environment
Get Schedule Summary	GET	Retrieves schedule details for a given schedule id
Artifacts
List Artifacts	GET	Retrieves a list of artifacts in an environment
Get Artifact	GET	Retrieves an artifact by a version name
Promote Artifact	POST	Promote artifacts from one environment to another
Connectors
List Custom Connectors	GET	Retrieves custom connector profiles for the requesting account
List Flex Connector Profiles	GET	Retrieves flex connector profiles
Secret References
List All Secret References	GET	Retrieves a list of all secret references
Create Secret Reference	POST	Create a secret reference
Agents (Runners)
List All Agents	GET	Lists all Maia runners, including summary details for each Maia runner
Create Agent	POST	Creates a new Maia runner with the specified configuration details
Get Agent Details	GET	Retrieves details for a specific Maia runner
Trigger Agent Command	POST	Triggers a specific command on a Maia runner
Get Agent Client Credentials	GET	Retrieves the Client ID and Client secret for the Maia runner
Perform Action On Agent Credentials	POST	Perform actions on Maia runner credentials, such as refreshing them
Audit Events
Query Audit Events	GET	Retrieves audit events for a given time range
Data Lineage
Get Lineage Events	GET	Retrieves OpenLineage events for a specified time period
Test Execution
Create Test Execution	POST	Triggers one or more test executions in a specific project
List Test Executions	GET	Returns a list of summarized test executions
Test Execution Status	GET	Returns the status of an executed test

Authentication Type · drop-down · required

The authentication method to authorize access to your Maia data. Currently supports OAuth 2.0 Client Credentials. Read Authenticating to the API to learn more.

Authentication · string · required

Select your authentication profile.

To create a new OAuth connection, read the OAuth section of the custom connector authentication guide.

Configure

URI Parameters · column editor · required

• Parameter Name: The name of a URI parameter.
• Parameter Value: The value of the corresponding parameter.

Required parameter	Endpoints	Description
server	All endpoints	Enter eu1 or us1, depending on the region of your Maia account. To find your account’s region, click the Profile & Account icon on the left side of the page.
api_version	All endpoints	Enter v1.
projectId	List All Environments, List All Published Pipelines, Execute Published Pipeline, Pipeline Execution Details, Pipeline Steps Status, List All Schedules, Create Schedule, List Artifacts, Get Artifact, Promote Artifact, List All Secret References, Create Secret Reference, Create Test Execution, Test Execution Status	projectId is unique to every project. Retrieve this value by using the List All Projects endpoint.
pipelineExecutionId	Pipeline Execution Details, Pipeline Steps Status	pipelineExecutionId is unique to every pipeline. Retrieve this value by using the Execute Published Pipeline endpoint.
secretReferenceName	Create Secret Reference	The name of Maia secret reference. This can be found in the Secret definitions tab in Designer, under the Name column.
agentId	Get Agent Details, Trigger Agent Command, Get Agent Client Credentials, Perform Action On Agent Credentials	The ID of the Maia runner to retrieve. To find this, in the left navigation, click , then Runners. Select the intended Maia runner, and click the Parameters tab.

Query Parameters · column editor · required

• Parameter Name: The name of a query parameter.
• Parameter Value: The value of the corresponding parameter.

Required parameter	Endpoints	Description
size	List All Projects, List All Environments, List All Published Pipelines, Get Pipeline Steps Status, List All Schedules, List Artifacts, List All Secret References, List All Agents, Query Audit Events, Get Lineage Events	Enter the number of records per page, ranging from 1 to 100.
environmentName	List All Published Pipelines, List Artifacts, Get Artifact, Create Test Execution	Enter the environment name. For example, test-environment-1.
consumedFrom	Get Flat-Rated Product Consumption	Enter the start date for the results. This value is inclusive, meaning results from this date onward are included. For example, 2024-11-01.
consumedBefore	Get Flat-Rated Product Consumption	Enter the end date for the results. This value is exclusive, meaning it includes only results occurring before (but not on) this date. For example, 2024-12-01.
consumedFrom	Get ETL Users Consumption	Enter the start date and time for the results. This value is inclusive, meaning results from this date and time onward are included. For example, 2024-07-01T00:00:00.123Z.
consumedBefore	Get ETL Users Consumption	Enter the end date and time for the results. This value is exclusive, meaning it includes only results occurring before (but not on) this date and time. For example,2024-07-31T00:00:00.123Z.
versionName	Get Artifact	The Version name when you Push local changes to the remote repository in Designer. For more information, read Git push local changes.
limit	Pipeline Executions	Enter the maximum number of results to return. The default value is set to 25.
from	Query Audit Events	Enter the earliest date and time of audit events to retrieve. The date time format must be in ISO 8601 format, for example: 2025-02-20T07:15:15.000-01:00.
to	Query Audit Events	Enter the latest date and time of audit events to retrieve. The date time format must be in ISO 8601 format, for example: 2025-02-21T07:15:15.000-01:00.
generatedFrom	Get Lineage Events	Include events generated on or after this date time. The value must be earlier than generatedBefore.
generatedBefore	Get Lineage Events	Include events generated up to, but not including, this date time. The value must be later than generatedFrom.
page	Get Lineage Events	The page number to use for pagination, starting at 0. Must be 0 or greater.

The Get ETL Users Consumption endpoint provides information about the number of credits charged for Matillion ETL users, and identifies which users contributed to those charges. Matillion ETL users are billed based on monthly active unique users, so ensure that the consumedFrom and consumedBefore parameters correspond to the timeframe of a single monthly invoice.

Header Parameters · column editor · required

• Parameter Name: The name of a header parameter.
• Parameter Value: The value of the corresponding parameter.

Required parameter	Endpoints	Description
Content-Type	Execute Published Pipeline, Pipeline Execution Status, Pipeline Steps Status, Create Schedule, Promote Artifact, Create Secret Reference, List All Agents, Create Agent, Get Agent Details, Trigger Agent Command, Get Agent Client Credentials, Perform Action On Agent Credentials, Query Audit Events, Get Lineage Events	Enter application/json.
accept	Execute Published Pipeline, Pipeline Execution Details, Pipeline Steps Status	Enter application/json.

Post Body · JSON

A JSON body to include as part of a POST request. Use Custom Connector to test your endpoints work as expected before moving to Designer pipelines.

You should also consult the developer documentation for the API you’re connecting to—as the developer portal may provide additional information about endpoints and requests.

For the Execute Published Pipeline endpoint, include the following POST Body. This example demonstrates a POST Body used to execute a pipeline:

{
  "pipelineName": "test-pipeline",
  "environmentName": "test-environment"
}

For the Create Schedule endpoint, include the following POST Body. This example demonstrates a POST Body used to create a schedule:

{
  "pipeline": {
    "pipelineName": "pipeline-name",
    "environmentName": "environment-name"
  },
  "schedule": {
    "cronExpression": "0 * * ? * * *",
    "cronTimezone": "Europe/Dublin",
    "effectiveFrom": "2022-05-19T12:37:44Z",
    "name": "schedule-name",
    "scheduleEnabled": false
  }
}

For the Promote Artifact endpoint, include the following POST Body. This example demonstrates a POST Body used to promote an artifact:

{
  "sourceEnvironmentName": "source-environment-name",
  "targetEnvironmentName": "environment-name",
  "versionName": "version-name"
}

For the Create Secret Reference endpoint, include the following POST Body. This example demonstrates a POST Body used to create a secret for an AWS Maia runner:

{
  "agentId": "agent-id",
  "agentType": "AWS",
  "description": "My secret reference",
  "secretReferenceType": "PASSWORD",
  "secretKey": "aws-secret-key",
  "secretName": "aws-secret-name"
}

For the Create Agent endpoint, include the following POST Body. This example demonstrates a POST Body used to create a new AWS Maia runner:

{
  "agentType": "data_productivity_cloud",
  "cloudProvider": "aws",
  "deployment": "fargate",
  "description": "An AWS Agent",
  "enableAutoUpdates": true,
  "name": "AWS Agent",
  "restrictedAccess": true,
  "trackName": "current"
}

For the Trigger Agent Command endpoint, include the following POST Body. This example demonstrates a POST Body used to trigger the RESTART Maia runner command:

{
  "command": "RESTART"
}

Other available Maia runner commands are:

• PAUSE
• RESUME

For the Perform Action On Agent Credentials endpoint, include the following POST Body. This example demonstrates a POST Body used to refresh Maia runner credentials:

{
  "action": "SECRET_REFRESH"
}

For other Maia runners, the POST body will vary. For example, in Azure, you must specify a value for vaultName.

Page Limit · integer

Enter a numeric value to use as the maximum number of records per page.

Destination

Select your cloud data warehouse.

Snowflake

Destination · drop-down · required

• Snowflake: Load your data into Snowflake. You’ll need to set a cloud storage location for temporary staging of the data.
• Cloud Storage: Load your data directly into your preferred cloud storage location.

Click either the Snowflake or Cloud Storage tab on this page for documentation applicable to that destination type.

Snowflake

Warehouse · drop-down · required

The Snowflake warehouse used to run the queries. The special value [Environment Default] uses the warehouse defined in the environment. Read Overview of Warehouses to learn more.

Database · drop-down · required

The Snowflake database. The special value [Environment Default] uses the database defined in the environment. Read Databases, Tables and Views - Overview to learn more.

Schema · drop-down · required

The Snowflake schema. The special value [Environment Default] uses the schema defined in the environment. Read Database, Schema, and Share DDL to learn more.

Table Name · string · required

The name of the table to be created.

Load Strategy · drop-down · required

• Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
• Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
• Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
• Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.

Clean Staged files · boolean · required

• Yes: Staged files will be destroyed after data is loaded. This is the default setting.
• No: Staged files are retained in the staging area after data is loaded.

Stage Access Strategy · drop-down

Select the stage access strategy. The strategies available depend on the cloud platform you select in Stage Platform.

• Credentials: Connects to the external stage (AWS, Azure) using your configured cloud provider credentials. Not available for Google Cloud Storage.
• Storage Integration: Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the Storage Integration property, through which you can select any of your existing Snowflake storage integrations.

Stage Platform · drop-down · required

Choose a data staging platform using the drop-down menu.

• Amazon S3: Stage your data on an AWS S3 bucket.
• Snowflake: Stage your data on a Snowflake internal stage.
• Azure Storage: Stage your data in an Azure Blob Storage container.
• Google Cloud Storage: Stage your data in a Google Cloud Storage bucket.

Click one of the tabs below for documentation applicable to that staging platform.

Amazon S3

Storage Integration · drop-down · required

Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.

Amazon S3 Bucket · drop-down · required

An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Snowflake

Internal Stage Type · drop-down · required

Select the Snowflake internal stage type. Use the Snowflake links provided to learn more about each type of stage.

• User: Each Snowflake user has a user stage allocated to them by default for file storage. You may find the user stage convenient if your files will only be accessed by a single user, but need to be copied into multiple tables.
• Named: A named stage provides high flexibility for data loading. Users with the appropriate privileges on the stage can load data into any table. Furthermore, because the stage is a database object, any security or access rules that apply to all objects will apply to the named stage.

Named stages can be altered and dropped. User stages cannot.

Named Stage · drop-down · required

Select your named stage. Read Creating a named stage to learn how to create a new named stage.

You can’t select a named stage if its name includes special characters or spaces.

Azure Storage

Storage Integration · drop-down · required

Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.

Storage Account · drop-down · required

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container · drop-down · required

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

Google Cloud Storage

Stage Access Strategy · drop-down

Select the stage access strategy. The strategies available depend on the cloud platform you select in Stage Platform.

• Storage Integration: Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the Storage Integration property, through which you can select any of your existing Snowflake storage integrations.

Storage Integration · drop-down · required

Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.

GCS Bucket · drop-down · required

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite · boolean · required

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Cloud Storage

Load Strategy · drop-down

• Append Files in Folder: Appends files to storage folder. This is the default setting.
• Overwrite Files in Folder: Overwrite existing files with matching structure.

See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:

Configuration	Description
Append files in folder with defined folder path and file prefix.	Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1.
Append files in folder without defined folder path and file prefix.	Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1.
Overwrite files in folder with defined folder path and file prefix.	Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.	Validation will fail. Folder path and file prefix must be supplied for this load strategy.

Folder Path · string

The folder path of the written files.

File Prefix · string

A string of characters to include at the beginning of the written files. Often used for organizing database objects.

Storage · drop-down · required

A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Click the tab that corresponds to your chosen cloud storage service.

Amazon S3

Amazon S3 Bucket · drop-down · required

An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Azure Storage

Storage Account · drop-down · required

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container · drop-down · required

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

Google Cloud Storage

GCS Bucket · drop-down · required

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite · boolean · required

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Databricks

Destination · drop-down · required

• Databricks: Load your data into Databricks. You’ll need to set a cloud storage location for temporary staging of the data.
• Cloud Storage: Load your data directly into your preferred cloud storage location.

Click either the Databricks or Cloud Storage tab on this page for documentation applicable to that destination type.

Databricks

Catalog · drop-down · required

Select a Databricks Unity Catalog. The special value [Environment Default] uses the catalog defined in the environment. Selecting a catalog will determine which databases are available in the next parameter.

Schema · drop-down · required

The Databricks schema. The special value [Environment Default] uses the schema defined in the environment. Read Create and manage schemas to learn more.

Table Name · string · required

The name of the table to be created.

Load Strategy · drop-down · required

• Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
• Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
• Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
• Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.

Column Mapping · column editor

In the tree structure representing the source data, select the checkboxes next to the fields that you want to output from this connector, then click Save.

Unselected fields will not be included in the output from this connector.

For each field you select, you can click the Edit Alias icon to change the name used for this field in all downstream components in this pipeline.

Outer Join · boolean

Determines what happens to a row when a mapped column comes from a field nested inside an array that’s empty or missing for that row. This property only appears once your Column Mapping includes a field nested inside an array.

• Yes: Keep the row, and populate the mapped column with NULL. This is the default setting.
• No: Omit the row from the output entirely.

Primary Keys · object selector

Select one or more columns to use as the table’s primary key. If you select multiple columns, the component creates a composite primary key using the combination of the selected columns.

Leave this blank to create the table without a primary key.

Clean Staged Files · boolean · required

• Yes: Staged files will be destroyed after data is loaded. This is the default setting.
• No: Staged files are retained in the staging area after data is loaded.

Stage Platform · drop-down · required

Choose a data staging platform using the drop-down menu.

• Amazon S3: Stage your data on an AWS S3 bucket.
• Azure Storage: Stage your data in an Azure Blob Storage container.

Click one of the tabs below for documentation applicable to that staging platform.

Amazon S3

Amazon S3 Bucket · drop-down · required

An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Azure Storage

Storage Account · drop-down · required

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container · drop-down · required

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

Cloud Storage

Load Strategy · drop-down

• Append Files in Folder: Appends files to storage folder. This is the default setting.
• Overwrite Files in Folder: Overwrite existing files with matching structure.

See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:

Configuration	Description
Append files in folder with defined folder path and file prefix.	Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1.
Append files in folder without defined folder path and file prefix.	Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1.
Overwrite files in folder with defined folder path and file prefix.	Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.	Validation will fail. Folder path and file prefix must be supplied for this load strategy.

Folder Path · string

The folder path of the written files.

File Prefix · string

A string of characters to include at the beginning of the written files. Often used for organizing database objects.

Storage · drop-down · required

A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Click the tab that corresponds to your chosen cloud storage service.

Amazon S3

Amazon S3 Bucket · drop-down · required

An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Azure Storage

Storage Account · drop-down · required

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container · drop-down · required

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

Google Cloud Storage

GCS Bucket · drop-down · required

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite · boolean · required

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Amazon Redshift

Destination · drop-down · required

• Redshift: Load your data into Amazon Redshift. You’ll need to set a cloud storage location for temporary staging of the data.
• Cloud Storage: Load your data directly into your preferred cloud storage location.

Click either the Amazon Redshift or Cloud Storage tab on this page for documentation applicable to that destination type.

Amazon Redshift

Schema · drop-down · required

The Amazon Redshift schema. The special value [Environment Default] uses the schema defined in the environment. Read Schemas to learn more.

For more information on using multiple schemas, read Schemas.

Table Name · string · required

The name of the table to be created.

Load Strategy · drop-down · required

• Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
• Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
• Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
• Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.

Clean Staged Files · boolean · required

• Yes: Staged files will be destroyed after data is loaded. This is the default setting.
• No: Staged files are retained in the staging area after data is loaded.

Amazon S3 Bucket · drop-down · required

An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Cloud Storage

Load Strategy · drop-down

• Append Files in Folder: Appends files to storage folder. This is the default setting.
• Overwrite Files in Folder: Overwrite existing files with matching structure.

See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:

Configuration	Description
Append files in folder with defined folder path and file prefix.	Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1.
Append files in folder without defined folder path and file prefix.	Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1.
Overwrite files in folder with defined folder path and file prefix.	Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.	Validation will fail. Folder path and file prefix must be supplied for this load strategy.

Folder Path · string

The folder path of the written files.

File Prefix · string

A string of characters to include at the beginning of the written files. Often used for organizing database objects.

Storage · drop-down · required

A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Click the tab that corresponds to your chosen cloud storage service.

Amazon S3

Amazon S3 Bucket · drop-down · required

An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Azure Storage

Storage Account · drop-down · required

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container · drop-down · required

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

Google Cloud Storage

GCS Bucket · drop-down · required

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite · boolean · required

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Google BigQuery

Destination · drop-down · required

Select the destination for your data:

• Google BigQuery: Load your data into a table in Google BigQuery.
• Cloud Storage: Load your data directly into files in your preferred cloud storage location. The format of these files can differ between source systems and will not have a file extension. Check the output to determine the format of the data.

Google BigQuery

Project · drop-down · required

Select the Google BigQuery project to load data into. The special value [Environment Default] uses the project defined in the environment.

Dataset · drop-down · required

Select the Google BigQuery dataset to load data into. The special value [Environment Default] uses the dataset defined in the environment.

Table Name · string · required

The name of the table to be created in your Google BigQuery project. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.

Load Strategy · drop-down · required

• Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
• Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
• Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
• Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.

Column Mapping · column editor

In the tree structure representing the source data, select the checkboxes next to the fields that you want to output from this connector, then click Save.

Unselected fields will not be included in the output from this connector.

For each field you select, you can click the Edit Alias icon to change the name used for this field in all downstream components in this pipeline.

Primary Keys · object selector

Select one or more columns to use as the table’s primary key. If you select multiple columns, the component creates a composite primary key using the combination of the selected columns.

Leave this blank to create the table without a primary key.

Clean Staged Files · boolean · required

• Yes: Staged files will be destroyed after data is loaded. This is the default setting.
• No: Staged files are retained in the staging area after data is loaded.

Partition Type · drop-down

Select the time period to use to partition the data loaded into your table, for example Day or Month.

GCS Bucket · drop-down · required

Select the Google Cloud Storage (GCS) bucket linked to your Google Cloud Platform account.

Overwrite · boolean · required

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Cloud Storage

Load Strategy · drop-down

• Append Files in Folder: Appends files to storage folder. This is the default setting.
• Overwrite Files in Folder: Overwrite existing files with matching structure.

Folder Path · string

The folder path for the files to be written to. Note that this path follows, but does not include, the bucket or container name.

File Prefix · string

A string of characters that precedes the name of the written files. This can be useful for organizing database objects.

Storage · drop-down · required

A cloud storage location to load your data into files for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Advanced Settings

Log Level · drop-down

Select the severity level of logging. Choose from Error, Warn, Info, Debug, or Trace. The default setting is Error.

Logs can be found in the Logging tab in when viewing details about a task after the pipeline has been run.

Load Selected Data · boolean

Choose whether to return the entire payload or only selected data objects. Read Response to learn how to select the data objects to include in your API response.

• No: (Default) Return the entire payload.
• Yes: Only return the objects that have been Set as selected data when configuring the response structure in the custom connector editor.

Deactivate soft delete for Azure blobs (Databricks)

If you intend to set your destination as Databricks and your stage platform as Azure Storage, you must turn off the “Enable soft delete for blobs” setting in your Azure account for your pipeline to run successfully. To do this:

1. In the Azure portal, navigate to your storage account.
2. In the menu, under Data management, click Data protection.
3. Clear the Enable soft delete for blobs checkbox. For more information, read Soft delete for blobs.