Pipelines

Pipelines are ‘s way of designing, organizing, and executing workflows. You use to build strings of configured components inside a pipeline and then run that pipeline to carry out a desired task such as loading or transforming data.

Pipelines are created, configured, and managed through the main user interface.

There are two types of pipeline in : orchestration and transformation.

• Orchestration pipelines deal with the loading of data from source system to target data warehouse. Typical orchestration components are connectors, flow logic components, and scripting components.
• Transformation pipelines deal with transforming table data that exists in your target data warehouse, typically after loading that data with an orchestration pipeline. Transformation components are often analogs of SQL operations such as creating and deleting tables, joining data, or performing calculations.

Components are the basic building blocks of pipelines. Each component is specifically applicable to one type of pipeline, orchestration or transformation, and can’t be added to the other type of pipeline. To learn more about components, read Components overview.

Any pipeline can be saved as a shared pipeline, which can then be referenced by any other pipeline in any project in your . This feature helps maintain consistency of core functionality over multiple different projects across the organization.

Our Pipelines tutorial is a step-by-step walkthrough that explains how to create an orchestration pipeline and a transformation pipeline using sample data provided by Matillion. This guide may be useful if you are new to Designer and want to practice creating pipelines and configuring components.

---

Adding a pipeline

To create a new pipeline:

1. In the user interface, click Files at the top of the canvas.
2. Click the + icon in the top right of the Files panel, and select either Orchestration pipeline or Transformation pipeline.
3. In Name, enter a name for the pipeline. The name may include alphanumerics, underscores, single spaces, parentheses, and hyphens. The name must be unique within each pipeline folder.
4. In the Description field, you can optionally enter a description for the pipeline. This is useful if you want to share or export this pipeline, as it provides context for other users. Max 1000 characters.
5. Click Add. The new pipeline immediately opens in a new tab on the canvas.

This will place the pipeline at the top level of your pipeline folder tree. To create a pipeline within a folder, read Pipeline folders, below.

The newly created pipeline is blank at this stage. You now need to add components to the pipeline to construct a workflow that will perform your data extraction and transformation tasks.

When you start using , you can also use the Orchestration pipeline or Transformation pipeline tiles to create your first pipelines, or click the Watch video links to see a short video tutorial on pipeline creation.

---

Managing pipelines

In the user interface, click Files to view your pipelines. The Files panel lists every pipeline you have created on your current branch. Icons identify the type of pipeline—a blue O for orchestration, or green T for transformation. You can use folders to organize your pipelines in a hierarchical structure.

Hover over a pipeline in the Files panel and click the three dots … next to its name to open a list of actions you can perform on the selected pipeline. The table below lists the available actions.

Action	Description
Open	Open this pipeline on the canvas. You can also do this by clicking the name of the pipeline.
Copy	Copy this pipeline. Read Copying pipelines.
Edit details	Change this pipeline’s name and description. Read Renaming pipelines.
Move	Move this pipeline to a different folder.
Delete	Delete this pipeline. Read Deleting pipelines.
Share	Turn this pipeline into a shared pipeline. Read Shared pipelines.
Export	Export this pipeline. Read Importing and exporting pipelines.
Add test	Create a test for this pipeline. Read Test pipelines.
Review	Review this pipeline according to your rules. Read Reviewing pipeline quality.

Copying pipelines

To copy and paste a pipeline:

1. Click Copy in the actions list, or by clicking the pipeline and pressing ctrl + c (Windows) or cmd + c (Mac).
2. Paste the copy into a folder using one of the following options:
   • Hover over the folder, click the three dots … next to its name, and click Paste.
   • Click the name of the folder in the Files panel, then press ctrl + v (Windows) or cmd + v (Mac) to paste it.

If you make a copy of a pipeline in the same folder as the original, the new pipeline will be called _copy_your-pipeline-name. Any additional copies will be called _copy_your-pipeline-name_2, ..._3, and so on.

Renaming pipelines

Renaming a pipeline will break any schedules, API calls, and stored variables that reference the renamed pipeline’s previous name.

can only update references to this pipeline in other pipelines in your project. You’ll need to update all other references yourself.

To edit a pipeline’s name or description:

1. Click Edit details in the actions list.
2. In the Pipeline name and Pipeline description fields, change the pipeline’s name and/or description as required.
3. Choose how to update references to this pipeline in other pipelines:
   • Update references automatically: will find and update any references to this pipeline in other pipelines in your project, such as Run Orchestration or Run Transformation components.
   • Rename this pipeline only: Do not update references to this pipeline in other pipelines. You’ll need to update any references to this pipeline yourself.
4. Click Update.

Deleting pipelines

Deleted pipelines cannot be recovered.

Deleting a pipeline will break any pipelines that contain Run Orchestration or Run Transformation components that run the deleted pipeline.

To delete a pipeline:

1. Click Delete in the actions list.
2. In the Delete pipeline dialog, click Delete to confirm.

Cancelling a running pipeline

You can cancel a pipeline while it’s running in one of two ways. A soft cancellation will allow the current task to complete and then no other pipeline tasks will run. A forced cancellation will interrupt and terminate the current task, set the pipeline status to “Stopped”, and will prevent the pipeline from receiving further information from the .

Click the Task history tab, and then click the “X” icon on the far-right of the task you wish to cancel. A single click will perform a soft cancellation. Clicking twice will perform a forced cancellation. A pop-up dialog will appear asking you to confirm the force cancel. Click, Yes, force cancel to terminate the pipeline, without waiting for any in-progress steps to finish.

You can also cancel pipeline runs from the Pipeline run history.

---

Pipeline folders

A branch may contain a large number of pipelines, which by default are listed alphabetically in the Files panel. To organize pipelines in a branch, you can arrange them into named folders. Folders can be nested inside folders, up to 10 levels deep, to create a structure that makes sense to you and your team. Pipeline names must be unique within a folder but can be duplicated in different folders.

To create a top-level folder, click Files at the top of the canvas, then click the + icon in the Files panel and click New folder. Enter a name for the folder and click Create. Folder names can contain alphanumeric characters, dashes, and underscores.

You can collapse or expand a folder by clicking the > icon.

Each folder has its own context menu where you can add a pipeline to the folder, add another folder as a subfolder, rename the folder, or delete the folder. Click the three dots … next to the folder name to open the context menu.

When you delete a folder, all the pipelines and subfolders it contains will also be deleted. However, active schedules for pipelines in a deleted folder will continue to use the latest published version of the pipelines. Deleting a folder can’t be undone.

You can move a pipeline between folders, or between root and folder, by drag-and-drop. You can’t move a pipeline to a folder where another pipeline with the same name already exists, and attempting to do so will result in a warning notice.

Be aware of the following when working with pipeline folders:

• Moving a pipeline into a different folder will break other pipelines that reference it via Run Orchestration or Run Transformation components. You will therefore need to update any pipelines that reference the moved pipeline.
• Renaming a folder will also break pipelines that reference a pipeline contained in the folder you have renamed. You will need to update any pipelines that reference pipelines in the renamed folder.
• Any existing schedules that use the moved pipeline will not be updated. The pipelines will have to be republished and the schedule recreated.
• Empty folders cannot be committed to Git. These remain on your local branch, but aren’t committed until they contain at least one file.
• Moving a pipeline to a different folder is considered a change to your branch, and therefore needs to be committed to Git.

Copying pipeline folders

To make a copy of a pipeline folder and all of its contents, including pipelines and subfolders, do one of the following:

• Click the pipeline folder to select it, then press ctrl + c (Windows) or cmd + c (Mac) to copy it.
• Hover over the pipeline folder, click the three dots … that appear next to it, and click Copy.

Then do one of the following:

• Click the folder in which you want to paste a copy of the pipeline folder, then press ctrl + v (Windows) or cmd + v (Mac) to paste it.
• Hover over the folder in which you want to paste a copy of the pipeline folder, click the three dots … that appear next to it, and click Paste.

If you make a copy of a pipeline folder in the same place as the original folder, the new pipeline folder will be called _copy_your-folder-name. Any additional copies will be called _copy_your-folder-name_2, ..._3, and so on.

The names of any pipelines and subfolders in the copied folder are not changed.

---

Importing and exporting pipelines

Pipelines can be exported and imported to essentially copy them between projects. Export copies the pipeline definition to a zipped data file in your local filesystem. You can then use the Import function to import that data into another project. You can also export/import an entire pipeline folder, including all the pipelines and subfolders it contains, or export/import the entire project including all folders and pipelines within it.

There is a limit on the size of the pipeline you want to import:

• Maximum compressed size: Fails if import exceeds 8 MB.
• Maximum decompressed size: Fails if import exceeds 100 MB.
• Maximum file size: Fails if any file exceeds 5 MB.
• Maximum files: Fails if import exceeds 2500 files.

Exporting a pipeline

When you export a pipeline, folder, or project, it is exported as a .zip file containing all the exported pipeline definitions in .yaml format. If you export a folder or project, the .zip file preserves the structure of the files and folders it contains. The file is exported to your local default download location. The name of the .zip file is in the format <project name>_<branch>_<file/folder name>_<yyyymmdd>T<hhmmss>.zip, which creates a unique name for each export.

The .zip file contains everything required to recreate the exported pipelines in a new project, including component property values, pipeline variables, component connections, and canvas notes. It does not include project-level configurations such as schedules, secrets, OAuths, and project variables. Make sure that you configure these in the target project before running the imported pipeline.

The export captures the current state of all exported pipelines as they exist in at the time they are exported. Since automatically saves your pipelines while you work, this means that the export captures all changes made to exported pipelines, including uncommitted changes.

To export a pipeline, folder, or project, hover over its name in the Files panel, then click the three dots … that appear next to it, and then click Export.

Importing a pipeline

A pipeline can only be imported if the importing user has the same role permissions as the user who exported it.

1. In the Files panel, hover over the folder that you want to import into, click the three dots … that appear next to it, and click Import.
2. Browse your filesystem to locate the exported .zip file that you want to import, select the file, and click Open.

The pipelines contained in the .zip file will be imported to the selected pipeline folder. If you are importing a pipeline folder or an entire project, it will form a subfolder of the folder you are importing into.

You cannot import a .zip file into a folder if those files already exist in the folder you are importing into. If you try this, you will receive an error stating Import unsuccessful: The files you tried to import already exist.

Importing a pipeline from the Matillion Exchange

Pre-built pipelines are available from Matillion Exchange. To import a pipeline or set of pipelines from Matillion Exchange into , follow these steps:

1. In the Files panel, click Add, then click Browse Exchange.
2. In the Browse Exchange panel that opens on the right of the canvas, search for and click the pipeline or set of pipelines that you want to import.

Click Learn more on one of the pipeline tiles to open Matillion Exchange and find more details about the corresponding pipeline or set of pipelines.

Your chosen pipelines will be added to a new folder called Imported from Exchange. Each pipeline or set of pipelines that you import from Matillion Exchange will be added to a subfolder inside this one.