> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Pipelines
export const m_runner = "Maia runner";
export const maia = "Maia";
export const designer = "Designer";
Pipelines are {maia}'s way of designing, organizing, and executing workflows. You use {designer} to build strings of configured [components](/docs/guides/components-overview) inside a pipeline and then run that pipeline to carry out a desired task such as loading or transforming data.
Pipelines are created, configured, and managed through the main [{designer} user interface](/docs/guides/using-designer).
There are two types of pipeline in {maia}: orchestration and transformation.
* Orchestration pipelines deal with the loading of data from source system to target data warehouse. Typical orchestration components are connectors, flow logic components, and scripting components.
* Transformation pipelines deal with transforming table data that exists in your target data warehouse, typically after loading that data with an orchestration pipeline. Transformation components are often analogs of SQL operations such as creating and deleting tables, joining data, or performing calculations.
Components are the basic building blocks of pipelines. Each component is specifically applicable to one type of pipeline, orchestration or transformation, and can't be added to the other type of pipeline. To learn more about components, read [Components overview](/docs/guides/components-overview).
Any pipeline can be saved as a [shared pipeline](/docs/guides/shared-pipelines), which can then be referenced by any other pipeline in any project in your {maia}. This feature helps maintain consistency of core functionality over multiple different projects across the organization.
Our [Pipelines tutorial](/docs/guides/pipelines-tutorial) is a step-by-step walkthrough that explains how to create an orchestration pipeline and a transformation pipeline using sample data provided by Matillion. This guide may be useful if you are new to Designer and want to practice creating pipelines and configuring components.
***
## Adding a pipeline
To create a new pipeline:
1. In the [{designer} user interface](/docs/guides/using-designer), click **Files** at the top of the canvas.
2. Click the **+** icon in the top right of the **Files** panel, and select either **Orchestration pipeline** or **Transformation pipeline**.
3. In **Name**, enter a name for the pipeline. The name may include alphanumerics, underscores, single spaces, parentheses, and hyphens. The name must be unique within each pipeline [folder](#pipeline-folders).
4. In the **Description** field, you can optionally enter a description for the pipeline. This is useful if you want to share or export this pipeline, as it provides context for other users. Max 1000 characters.
5. Click **Add**. The new pipeline immediately opens in a new tab on the canvas.
This will place the pipeline at the top level of your pipeline folder tree. To create a pipeline within a folder, read [Pipeline folders](#pipeline-folders), below.
The newly created pipeline is blank at this stage. You now need to add [components](/docs/guides/components-overview) to the pipeline to construct a workflow that will perform your data extraction and transformation tasks.
When you start using {designer}, you can also use the **Orchestration pipeline** or **Transformation pipeline** tiles to create your first pipelines, or click the **Watch video** links to see a short video tutorial on pipeline creation.
***
## Managing pipelines
In the [{designer} user interface](/docs/guides/using-designer), click **Files** to view your pipelines. The **Files** panel lists every pipeline you have created on your current branch. Icons identify the type of pipeline—a blue `O` for orchestration, or green `T` for transformation. You can use [folders](#pipeline-folders) to organize your pipelines in a hierarchical structure.
Hover over a pipeline in the **Files** panel and click the three dots **...** next to its name to open a list of actions you can perform on the selected pipeline. The table below lists the available actions.
| Action | Description |
| ------------ | --------------------------------------------------------------------------------------------------------------- |
| Open | Open this pipeline on the {designer} canvas. You can also do this by clicking the name of the pipeline. |
| Copy | Copy this pipeline. Read [Copying pipelines](#copying-pipelines). |
| Edit details | Change this pipeline's name and description. Read [Renaming pipelines](#renaming-pipelines). |
| Move | Move this pipeline to a different folder. |
| Delete | Delete this pipeline. Read [Deleting pipelines](#deleting-pipelines). |
| Share | Turn this pipeline into a shared pipeline. Read [Shared pipelines](/docs/guides/shared-pipelines). |
| Export | Export this pipeline. Read [Importing and exporting pipelines](#importing-and-exporting-pipelines). |
| Add test | Create a test for this pipeline. Read [Test pipelines](/docs/guides/test-pipelines). |
| Review | Review this pipeline according to your rules. Read [Reviewing pipeline quality](/docs/guides/pipeline-quality). |
### Copying pipelines
To copy and paste a pipeline:
1. Click **Copy** in the actions list, or by clicking the pipeline and pressing `ctrl + c` (Windows) or `cmd + c` (Mac).
2. Paste the copy into a folder using one of the following options:
* Hover over the folder, click the three dots **...** next to its name, and click **Paste**.
* Click the name of the folder in the **Files** panel, then press `ctrl + v` (Windows) or `cmd + v` (Mac) to paste it.
If you make a copy of a pipeline in the same folder as the original, the new pipeline will be called `_copy_your-pipeline-name`. Any additional copies will be called `_copy_your-pipeline-name_2`, `..._3`, and so on.
### Renaming pipelines
Renaming a pipeline will break any [schedules](/docs/guides/schedules), API calls, and stored variables that reference the renamed pipeline's previous name.
{maia} can only update references to this pipeline in other pipelines in your project. You'll need to update all other references yourself.
To edit a pipeline's name or description:
1. Click **Edit details** in the actions list.
2. In the **Pipeline name** and **Pipeline description** fields, change the pipeline's name and/or description as required.
3. Choose how to update references to this pipeline in other pipelines:
* **Update references automatically:** {maia} will find and update any references to this pipeline in other pipelines in your project, such as [Run Orchestration](/docs/components/run-orchestration) or [Run Transformation](/docs/components/run-transformation) components.
* **Rename this pipeline only:** Do not update references to this pipeline in other pipelines. You'll need to update any references to this pipeline yourself.
4. Click **Update**.
### Deleting pipelines
Deleted pipelines *cannot* be recovered.
Deleting a pipeline will break any pipelines that contain [Run Orchestration](/docs/components/run-orchestration) or [Run Transformation](/docs/components/run-transformation) components that run the deleted pipeline.
To delete a pipeline:
1. Click **Delete** in the actions list.
2. In the **Delete pipeline** dialog, click **Delete** to confirm.
### Cancelling a running pipeline
You can cancel a pipeline while it's running in one of two ways. A **soft** cancellation will allow the current task to complete and then no other pipeline tasks will run. A **forced** cancellation will interrupt and terminate the current task, set the pipeline status to "Stopped", and will prevent the pipeline from receiving further information from the {m_runner}.
Click the **Task history** tab, and then click the "X" icon on the far-right of the task you wish to cancel. A single click will perform a **soft** cancellation. Clicking twice will perform a **forced** cancellation. A pop-up dialog will appear asking you to confirm the force cancel. Click, **Yes, force cancel** to terminate the pipeline, without waiting for any in-progress steps to finish.
You can also cancel pipeline runs from the [Pipeline run history](/docs/guides/pipeline-run-history#cancel-pipeline-runs).
***
## Pipeline folders
A branch may contain a large number of pipelines, which by default are listed alphabetically in the **Files** panel. To organize pipelines in a branch, you can arrange them into named folders. Folders can be nested inside folders, up to 10 levels deep, to create a structure that makes sense to you and your team. Pipeline names must be unique within a folder but can be duplicated in different folders.
To create a top-level folder, click **Files** at the top of the canvas, then click the **+** icon in the **Files** panel and click **New folder**. Enter a name for the folder and click **Create**. Folder names can contain alphanumeric characters, dashes, and underscores.
You can collapse or expand a folder by clicking the **>** icon.
Each folder has its own context menu where you can add a pipeline to the folder, add another folder as a subfolder, rename the folder, or delete the folder. Click the three dots **...** next to the folder name to open the context menu.
When you delete a folder, all the pipelines and subfolders it contains will also be deleted. However, active schedules for pipelines in a deleted folder *will* continue to use the latest published version of the pipelines. Deleting a folder can't be undone.
You can move a pipeline between folders, or between root and folder, by drag-and-drop. You can't move a pipeline to a folder where another pipeline with the same name already exists, and attempting to do so will result in a warning notice.
Be aware of the following when working with pipeline folders:
* Moving a pipeline into a different folder will break other pipelines that reference it via [Run Orchestration](/docs/components/run-orchestration) or [Run Transformation](/docs/components/run-transformation) components. You will therefore need to update any pipelines that reference the moved pipeline.
* Renaming a folder will also break pipelines that reference a pipeline contained in the folder you have renamed. You will need to update any pipelines that reference pipelines in the renamed folder.
* Any existing [schedules](/docs/guides/schedules) that use the moved pipeline will not be updated. The pipelines will have to be republished and the schedule recreated.
* Empty folders cannot be [committed to Git](/docs/guides/git-commit). These remain on your local branch, but aren't committed until they contain at least one file.
* Moving a pipeline to a different folder is considered a change to your branch, and therefore needs to be [committed to Git](/docs/guides/git-commit).
### Copying pipeline folders
To make a copy of a pipeline folder and all of its contents, including pipelines and subfolders, do one of the following:
* Click the pipeline folder to select it, then press `ctrl + c` (Windows) or `cmd + c` (Mac) to copy it.
* Hover over the pipeline folder, click the three dots **...** that appear next to it, and click **Copy**.
Then do one of the following:
* Click the folder in which you want to paste a copy of the pipeline folder, then press `ctrl + v` (Windows) or `cmd + v` (Mac) to paste it.
* Hover over the folder in which you want to paste a copy of the pipeline folder, click the three dots **...** that appear next to it, and click **Paste**.
If you make a copy of a pipeline folder in the same place as the original folder, the new pipeline folder will be called `_copy_your-folder-name`. Any additional copies will be called `_copy_your-folder-name_2`, `..._3`, and so on.
The names of any pipelines and subfolders in the copied folder are not changed.
***
## Importing and exporting pipelines
Pipelines can be exported and imported to essentially copy them between projects. **Export** copies the pipeline definition to a zipped data file in your local filesystem. You can then use the **Import** function to import that data into another project. You can also export/import an entire pipeline folder, including all the pipelines and subfolders it contains, or export/import the entire project including all folders and pipelines within it.
There is a limit on the size of the pipeline you want to import:
* Maximum compressed size: Fails if import exceeds 8 MB.
* Maximum decompressed size: Fails if import exceeds 100 MB.
* Maximum file size: Fails if any file exceeds 5 MB.
* Maximum files: Fails if import exceeds 2500 files.
### Exporting a pipeline
When you export a pipeline, folder, or project, it is exported as a `.zip` file containing all the exported pipeline definitions in `.yaml` format. If you export a folder or project, the `.zip` file preserves the structure of the files and folders it contains. The file is exported to your local default download location. The name of the `.zip` file is in the format `___T.zip`, which creates a unique name for each export.
The `.zip` file contains everything required to recreate the exported pipelines in a new project, including component property values, pipeline variables, component connections, and canvas notes. It does *not* include project-level configurations such as schedules, secrets, OAuths, and project variables. Make sure that you configure these in the target project before running the imported pipeline.
The export captures the current state of all exported pipelines as they exist in {designer} at the time they are exported. Since {designer} automatically saves your pipelines while you work, this means that the export captures all changes made to exported pipelines, including uncommitted changes.
To export a pipeline, folder, or project, hover over its name in the **Files** panel, then click the three dots **...** that appear next to it, and then click **Export**.
### Importing a pipeline
A pipeline can only be imported if the importing user has the same [role permissions](/docs/administration/project-roles) as the user who exported it.
1. In the **Files** panel, hover over the folder that you want to import into, click the three dots **...** that appear next to it, and click **Import**.
2. Browse your filesystem to locate the exported `.zip` file that you want to import, select the file, and click **Open**.
The pipelines contained in the `.zip` file will be imported to the selected pipeline folder. If you are importing a pipeline folder or an entire project, it will form a subfolder of the folder you are importing into.
You cannot import a `.zip` file into a folder if those files already exist in the folder you are importing into. If you try this, you will receive an error stating `Import unsuccessful: The files you tried to import already exist`.
### Importing a pipeline from the Matillion Exchange
Pre-built {maia} pipelines are available from [Matillion Exchange](https://exchange.matillion.com/data-productivity-cloud/pipeline/). To import a pipeline or set of pipelines from Matillion Exchange into {maia}, follow these steps:
1. In the **Files** panel, click **Add**, then click **Browse Exchange**.
2. In the **Browse Exchange** panel that opens on the right of the canvas, search for and click the pipeline or set of pipelines that you want to import.
Click **Learn more** on one of the pipeline tiles to open Matillion Exchange and find more details about the corresponding pipeline or set of pipelines.
Your chosen pipelines will be added to a new folder called **Imported from Exchange**. Each pipeline or set of pipelines that you import from Matillion Exchange will be added to a subfolder inside this one.