Components overview

Components are the basic building blocks of pipelines. Each component represents a single task within the pipeline. For example, the Create Table component executes a task that creates a table when the pipeline runs, while the Salesforce Query component executes a task that returns data from Salesforce. Multiple components can be linked together to perform complex pipelines involving multiple tasks. For example, a pipeline may pull data from Salesforce, filter the data according to some criteria, and write the data to files in an S3 bucket, using the Salesforce Query, Filter, and S3 Unload components executed sequentially.

Each component is specifically applicable to one type of pipeline, orchestration or transformation, and you can’t add it to the other type of pipeline. However, see the Run Transformation component for a way to include a transformation pipeline (and its components) in an orchestration pipeline.

Due to considerations such as security and data flow, some components can only be used in a Hybrid SaaS environment. This is clearly stated in the component description. Read Matillion Full SaaS vs Hybrid SaaS for more details.

---

View components

To view the components that make up a pipeline, click Files at the top of the canvas, then click the pipeline you want to open. This displays the full pipeline on the canvas. Each component is represented by a separate icon on the canvas, and the connections between the component icons show the pipeline flow—the order in which components are executed.

When you have a pipeline open, you can view the components available in that pipeline by clicking the Add component icon on the canvas.

---

Add components to a pipeline

To add a component to a pipeline, open the pipeline in the canvas and use any of the following methods to open the Component dialog, where you can search for and click the component you want to add:

• Click the Add component icon on the left of the canvas or the + icon on an empty canvas. The new component will appear on the canvas and will not be connected to any existing components.
• Click the + icon to the right of a component already on the canvas. When using this method, you can also select the Output Connection that is used to connect the previous component to the new component (On success, Unconditional, or On failure).
• Click on an existing connection line linking two components. The new component will be placed directly between the two existing components.

All new components are given a default name when placed on the canvas, which can be changed in the component’s properties.

Search and filter components

The Add component dialog only shows the components that can be used in the type of pipeline currently open. For example, Create Table is an orchestration component that can only be added to orchestration pipelines. As a result, it is only shown in the Add component dialog when you have an orchestration pipeline open.

Use the Search box in the Add component dialog to search for a component by name or function. For example, the Send Email component will appear if you search for messaging. The search will also attempt to match based on synonyms—for example, searching for “logic” will find the If component.

Click one or more of the category filters under the Search field to show only the selected types of component.

Add warehouse data to a pipeline

To bring existing warehouse data into a transformation pipeline, you can also click the Add component icon and select the Warehouse data tab. By default, this tab shows all tables and views in your environment’s default database—use the drop-down in the top left of the tab to select a different database, and search for schemas within the selected database.

Select a table or view to see its metadata and sample the data it contains. Once you’ve selected the table or view you want to use in your transformation pipeline, click Use to create a Table Input component in your pipeline containing the selected data.

Component context menu

Right-click a component to display the component context menu, where you can perform the actions listed below.

Action	Description	Windows shortcut	Mac shortcut
Validate pipeline	Check that your pipeline is configured correctly without performing any data manipulation.	SHIFT + CTRL + ENTER	SHIFT + CMD + RETURN
Run pipeline	Run the pipeline.	CTRL + ENTER	CMD + RETURN
Run only component	Orchestration pipelines only. Run this component without running any other components in the pipeline.
Run from component	Orchestration pipelines only. Run this component, and everything after it in the pipeline. No components before this one will run.
Add note	Add a text note to the canvas to document your pipeline.
Add note using	Tell to add a pipeline note to the canvas.
Copy	Make a copy of the component for pasting elsewhere.	CTRL + C	CMD + C
Delete component	Remove the component from the pipeline canvas.	DELETE	DELETE

For more information about Windows and Mac shortcuts, read Keyboard shortcuts in Windows and Mac keyboard shortcuts, respectively.

Component action bar

Click a component to display the component action bar, where you can perform the actions in the table below. Some actions are only available in orchestration pipelines.

If you select multiple components, the Copy, Skip on run, and Delete actions are available in the action bar.

Action	Description
Run only component	Orchestration pipelines only. Run this component without running any other components in the pipeline.
Run from component	Orchestration pipelines only. Run this component, and everything after it in the pipeline. No components before this one will run.
Skip on run	Orchestration pipelines only. Temporarily disable this component. For more information, read Skip components.
Copy	Copy this component. For more information, read Copy components.
Iterate	Orchestration pipelines only. Add an iterator to this component. For more information, read Iterate components.
Delete	Delete this component.

---

Copy components

You can copy a component to create a duplicate elsewhere in the pipeline. Use the component’s context menu or the component action bar to copy the component, and then use the context menu to paste a copy of the component, including all configured properties, elsewhere on the canvas. Connect the copy into the pipeline as required.

You can also copy a component between pipelines. Use the component’s context menu to copy the component, switch to a different pipeline, and use the context menu to paste the component onto the new pipeline. The copied component is identical to the original, including all configured properties. The two pipelines must be of the same type; you can’t paste from an orchestration pipeline to a transformation pipeline, or vice versa.

Select multiple components by holding down the SHIFT key while you click all the components you want to select, or hold down SHIFT while you drag the mouse pointer over the components you want to select. You can then copy all selected components as a single operation. To deselect a single component, hold the SHIFT key and click the component. To deselect all components, click on any empty part of the canvas.

---

Connect components

Components must be connected in the order they’ll execute in the pipeline. Each component is surrounded by connection rings: usually (but not always) one on the left and three on the right. To link two components, hover over a connection ring on the right of one component until the pointer changes to a + symbol. Then click and drag a line to the connection ring on the left of the second component. This will draw a permanent connection between the components.

To disconnect components, first click the line that joins them. Then, either click Delete in the action bar, or press Backspace or Delete on your keyboard.

The order in which components are joined is crucial to the order in which tasks are executed in the pipeline. The flow of pipeline execution proceeds from left to right, so the connection rings at the left of a component represent the input to the component, passed from the previous component, while the rings on the right represent the output of the component, which is passed to the next component.

To take an example of a Salesforce Query component taking data from Salesforce and writing it to an S3 bucket with S3 Unload, the pipeline starts with the Salesforce Query, so we place that on the left side of the canvas. One of the output rings on the right of that component should then be connected to the input ring at the left of the S3 Unload component, which naturally should be placed at the right side of the canvas for convenience. This tells the pipeline which order these components are to be executed in, and that the data generated by the first component should be made available for the second component to work with.

You can create complex, branching pipelines by linking two or more of a component’s outputs to two or more other components.

The connection rings on the right of a component are color-coded, to denote the following specific functions:

• Green: Continue on to the next component if this component executed successfully.
• Red: Continue on to the next component if this component failed to execute.
• Gray: Continue on to the next component regardless of the success or failure of this component.

These connection options allow you to direct the pipeline flow according to whether the component task is successful or not, creating different branches for each outcome.

Once two components are connected, a connecting line is visible, which will match the color-coding of the connection type. Most connection lines will also include an icon for added clarity of the connection type:

• Gray: Continue on to the next component regardless of the success or failure of this component.
• Green “thumb up”: Continue on to the next component if this component executed successfully.
• Red “X”: Continue on to the next component if this component failed to execute.
• Blue “thumb up”: Continue on to the next component if this component’s condition resolves to true.
• Orange “thumb down”: Continue on to the next component if this component’s condition resolves to false.

---

Skip components

Use the Skip on run option to temporarily disable a component in your pipeline. This allows you to run the pipeline as if it did not contain the skipped component, without actually needing to remove the component from the pipeline.

To skip a component, click the component and then click Skip on run in the component action bar. The component on the pipeline canvas will appear darker, and the Skip on run icon in the action bar will change to show that this component will be skipped.

Additionally, when you run a pipeline containing a skipped component and then open the detailed task view, the skipped component’s success icon is grayed out and its Message notes that this component is skipped.

To un-skip a component, click the component and click Unskip on run. The appearance of the component and icon will return to normal.

You can only skip components in orchestration pipelines, and you can’t skip Flow components such as If.

---

Iterate on components

Use iterators to loop a component—repeating it without adding multiple copies to your pipeline.

There are two ways to attach an iterator to a component:

• Click the component and then click Iterate in the component action bar. Next, select the iterator you want to use. The iterator will automatically be attached to the selected component on the canvas.
• Search for and click an iterator component using the Add component option. Next, use the connection ring beneath the iterator to connect it to the input of the component you want to iterate on.

The following iterators are available in :

• Use the File Iterator to loop over matching files in a remote file system.
• Use the Fixed Iterator to loop over values of a simple sequence.
• Use the Grid Iterator to loop over the rows of a grid variable.
• Use the Loop Iterator to loop over values of a simple sequence using a for loop.
• Use the Table Iterator to loop over rows of data within a table.

For information about attaching, stacking, and detaching multiple iterators, read Attaching, stacking, and detaching iterators.

---

Delete components

To delete a component from the canvas, you can:

• Select the component, then click Delete in the component action bar.
• Right-click the component and click Delete on the context menu.
• Select the component and press Delete on your keyboard.

---

Component properties

Every component has properties that can be configured to tailor the component’s behavior. When a component is selected on the canvas, its properties are displayed in the Properties panel on the right of the canvas.

Each component’s properties are described in its reference page in the Components documentation. Click the Help icon in the top right of a component’s Properties panel to view its documentation.

Configuring a component’s properties doesn’t affect any other components, even components of the same type.

Each component’s name must be unique within a pipeline and cannot be left blank.

When configuring a component with properties in sections, such as a connector, click the header of each section to expand or collapse it. You can also toggle on Expand all to display all available properties. This toggle is off by default.

Preview SQL

Most transformation components produce SQL statements that act on your data. Once a transformation component has been validated, you can toggle Preview SQL on at the top of the Properties panel to see the raw SQL of the selected component in a read-only editor.

Variables in properties

Many properties allow you to use variables or grid variables to set them dynamically. For example, any connectors that output to file storage can have the storage folder path set using variables, as described in Organizing file storage.

Reorder items

Many property dialogs allow you to reorder rows of information. For example, when you configure the Columns property of the Create Table component, you can reorder the columns. To do this in any applicable editor, use the ⋮⋮ icon to drag and drop a single row to change its position, or use the checkboxes to select multiple rows and then drag all selected rows into the required order.

Reordering rows can have a significant effect on your data. For example, it may change the order in which your data is displayed and processed, or the order in which filter conditions are applied.

Text mode

Text mode is a feature that lets you rapidly add information to a property dialog. Instead of typing or selecting values in the individual fields of the dialog, you can click the Text mode toggle to open a multi-line editor that lets you add all the items in a single block, one item to a line, with the “fields” separated by tabs.

For example, when you complete the Columns property of the Create Table component, you are required to specify Column Name, Data Type, Size, and so on, for every column in the table. In text mode you can type the following:

Name    Text    64  0   Yes No
Descr   Text    512 0   No  No
Cost    Number  5   0   Yes Yes

Once you have entered details in text mode, you must switch out of text mode by clicking the Text mode toggle once more, to verify the details are valid before you continue. For example, if you mistyped a data type as Numbor, the Data Type field in the dialog would reject the invalid type and remain blank. You must correct the type, either directly in the field or in text mode again, before you click Save in the dialog.

You can go into text mode to edit the fields of an existing dialog, regardless of whether it was originally completed in text mode or not.

Text mode also lets you paste values from other text sources, as long as the values match the columns in the dialog and are separated by tabs and new lines. You can also rapidly copy the fields from a completed dialog to another identical dialog, allowing rapid duplication of properties between or within components:

1. Open a completed dialog in text mode.
2. Copy the text containing the column definitions.
3. Open another dialog, either of the same type or having identical field layout.
4. Paste the copied text into the text mode field of the new dialog.