Test pipelines

A test is a pipeline that runs another pipeline and checks its results. Use tests to verify that your pipelines behave correctly before you deploy them to production. Any pipeline can have multiple tests associated with it, allowing you to check different functions and scenarios for the pipeline.

Tests are created on the canvas in the same way as an orchestration pipeline, and will appear as pipelines in the Files panel. Many components available to orchestration pipelines are also available in tests. In addition, a set of dedicated test components—such as assert and compare components—are available exclusively to tests. These components let you check the outputs of the pipeline to test. Available components are summarized below.

You can create tests manually as described on this page, but we recommend using to generate them automatically. Prompt with “Create a test for [pipeline name]”, and will ask you some questions about what you want to achieve, then generate a test with logical defaults and test data. You can edit the resulting test to customize it further.

---

Test panel

The Test panel lists all tests associated with a pipeline and lets you manage and run the tests.

To open the Test panel, click the Tests tab at the bottom of the pipeline canvas. Each test in the list has action icons that let you:

• Run the test.
• Edit the test (opens the test on the canvas).
• Delete the test.

To run all tests for the current pipeline at once, click Run all tests from the Test panel. Results are displayed in the panel once each test completes. Tests run sequentially.

To create a new test from the Test panel, click Create test. This opens the Add test dialog.

---

Creating a test

You can create a test from the Test panel of the pipeline you want to test, or from the Files panel.

From the Test panel

1. Open the pipeline you want to test.
2. Click the Tests tab at the bottom of the canvas.
3. Click Add a test. The Add test dialog opens.
4. Complete the following properties in the dialog:
   • Test name: A name for the test.
   • Target pipeline: This is pre-filled with the current pipeline and can’t be edited.
   • Folder: The folder in which to save the test. The default is the folder containing the current pipeline, but you can select a different folder from the drop-down.
   • Description: An optional description of the test. Limited to 1000 characters.
5. Click Add.

From a file in the Files panel

1. In the Files panel, locate the pipeline you want to test.
2. Click the … (more options) menu next to the pipeline, then click Add test. The Add test dialog opens.
3. Complete the dialog properties as described above. The Target pipeline field is pre-filled with the pipeline you selected.
4. Click Add.

From a folder in the Files panel

1. In the Files panel, click the … (more options) menu next to any folder name, then click Add → Test. The Add test dialog opens.
2. Complete the dialog properties as described above. For Target pipeline, use the drop-down to select a pipeline. If you have a pipeline open on the canvas, that will be the default selection, but you can select any other pipeline in the folder.
3. Click Add.

In each case, opens a new test on the canvas, pre-populated with a test template.

A test has an implicit link to the pipeline it’s testing, based on the pipeline named in the “Run Pipeline To Test” component.

For this reason, each test is stored by default in the same folder as the pipeline it tests, but you can select a different folder if you wish. For example, it may suit your folder plan for you to put all tests in a dedicated folder. The test will still be linked to the original target pipeline regardless of where you save it.

---

Test structure

When you create a test manually (not through ), loads a template containing a set of pre-configured components and pipeline variables. The template isn’t a complete, functioning test—it’s intended as a starting point for you to add components to set up your test data and assert the expected results.

The template establishes the following general structure:

1. Set up a test schema: Creates a temporary schema in your Cloud Data Warehouse to isolate the test from your production data.
2. Set up input data: Populates the test schema with the tables and data required by the pipeline to test.
3. Run the pipeline to test: Executes the pipeline you are testing.
4. Assert the results: Checks that the pipeline produced the expected output.
5. Tear down the test schema: Drops the test schema, removing all temporary data.

For example, if your pipeline simply creates a table, the test could create a test schema, run the pipeline, and then check that the table was created with the correct values. Once the test finishes, the test schema is dropped.

Test schema

The test schema is a temporary schema created in your Cloud Data Warehouse specifically for the test run. It allows you to run tests in isolation without affecting your production data.

The template uses the SQL Script component (labeled as Create Test Schema on the canvas) to create the test schema. It also creates some pre-configured pipeline variables—for example, test_database and test_schema—that reference this schema. Use these variables throughout your test wherever you need to refer to the test schema. You can edit the default values of these variables if required.

You can run tests against your production schema, but we recommend using a test schema to avoid any risk to your production data.

Input data

After the schema is created, populate it with the tables and data needed by the pipeline to test. You can use components such as:

• Create Table From Values: Creates a table directly from values you enter in the component.
• SQL Script: Executes SQL statements to create and populate tables.
• Zero Copy Clone: (Snowflake only) Clones an existing schema, providing a fast way to create a copy of production data for testing.

Running the pipeline to test

Use the Run Pipeline To Test component to execute the pipeline you are testing. This component runs the target pipeline to completion before the test continues. The target pipeline runs within the context of the test schema, so it reads from and writes to the temporary tables you have set up.

Asserting results

After the pipeline to test has run, use assert or compare components to verify the results. The following test components are available for this purpose:

• Assert Table: Confirms that a specified table exists.
• Assert Table Values: Checks that a table contains the expected values.
• Assert External Table: Confirms that an external table has been correctly created.
• Assert Scalar Variables: Verifies that a variable holds the expected value.
• Compare Table Values: Checks that two tables contain identical values. This is useful if your pipeline creates a table and you want to verify it against a separately defined expected-results table.

---

Available components

Tests support a subset of orchestration pipeline components, as well as some components that are available exclusively to tests.

Dedicated test components

Component	Description
Assert Table Values	Check that the value of a specified table matches the values declared in the component.
Compare Table Values	Check that the values of two specified tables match.
Create Table From Values	Create a table in the warehouse directly from values you enter.
Run Pipeline To Test	Run an orchestration or transformation pipeline as the subject of a test.
Zero Copy Clone	Clone a schema (Snowflake only).

Available orchestration components

Component	Description
And	Wait for all inputs to complete before continuing the pipeline.
Append To Grid	Append data to a pre-existing grid variable.
Assert External Table	Confirm that an external table has been correctly created.
Assert Scalar Variables	Verify that a variable value is as expected.
Assert Table	Confirm that a target table has been correctly created.
Azure Queue Storage Message	Post a message to Azure Queue Storage.
Cloud Pub/Sub	Post a message to a Google Cloud Pub/Sub topic.
CloudWatch Publish	Publish metrics to CloudWatch.
Create External Table	Define a read-only table that references data stored outside of your data warehouse.
Create Table	Create or replace a table.
CSV Load	Load CSV data into a newly created table from objects stored in various sources.
Delete Tables	Delete a list of tables.
End Failure	Mark the pipeline as failed, even if successful.
End Success	Mark the pipeline as succeeded, even if certain parts failed.
File Iterator	Loop over matching files in a remote file system.
Fixed Iterator	Loop over values of a simple sequence.
Grid Iterator	Loop over the rows of a grid variable.
If	Evaluate an expression and direct the flow of the pipeline based on the result.
JDBC Table Metadata To Grid	Take the metadata from a JDBC table to populate a grid variable.
JSON Load	Load JSON data into a newly created table from objects stored in various sources.
Loop Iterator	Loop over values of a simple sequence as a for loop.
Or	Wait for any one input to complete before the pipeline continues.
Parquet Load	Load Parquet data into a newly created table from objects stored in various sources.
Print Variables	Write the values of variables into the task message.
Python Pushdown	Execute a script and write output into the task message.
Python Script	Run a Python script in-process via the Python 3 interpreter.
Query Result To Grid	Write a custom SQL query and store the results in a grid variable.
Query Result To Scalar	Write a custom SQL query and store the results in a scalar variable.
Refresh External Table	Refresh an external table, re-syncing it with the target data.
Run Orchestration	Run an orchestration pipeline to completion.
Run Shared Pipeline	Run a shared pipeline to completion.
Run Transformation	Run a transformation pipeline to completion.
Send Email	Send emails upon any pipeline failure or success.
SNS Message	Send a simple message to an SNS topic.
SQL Script	Execute one or more SQL statements.
SQS Message	Post a message to an Amazon Simple Queue Service queue.
Table Iterator	Loop over rows of data within a table or view.
Table Metadata To Grid	Take the metadata from a table to populate a grid variable.
Truncate Table	Remove all rows from a table while keeping the table intact.
Update Scalar	Update the values of scalar variables.
Webhook Post	Send a payload or message to a unique URL.