Cloud provider credentials

Each environment in your project should have at least one set of cloud credentials associated with it. This allows you to access account resources on different platforms other than that hosting your project. For example, if your project is on AWS and you want to access resources in Azure, you need to associate your Azure cloud credentials with the environment.

• In a Matillion Full SaaS project, you must specify a cloud connection to access resources in your own cloud provider account.
• In a Hybrid SaaS runner project that needs to connect to cloud resources outside the platform hosting your , you need to provide credentials to connect to that platform.

Each credential allows access to a single cloud platform. Supported platforms are:

• Amazon Web Services
• Azure
• Google Cloud Platform

Credentials are configured at the project level and apply to selected environments within the project. They can then be used by any pipelines that use those environments.

only uses your cloud provider credentials to connect to the resources specified within your pipeline component configurations. Your credentials are not used to access or retrieve secrets.

Understanding environment defaults

Each cloud connection should be given default values for the connection credentials. Default values are a fallback for all environments that have defaults enabled. You can then add specific overrides for each environment as needed. Pipelines will fail in any environment that lacks either a default or an override.

As you can add default values and specific overrides, it’s important to understand how they interact, and which credentials an environment uses.

• If you only fill in the defaults for the cloud connection, you can use that connection in any environment. Every environment that uses the connection will use the default credentials.
• If you don’t fill in the defaults for the cloud connection, you can only use the connection in environments that you have specified overrides for. The environment will use the override credentials.
• If you fill in the defaults for the cloud connection and also fill in some environmental overrides, you can use that connection in any environment. Every environment that you have specified overrides for will use the override credentials. Every other environment will use the default credentials.
• If you don’t fill in the defaults for the cloud connection, and don’t associate the connection with any environment you have specified overrides for, the connection then becomes essentially dormant, not used or usable by any environment.

---

Creating a cloud provider credential

To set up a new cloud provider credential, follow these steps:

1. In your project, click the Cloud credentials tab.

   This tab lists all cloud credentials currently created.

2. Click Add cloud credential.

3. Choose the cloud provider and click Continue.

4. Complete the following fields:

   • Name: A unique, descriptive name for your cloud provider credential. This is used to reference this connection in pipelines. It can’t be changed after creation.
   • Provider: This is pre-populated with the provider chosen on the previous screen and can’t be changed here.
   • Description: An optional description of the credential. This is for your information only, and does not affect the credential.
   • Runner: This is only required if you host your own in a Hybrid SaaS deployment. Use the drop-down to select the that will be used to store these credentials securely in .

5. Enter credentials for the connection. This includes both Default values and as many environment-specific credentials as you require. Read Understanding environment defaults, above, for more details.

   AWS

   • Access key ID: Access key ID for an existing Amazon IAM user.
   • Secret access key: Secret access key for an existing Amazon IAM user.

   Obtain these credentials from the AWS console, as described in Manage access keys for IAM users.

   Azure

   • Tenant ID: The Azure Tenant ID, also known as Directory ID.
   • Client ID: The Azure Client ID, also known as Application ID.
   • Secret key: The Azure Secret Key.

   Obtain these credentials from the Azure portal, as described in Register a Microsoft Entra app and create a service principal and Add and manage application credentials in Microsoft Entra ID.

   For Azure Databricks, an additional step is required to connect to cloud storage. For further information, read Accessing Storage using a service principal and Microsoft Entra ID.

   To connect to Data Lake Storage Gen2 and Blob Storage, read Connect to Azure Data Lake Storage Gen2 and Blob Storage.

   GCP

   • Service account key: Enter the service account key to access services such as Google BigQuery and Google Cloud Storage. Make sure that the key is in a valid JSON format, as in the following example:

     {
     "type": "service_account",
     "project_id": "abcde",
     "private_key_id": "",
     "private_key": "",
     "client_email": "[e-mail]",
     "client_id": "XXXXXXXXXXXXX",
     "auth_uri": "https://accounts.google.com/o/oauth2/auth",
     "token_uri": "https://accounts.google.com/o/oauth2/token",
     "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
     "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/abcde%40appspot.gserviceaccount.com"
     }
     

   Obtain this credential from the GCP console, as described in Create and delete service account keys.

6. Click Add to move to the Update environment defaults screen.

7. Select the environments where this connection should act as the default credential for accessing cloud resources. Any number of environments can be selected, or the step can be skipped without making any selection. For any environments you select, ensure that you specified default credentials or environment-specific credentials in step 5. Read Understanding environment defaults, above, for more details.

8. If you select environments to use the connection, click Update. Otherwise, click Skip.

---

Deleting a cloud provider credential

1. In your project, click the Cloud credentials tab.

   This tab lists all cloud credentials currently created.

2. Click the three dots … in the row of the credential you want to delete.

3. Click Delete cloud credential.

4. Click Yes, delete to confirm deletion. Otherwise, click Cancel.

---

Editing a cloud provider credential

1. In your project, click the Cloud credentials tab.

   This tab lists all cloud credentials currently created.

2. Click the three dots … in the row of the credential you want to edit.

3. Click Edit cloud credential.

4. Edit any of the details described in Creating a cloud provider credential. You can’t rename the credential here.

5. Click Update.

---

Roles and permissions for Cloud Storage

To load your data directly into your preferred cloud storage destination, such as S3, Azure Blob Storage, or Google Cloud Storage, you’ll need to apply the following roles and permissions to your cloud accounts:

Cloud Storage	Role/policy	Permissions
S3	An appropriate policy name	s3:DeleteObject s3:GetObject s3:PutObject s3:ListAllMyBuckets
Azure storage	Storage Blob Data Contributor For information about adding this role to your account, read Assign Azure roles using the Azure portal.	Actions: Microsoft.Storage/storageAccounts/blobServices/containers/delete Microsoft.Storage/storageAccounts/blobServices/containers/read Microsoft.Storage/storageAccounts/blobServices/containers/write Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action DataActions: Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action
	Storage Account Contributor For information about adding this role to your account, read Assign Azure roles using the Azure portal.	Actions: Microsoft.Authorization//read Microsoft.Insights/alertRules/ Microsoft.Insights/diagnosticSettings/ Microsoft.Network/virtualNetworks/subnets/joinViaServiceEndpoint/action Microsoft.ResourceHealth/availabilityStatuses/read Microsoft.Resources/deployments/ Microsoft.Resources/subscriptions/resourceGroups/read Microsoft.Storage/storageAccounts/ Microsoft.Support/
Google Cloud Storage	Storage Admin	storage.buckets.*

---

Roles and permissions for Amazon Bedrock Prompt

The Amazon Bedrock Prompt component requires permissions to be set to allow access to the Bedrock large language models (LLMs).

Coarse-grained permission:

• AmazonBedrockFullAccess

Fine-grained permission:

• bedrock:InvokeModel on the models that need to be accessible.

To allow all models, use:

{
  "Sid": "InvokeModel",
  "Effect": "Allow",
  "Action": [
    "bedrock:InvokeModel"
  ],
  "Resource": "arn:aws:bedrock:*::foundation-model/*"
}

To restrict access to a single model, for example anthropic.claude-3-sonnet-20240229-v1:0, use:

{
  "Sid": "InvokeModel",
  "Effect": "Allow",
  "Action": [
    "bedrock:InvokeModel"
  ],
  "Resource": "arn:aws:bedrock:*::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0"
}