Cloud provider credentials

Each environment in your project should have at least one set of cloud credentials associated with it. This allows you to access account resources on different platforms other than that hosting your project. For example, if your project is on AWS and you want to access resources in Azure, you need to associate your Azure cloud credentials with the environment.

• In a Matillion Full SaaS project, you must specify a cloud connection to access resources in your own cloud provider account.
• In a Hybrid SaaS runner project that needs to connect to cloud resources outside the platform hosting your Maia runner, you need to provide credentials to connect to that platform.

Each credential allows access to a single cloud platform. Supported platforms are:

• Amazon Web Services
• Azure
• Google Cloud

Credentials are configured at the project level and apply to selected environments within the project. They can then be used by any pipelines that use those environments.

Maia only uses your cloud provider credentials to connect to the resources specified within your pipeline component configurations. Your credentials are not used to access or retrieve secrets.

Each cloud provider credential is created as a Connection at the project level.

To create a cloud provider credential, read Add a connection. Follow the process given there, then configure the specific connection as described below.

AWS

• Access key ID: Access key ID for an existing Amazon IAM user.
• Secret access key: Secret access key for an existing Amazon IAM user.

Obtain these credentials from the AWS console, as described in Manage access keys for IAM users.

Azure

• Tenant ID: The Azure Tenant ID, also known as Directory ID.
• Client ID: The Azure Client ID, also known as Application ID.
• Secret key: The Azure Secret Key.

Obtain these credentials from the Azure portal, as described in Register a Microsoft Entra app and create a service principal and Add and manage application credentials in Microsoft Entra ID.

For Azure Databricks, an additional step is required to connect to cloud storage. For further information, read Accessing Storage using a service principal and Microsoft Entra ID.

To connect to Data Lake Storage Gen2 and Blob Storage, read Connect to Azure Data Lake Storage Gen2 and Blob Storage.

Google Cloud

• Service account key: Enter the service account key to access services such as Google BigQuery and Google Cloud Storage. Make sure that the key is in a valid JSON format, as in the following example:

  {
  "type": "service_account",
  "project_id": "abcde",
  "private_key_id": "",
  "private_key": "",
  "client_email": "[e-mail]",
  "client_id": "XXXXXXXXXXXXX",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/abcde%40appspot.gserviceaccount.com"
  }
  

Obtain this credential from the Google Cloud console, as described in Create and delete service account keys.

---

Roles and permissions for cloud storage

To load your data directly into your preferred cloud storage destination, such as S3, Azure Blob Storage, or Google Cloud Storage, you’ll need to apply the following roles and permissions to your cloud accounts.

S3

Amazon S3 documentation.

• Role/policy: An appropriate policy name

  • Permissions:
    • s3:DeleteObject
    • s3:GetObject
    • s3:PutObject
    • s3:ListBucket
    • s3:GetBucketLocation
    • s3:ListAllMyBuckets
  Omitting s3:ListAllMyBuckets doesn’t necessarily cause pipeline errors, though it may cause validation errors. Verify the specific requirements for your use case before omitting it.

Azure storage

Azure supports two authorization methods for Blob Storage access:

• Shared Access Signature (SAS): Grants temporary access to specific blobs or containers without exposing account keys. Allows defining specific read, write, or list permissions with start and expiry times.
• Azure roles: Assigns built-in or custom roles based on the Azure Role-Based Access Control (RBAC) system, granting access to specific containers, blobs, or entire storage accounts.

Maia uses the Azure roles method, as described below.

• Role/policy: Storage Blob Data Contributor

  For information about adding this role to your account, read Assign Azure roles using the Azure portal.

  • Permissions:

    • Actions:

      • Microsoft.Storage/storageAccounts/blobServices/containers/delete
      • Microsoft.Storage/storageAccounts/blobServices/containers/read
      • Microsoft.Storage/storageAccounts/blobServices/containers/write
      • Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action

    • DataActions:

      • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete
      • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read
      • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write
      • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action
      • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action

• Role/policy: Storage Account Contributor

  For information about adding this role to your account, read Assign Azure roles using the Azure portal.

  • Permissions:

    • Actions:

      • Microsoft.Authorization//read
      • Microsoft.Insights/alertRules/
      • Microsoft.Insights/diagnosticSettings/
      • Microsoft.Network/virtualNetworks/subnets/joinViaServiceEndpoint/action
      • Microsoft.ResourceHealth/availabilityStatuses/read
      • Microsoft.Resources/deployments/
      • Microsoft.Resources/subscriptions/resourceGroups/read
      • Microsoft.Storage/storageAccounts/
      • Microsoft.Support/

• To interact with Azure Queue Storage, your Azure account must have the role AQS Storage Queue Data Message Sender assigned.
• To integrate with Azure Data Lake Storage, roles such as Storage Blob Data Owner and Storage Blob Data Contributor may be relevant for data access and management.

Google Cloud Storage

Google Cloud Storage documentation.

• Role/policy: Storage Admin

  • Permissions:

    • storage.buckets.*

---

Roles and permissions for Amazon Bedrock Prompt

The Amazon Bedrock Prompt component requires permissions to be set to allow access to the Bedrock large language models (LLMs).

Coarse-grained permission:

• AmazonBedrockFullAccess

Fine-grained permission:

• bedrock:InvokeModel on the models that need to be accessible.

To allow all models, use:

{
  "Sid": "InvokeModel",
  "Effect": "Allow",
  "Action": [
    "bedrock:InvokeModel"
  ],
  "Resource": "arn:aws:bedrock:*::foundation-model/*"
}

To restrict access to a single model, for example anthropic.claude-3-sonnet-20240229-v1:0, use:

{
  "Sid": "InvokeModel",
  "Effect": "Allow",
  "Action": [
    "bedrock:InvokeModel"
  ],
  "Resource": "arn:aws:bedrock:*::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0"
}