Full SaaS architecture
In Full SaaS deployments, Matillion will host the required and project metadata. This will call out to data warehouses and source services in the customer cloud.
Hybrid SaaS architecture
In Hybrid SaaS deployments, the is hosted in the customer’s cloud (Amazon Web Services or Microsoft Azure) and can connect to services such as sources and data warehouses from there.
Detailed Architecture diagram
In , users have the option to choose between two deployment models: the Full SaaS model and the Hybrid SaaS model, each offering distinct setups for deployment and workflow execution. In the Full SaaS model, Matillion provides and manages the hosted , alleviating users from concerns related to deployment, upgrades, and monitoring. The Matillion-hosted directly interfaces with and Matillion hosted vault, so secrets can be defined directly from within your project. In the Hybrid SaaS model, users have more control over their infrastructure as they deploy and manage their own within their cloud environment. They must also manage their own secret vault and secrets on their chosen cloud platform. is built on top of Git. Git offers a range of features, including version control, collaboration, branching and merging capabilities, and a distributed system architecture. Additionally, users can integrate their own Git repository with for version control and collaboration on pipelines. In , Git seamlessly operates without users needing to install or maintain local copies of the application code. Instead, all Git interactions are supported natively in the application. Read our GitHub app overview.
Workflow expanded
Referring to the architecture diagram above, the following notes can be made on the high-level workflow used by and its task execution in :Authentication and secret management
- Users authenticate themselves to access .
- Secrets, including API keys and other credentials, are managed securely. This ensures that only authorized users can access sensitive data.
Pipeline design and management
- Users design and configure data pipelines using .
- integrates with the Component Information service to provide metadata and handle design-time requests.
- You can also integrate Git with for version control and collaboration on pipelines. This allows you to track changes, revert to previous versions, and work together on pipelines as a team.
Maia runner management and monitoring
- The Agent Manager deploys, upgrades, and monitors s within .
- It also queries connection statuses to ensure seamless operation.
Workflow Orchestration and observability
- The Workflow Execution Engine orchestrates pipeline execution.
- Maia offers pipeline run history features for monitoring and performance tracking.
Task execution and scheduling
- Task requests are sent to the Agent Gateway for direct communication with customer-hosted s.
- The scheduler coordinates pipeline executions based on schedules.
API gateway for public exposure
- Grant controlled access to data pipelines via secure, public API endpoints.
- Empower external applications to interact with your pipeline data programmatically.
- Authentication and authorization mechanisms for robust API access control.
Maia runner communication and secret access
- Matillion-hosted s communicate with and Hosted Vault for secure access to customer secrets.
- They also use the Connector Service to retrieve data from various sources such as Salesforce, SAP, and databases.
Customer secret management and Maia runner responsibilities
- Customer secret vaults securely store and retrieve customer secrets.
- Customer-hosted s are responsible for running any components in data processing pipelines.