Terrateam Deployment Workflow

Deep dive into how Terrateam orchestrates infrastructure deployments across DocuStack's multi-repository architecture.

Overview

DocuStack uses Terrateam for infrastructure deployment with a unique multi-repository architecture:

Repository	Purpose	Contents
`docustack-mono`	Application code	Lambda functions, services
`docustack-infrastructure-modules`	Infrastructure catalog	Versioned Terraform modules
`docustack-infrastructure-live`	Environment configs	Terragrunt configurations

Key Insight: Terrateam runs from docustack-infrastructure-live but accesses Lambda code from docustack-mono via relative filesystem paths.

Architecture

Repository Layout

~/development/docustack/
│
├── docustack-mono/                          # Application Repository
│   └── services/
│       └── lambdas/                         # Lambda code lives here
│           ├── nightly-scheduler/
│           ├── bastion-orchestrator/
│           ├── infra-orchestrator/
│           ├── ip-whitelist-manager/
│           └── db-init/
│
├── docustack-infrastructure-modules/        # Infrastructure Catalog
│   └── modules/
│       ├── nightly-scheduler/               # Terraform module
│       ├── bastion-orchestrator/
│       └── ...
│
└── docustack-infrastructure-live/           # Environment Configurations
    ├── .terrateam/
    │   └── config.yml                       # Terrateam runs from here
    ├── _envcommon/
    │   └── bastion-orchestrator.hcl         # Points to monorepo Lambda code
    └── dev/us-east-1/
        └── bastion-orchestrator/
            └── terragrunt.hcl

Data Flow

┌─────────────────────────────────────────────────────────────────────────┐
│                         GitHub Pull Request                              │
│                    (docustack-infrastructure-live)                       │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         Terrateam Cloud                                  │
│  • Detects PR in infrastructure-live repo                               │
│  • Triggers GitHub Actions workflow                                     │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                    GitHub Actions Runner                                 │
│  • Checks out docustack-infrastructure-live                             │
│  • Assumes AWS role via OIDC                                            │
│  • Runs Terragrunt commands                                             │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         Terragrunt                                       │
│  • Reads _envcommon/bastion-orchestrator.hcl                            │
│  • Resolves lambda_source_dir path to monorepo                          │
│  • Downloads Terraform module from GitHub                               │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         Terraform Module                                 │
│  • Receives lambda_source_dir variable                                  │
│  • Uses archive_file to ZIP Lambda code                                 │
│  • Deploys Lambda function to AWS                                       │
└─────────────────────────────────────────────────────────────────────────┘

How Lambda Code is Deployed

The Path Resolution

Question: How does Terrateam access Lambda code from docustack-mono when running from docustack-infrastructure-live?

Answer: Via relative filesystem paths resolved by Terragrunt.

Step-by-Step Breakdown

1. Environment Configuration

_envcommon/bastion-orchestrator.hcl
terraform {
  source = "git::git@github.com:docustackapp/docustack-infrastructure-modules.git//modules/bastion-orchestrator?ref=v1.0.0"
}

inputs = {
  name        = "${local.project_name}-${local.account_name}"
  environment = local.account_name

  # Points to Lambda code in monorepo via relative path
  lambda_source_dir = "${dirname(find_in_parent_folders("root.hcl"))}/../docustack-mono/services/lambdas/bastion-orchestrator"
}

Path Resolution:

# Starting from: docustack-infrastructure-live/dev/us-east-1/bastion-orchestrator/
find_in_parent_folders("root.hcl")
# Returns: /path/to/docustack-infrastructure-live/root.hcl

dirname(...)
# Returns: /path/to/docustack-infrastructure-live

/../docustack-mono/services/lambdas/bastion-orchestrator
# Final path: /path/to/docustack-mono/services/lambdas/bastion-orchestrator

2. Terraform Module

modules/bastion-orchestrator/main.tf
variable "lambda_source_dir" {
  description = "Absolute path to the Lambda source directory"
  type        = string
}

data "archive_file" "lambda" {
  type        = "zip"
  source_dir  = var.lambda_source_dir  # Points to docustack-mono
  output_path = "${path.module}/lambda_bastion_orchestrator.zip"
  excludes    = ["__pycache__", "*.pyc", ".pytest_cache", ".venv"]
}

resource "aws_lambda_function" "orchestrator" {
  function_name    = local.orchestrator_function_name
  filename         = data.archive_file.lambda.output_path
  source_code_hash = data.archive_file.lambda.output_base64sha256
  # ... other configuration
}

What Happens:

Terraform reads lambda_source_dir variable (points to monorepo)
archive_file data source ZIPs the Lambda code from monorepo
ZIP is created in the module's temporary directory
Lambda function is deployed with the ZIP file
source_code_hash ensures Lambda updates when code changes

Deployment Scenarios

Scenario 1: Lambda Code Changes Only

Trigger: Developer updates Lambda code in docustack-mono/services/lambdas/

# 1. Update Lambda code in monorepo
cd ~/development/docustack/docustack-mono
git checkout -b feat/bastion-timeout-extension
# Edit services/lambdas/bastion-orchestrator/handler.py
git commit -m "feat: add bastion timeout extension to 4 hours"
git push origin feat/bastion-timeout-extension

# 2. Trigger deployment via infrastructure-live PR
cd ~/development/docustack/docustack-infrastructure-live
git checkout -b deploy/bastion-timeout-extension
# No file changes needed! Just create PR to trigger Terrateam
git commit --allow-empty -m "deploy: bastion timeout extension"
git push origin deploy/bastion-timeout-extension

# 3. Open PR on GitHub
# 4. Comment: "terrateam plan"
# 5. Review plan showing Lambda update
# 6. Comment: "terrateam apply"

tip

Lambda code changes require no changes to infrastructure code. The archive_file data source automatically detects the code change via source_code_hash.

Scenario 2: Infrastructure Configuration Changes

Trigger: Developer updates Terraform module or environment configuration

# 1. Update Terraform module
cd ~/development/docustack/docustack-infrastructure-modules
git checkout -b feat/bastion-cloudwatch-metrics
# Edit modules/bastion-orchestrator/iam.tf
git commit -m "feat: add CloudWatch metrics permissions"
git push origin feat/bastion-cloudwatch-metrics
# Merge PR, then tag:
git tag v1.1.0
git push --tags

# 2. Update module version in live repo
cd ~/development/docustack/docustack-infrastructure-live
git checkout -b update/bastion-orchestrator-v1.1.0
# Edit _envcommon/bastion-orchestrator.hcl
# Change: source = "...?ref=v1.1.0"
git commit -m "chore: update bastion-orchestrator module to v1.1.0"
git push origin update/bastion-orchestrator-v1.1.0

# 3. Open PR, Terrateam auto-plans
# 4. Review and comment: "terrateam apply"

Scenario 3: Combined Changes

When both Lambda code and infrastructure need updates:

Update Lambda code in docustack-mono
Update Terraform module in docustack-infrastructure-modules
Tag new module version
Update version reference in docustack-infrastructure-live
Deploy via Terrateam PR

Terrateam Configuration

Stacks Configuration

Terrateam uses stacks to manage deployment dependencies:

.terrateam/config.yml
# OIDC Authentication
hooks:
  all:
    pre:
      - type: oidc
        provider: aws
        role_arn: "arn:aws:iam::216482851263:role/terrateam-dev"

# Engine Configuration
engine:
  name: terragrunt
  version: "0.67.16"

# Directory Tags for Stack Organization
dirs:
  "**/bastion-orchestrator":
    tags: [security, orchestration]
  "**/infra-orchestrator":
    tags: [orchestration, lambda]
  "**/nightly-scheduler":
    tags: [orchestration, scheduler]
  "**/db-init-lambda":
    tags: [data, initialization]
  "**/ip-whitelist":
    tags: [security, network]

Stack Deployment Order

Foundation - bootstrap, vpc (no dependencies)
Compute - ecs-cluster, ecr (depends on foundation)
Data - rds, db-init-lambda (depends on foundation)
Security - bastion, bastion-orchestrator, ip-whitelist (depends on foundation)
Orchestration - temporal, nightly-scheduler, infra-orchestrator (depends on compute + data)
Integration - slack-bot (depends on all other layers)

Multi-Repository Access

Local Development

Requirement: All three repositories must be cloned side-by-side:

~/development/docustack/
├── docustack-mono/
├── docustack-infrastructure-modules/
└── docustack-infrastructure-live/

Verification:

cd ~/development/docustack/docustack-infrastructure-live/dev/us-east-1/bastion-orchestrator
terragrunt plan

# Terragrunt will:
# 1. Resolve lambda_source_dir to ../../docustack-mono/services/lambdas/bastion-orchestrator
# 2. Download module from GitHub (docustack-infrastructure-modules)
# 3. Package Lambda code from monorepo
# 4. Generate Terraform plan

CI/CD (Terrateam Cloud)

For Terrateam to access Lambda code in CI/CD, the GitHub Actions workflow checks out both repositories:

.github/workflows/terrateam.yml
steps:
  # Checkout infrastructure-live (main repo)
  - name: Checkout infrastructure-live
    uses: actions/checkout@v4
    with:
      path: docustack-infrastructure-live

  # Checkout monorepo (for Lambda code)
  - name: Checkout monorepo
    uses: actions/checkout@v4
    with:
      repository: docustackapp/docustack-mono
      path: docustack-mono
      token: ${{ secrets.GITHUB_TOKEN }}

  # Run Terrateam
  - name: Run Terrateam Action
    uses: terrateamio/action@v1
    with:
      work-token: '${{ github.event.inputs.work-token }}'

Security Considerations

OIDC Authentication

Terrateam uses OpenID Connect (OIDC) to assume AWS roles without long-lived credentials:

No AWS access keys stored in GitHub
Short-lived credentials (1 hour)
Audit trail via CloudTrail
Least privilege access per environment

Lambda Code Integrity

resource "aws_lambda_function" "orchestrator" {
  source_code_hash = data.archive_file.lambda.output_base64sha256
}

What This Ensures:

Lambda only updates when code actually changes
Prevents accidental deployments
Provides audit trail of code changes

Secrets Management

Always:

Use AWS Secrets Manager for sensitive data
Reference secrets via Terraform data sources
Rotate secrets regularly

Never:

Hardcode secrets in Lambda code
Store secrets in Terraform state
Pass secrets via environment variables in Terragrunt

Troubleshooting

Lambda Code Not Found

Error:

Error: error archiving directory: could not archive missing directory: 
/path/to/docustack-mono/services/lambdas/bastion-orchestrator

Solution:

# Local development - verify repos are present
cd ~/development/docustack
ls -la  # Should show all three repos

# CI/CD - ensure multi-repository checkout in workflow

Lambda Not Updating

Symptom: Lambda code changes not reflected in AWS after deployment.

Debug:

cd ~/development/docustack/docustack-infrastructure-live/dev/us-east-1/bastion-orchestrator
terragrunt plan

# Look for:
# ~ source_code_hash = "abc123..." -> "def456..."

Solution:

# Force Lambda update
terragrunt taint aws_lambda_function.orchestrator
terragrunt apply

Module Version Mismatch

Error:

Error: Failed to download module from git::git@github.com:docustackapp/
docustack-infrastructure-modules.git//modules/bastion-orchestrator?ref=v1.1.0

Solution:

cd ~/development/docustack/docustack-infrastructure-modules
git tag  # Verify tag exists
git push --tags  # Push tags to GitHub

OIDC Authentication Failure

Error:

Error: Error assuming role: AccessDenied: User is not authorized to perform: sts:AssumeRoleWithWebIdentity

Solution: Verify IAM role trust policy includes the correct GitHub repository:

{
  "Condition": {
    "StringLike": {
      "token.actions.githubusercontent.com:sub": "repo:docustackapp/docustack-infrastructure-live:*"
    }
  }
}

Best Practices

Lambda Code Development

Do	Don't
Keep Lambda code in `docustack-mono/services/lambdas/`	Put Lambda code in infrastructure modules repo
Use `requirements.txt` for Python dependencies	Hardcode configuration values
Include README.md in each Lambda directory	Include large dependencies (use Lambda layers)
Write unit tests for Lambda functions	Commit `.pyc` files or `__pycache__`

Infrastructure Changes

Do	Don't
Version Terraform modules with semantic versioning	Make breaking changes without version bump
Test module changes in dev environment first	Skip testing in dev environment
Document breaking changes in CHANGELOG.md	Update prod directly without staging validation

Deployment Workflow

Do	Don't
Always review Terraform plan before applying	Apply changes without reviewing plan
Deploy to dev → staging → prod	Deploy to prod without staging validation
Monitor CloudWatch logs after deployment	Ignore Terraform warnings

Summary

Deployment Patterns

Change Type	Repository	Workflow
Lambda code only	`docustack-mono`	Update code → Trigger Terrateam via PR
Infrastructure only	`docustack-infrastructure-modules` + `docustack-infrastructure-live`	Update module → Tag version → Update live config → Deploy
Both	All three repos	Update code + module → Tag → Update config → Deploy

Key Takeaways

Lambda code lives in docustack-mono - Application code stays with application
Terraform modules live in docustack-infrastructure-modules - Infrastructure patterns are versioned
Environment configs live in docustack-infrastructure-live - Terrateam runs from here
Relative paths connect the repos - lambda_source_dir points to monorepo
Multi-repo checkout required for CI/CD - Terrateam needs access to both repos

Overview​

Architecture​

Repository Layout​

Data Flow​

How Lambda Code is Deployed​

The Path Resolution​

Step-by-Step Breakdown​

1. Environment Configuration​

2. Terraform Module​

Deployment Scenarios​

Scenario 1: Lambda Code Changes Only​

Scenario 2: Infrastructure Configuration Changes​

Scenario 3: Combined Changes​

Terrateam Configuration​

Stacks Configuration​

Stack Deployment Order​

Multi-Repository Access​

Local Development​

CI/CD (Terrateam Cloud)​

Security Considerations​

OIDC Authentication​

Lambda Code Integrity​

Secrets Management​

Troubleshooting​

Lambda Code Not Found​

Lambda Not Updating​

Module Version Mismatch​

OIDC Authentication Failure​

Best Practices​

Lambda Code Development​

Infrastructure Changes​

Deployment Workflow​

Summary​

Deployment Patterns​

Key Takeaways​

Overview

Architecture

Repository Layout

Data Flow

How Lambda Code is Deployed

The Path Resolution

Step-by-Step Breakdown

1. Environment Configuration

2. Terraform Module

Deployment Scenarios

Scenario 1: Lambda Code Changes Only

Scenario 2: Infrastructure Configuration Changes

Scenario 3: Combined Changes

Terrateam Configuration

Stacks Configuration

Stack Deployment Order

Multi-Repository Access

Local Development

CI/CD (Terrateam Cloud)

Security Considerations

OIDC Authentication

Lambda Code Integrity

Secrets Management

Troubleshooting

Lambda Code Not Found

Lambda Not Updating

Module Version Mismatch

OIDC Authentication Failure

Best Practices

Lambda Code Development

Infrastructure Changes

Deployment Workflow

Summary

Deployment Patterns

Key Takeaways