Kuiper SU Deploy Automation — Setup Guide

This guide covers everything required to enable Kuiper client pack (SU) deployment automation for an existing customer environment. Once configured, Epic client packs deploy through two paths:

• On demand — the GitHub Actions Deploy Kuiper Package workflow deploys a client pack by RA number.
• Automatically — a daily EventBridge-scheduled check queries the ODB for its currently installed Epic RA and, when the ODB has advanced past the last RA deployed to the W&SS servers, triggers the same deployment and records the new RA.

Either way, the deployment runbook powers on the W&SS servers, drives the deployment through Kuiper's API, polls until it settles, and returns the servers to pilot-light state.

How It Works

GitHub Actions (Deploy Kuiper Package)  ──────────────┐
                                                      ├─>  Kuiper-Package-Deploy-Runbook
EventBridge (daily) ─> SU-Deploy-Check-Runbook ───────┘      1. startGroupInstances  — power on the W&SS servers
  1. getCurrentRA      — Get-Installed-RA-Runbook:           2. waitForBoot          — sleep 3 minutes
     query the ODB IRIS instance for the highest             3. deployKuiperPackage  — kuiper_api_function:
     installed Epic RA (^EMS global via SSM RunCommand)          submit the deploy (downtime-tolerant while the
  2. getLastDeployedRA — read /epic/ire/su/                      client systems status is 'inactive')
     last_deployed_ra                                        4. waitForDeployment    — poll status every 10 min
  3. compareRA         — deploy only if the ODB's RA             (up to 9 iterations)
     is newer                                                5. runPackageCleanup    — uninstall old packages past
  4. deployKuiper      — run the deploy runbook, then            the retention limit, poll until settled
     update the last-deployed-RA parameter                   6. notifyDeploySummary  — email installed/uninstalled
                                                             7. stopInstances        — return to pilot light

The kuiper_api_function Lambda runs PowerShell on a custom provided.al2023 runtime layer, imports the KuiperAPI module at cold start, and authenticates to the Kuiper server using a service account whose password is read from an SSM SecureString. If any step fails — including the deploy submission itself — the runbook still routes through the stop step so the servers are never left running.

After a successful deploy, the runbook enforces package retention: it keeps the newest MaxInstalledPackages (default 2) installed packages per Kuiper group and uninstalls the rest, so old client packs don't accumulate on the W&SS servers. Both the install and uninstall phases read the client systems activation status parameter just before running — while the environment is inactive, Kuiper is allowed to reboot member machines to complete the operation (DowntimeTolerant); when it's active, reboots are withheld.

Prerequisites

1. Kuiper service account

The automation authenticates to the Kuiper API with a dedicated service account (KUIPER_USERNAME, e.g. kuiper-api).

Admin permissions on Hyperspace Web and Interconnect servers required

The account chosen to run the API must have administrator permissions on the servers that run Hyperspace Web and Interconnect. Kuiper performs the client pack installation on those servers under this account's identity; without local admin rights on them, deployments will fail even though API authentication succeeds.

In practice this means adding the service account to the local Administrators group (directly or via a domain group / GPO) on every HSW and ICF server in the target environment.

The account also needs to be added to a Kuiper security group that grants it the rights to deploy packages.

2. Patching automation components

The runbook reuses components from the Patching Automation implementation. These must already exist:

• ec2_stop_function and start_resource_group_instances_function Lambdas (Lambda Function)
• The epic-servers-group resource group targeting the W&SS servers via the PatchGroup tag (Resource Groups)
• Lambda_AutomationRole and the lambda_automation_role_assume_policy policy document
• The /client_systems/ire/status SSM parameter (SSM Parameters) — the deploy runbook reads it to decide whether Kuiper may reboot member machines (DowntimeTolerant) during installs and cleanup uninstalls

3. Network reachability

The kuiper_api_function Lambda attaches to private subnets and must be able to reach:

Destination	Purpose
The Kuiper server (KUIPER_URL)	All API calls — the subnets must route to wherever Kuiper runs.
sapphirefilerepo.blob.core.windows.net (internet egress)	Downloads the KuiperAPI PowerShell module at cold start (NAT gateway or equivalent egress).
SSM (VPC endpoint or NAT)	The Parameters and Secrets extension fetches the Kuiper password.

4. S3 bucket for the layer zip

The PowerShell runtime layer zip exceeds Lambda's 70 MB inline publish limit, so the Lambda Layer module stages it through an existing S3 bucket the pipeline role can write to.

5. CI build step

The layer content is assembled by src/functions/powershell_runtime/build.sh, which the plan/apply workflows run automatically via the build-powershell-layer composite action. No manual step is needed when deploying through the pipelines; for local applies, run the script first on Linux/WSL (native Windows will not set the executable bits).

---

1. Existing Maps (augment)

Highlighted lines indicate a value that must be updated to match the customer's deployment.

Security Group

Add an egress-only security group for the Lambda inside the VPC whose private subnets it will attach to:

vpcs = {
  # ... existing entries ...
  SharedInfra = {
    security_groups = {
      # ... existing entries ...
      KuiperLambdaSG = {
        egress = {
          Allow_Outbound = {
            ip_protocol = "-1"
            cidr_ipv4   = "0.0.0.0/0"
          }
        }
      }
    }
  }
}

Policy Documents

policy_documents = {
  # ... existing entries ...
  kuiper_lambda_policy = {
    statement = [
      {
        effect    = "Allow"
        actions   = ["ssm:GetParameter", "ssm:GetParameters"]
        resources = ["arn:aws:ssm:*:*:parameter/kuiper/*"]
      },
      {
        effect    = "Allow"
        actions   = ["kms:Decrypt"]
        resources = ["*"]
        # Scope decrypt to the aws/ssm key backing the password SecureString.
        condition = {
          StringEquals = {
            "kms:ViaService" = ["ssm.us-west-2.amazonaws.com"]
          }
        }
      }
    ]
  }
}

Note

The kms:ViaService region must match the customer's deployment region. If the SecureString uses a customer-managed KMS key (key_id set on the parameter), scope resources to that key's ARN instead.

IAM Policies

iampolicies = {
  # ... existing entries ...
  kuiper_lambda_policy = {
    policy = "kuiper_lambda_policy"
  }
}

IAM Roles

iamroles = {
  # ... existing entries ...
  Kuiper_Lambda_Role = {
    policies = [
      "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
      "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole",
      "kuiper_lambda_policy"
    ],
    assume_role_policy = "lambda_automation_role_assume_policy"
  }
}

Daily SU deploy check — policy and role attachment

The daily check runs under the existing EventBridge_SSM_AutomationRole, which needs additional permissions to resolve the ODB instance, run the IRIS query via SSM RunCommand, read/update the last-deployed-RA parameter, and publish deploy notifications. The SendCommand grant is scoped to the customer's ODB instance(s) by Name tag:

policy_documents = {
  # ... existing entries ...
  # Permissions for the daily SU deploy check automation
  su_deploy_check_policy = {
    statement = [
      {
        sid    = "ReadOnlyDiscovery"
        effect = "Allow"
        actions = [
          "ec2:DescribeInstances",
          "ssm:GetCommandInvocation", "ssm:ListCommands", "ssm:ListCommandInvocations",
          "ssm:DescribeInstanceInformation"
        ]
        resources = ["*"] # these actions do not support resource-level restrictions
      },
      {
        sid       = "SendRAQueryDocument"
        effect    = "Allow"
        actions   = ["ssm:SendCommand"]
        resources = ["arn:aws:ssm:*::document/AWS-RunShellScript"]
      },
      {
        sid       = "SendRAQueryToOdbOnly"
        effect    = "Allow"
        actions   = ["ssm:SendCommand"]
        resources = ["arn:aws:ec2:*:*:instance/*"]
        condition = {
          StringLike = { "ssm:resourceTag/Name" = ["PRDODB", "TSTODB"] } # the customer's ODB Name tag(s)
        }
      },
      {
        sid       = "SuStateParameters"
        effect    = "Allow"
        actions   = ["ssm:GetParameter", "ssm:GetParameters", "ssm:PutParameter"]
        resources = ["arn:aws:ssm:*:*:parameter/epic/ire/su/*"]
      },
      {
        sid       = "ReadClientSystemsStatus"
        effect    = "Allow"
        actions   = ["ssm:GetParameter", "ssm:GetParameters"]
        resources = ["arn:aws:ssm:*:*:parameter/client_systems/ire/status"]
      },
      {
        sid       = "PublishSuDeployNotifications"
        effect    = "Allow"
        actions   = ["sns:Publish"]
        resources = ["arn:aws:sns:*:*:SU-deploy-SNS-topic"]
      }
    ]
  }
}

iampolicies = {
  # ... existing entries ...
  su_deploy_check_policy = {
    policy = "su_deploy_check_policy"
  }
}

iamroles = {
  # ... existing entries ...
  EventBridge_SSM_AutomationRole = {
    policies = [
      "eventbridge_ssm_automation_policy",
      "ssm_lambda_invoke",
      "su_deploy_check_policy"   # <- add to the existing role
    ],
    assume_role_policy = "eventbridge_ssm_automation_assume"
  }
}

EventBridge Rules and Targets (daily schedule + failure notifications)

Two rules: the daily check schedule, and a pattern rule that emails when either SU deploy runbook fails or times out:

eb_rules = {
  # ... existing entries ...
  su-deploy-check-daily = {
    description         = "Daily check of the ODB installed Epic RA; deploys new Kuiper client packages when RA updates"
    schedule_expression = "cron(0 12 ? * * *)"
  }
  su-deploy-automation-failure = {
    description = "Notify when SU deploy automation executions fail or time out"
    event_pattern = {
        source      = ["aws.ssm"]
        detail-type = ["EC2 Automation Execution Status-change Notification"]
        detail = {
            Definition = ["SU-Deploy-Check-Runbook", "Kuiper-Package-Deploy-Runbook"]
            Status     = ["Failed", "TimedOut"]
        }
    }
  }
}

eb_targets = {
  # ... existing entries ...
  su-deploy-check-target = {
    ssm_runbook = "SU-Deploy-Check-Runbook-TF"
    rule        = "su-deploy-check-daily"
    role_arn    = "EventBridge_SSM_AutomationRole"
  }
  su-deploy-failure-notify = {
    rule = "su-deploy-automation-failure"
    arn  = "su-deploy-sns-topic"
    input_transformer = {
        paths = {
            document    = "$.detail.Definition"
            status      = "$.detail.Status"
            executionId = "$.detail.ExecutionId"
        }
        template = "{\"Subject\":\"SU Deploy Automation Failure\",\"Message\":\"SU deploy automation failure - Document: <document> - Status: <status> - Execution ID: <executionId>\"}"
    }
  }
}

SNS Topic, Policy and Subscription (deploy notifications)

Failure events (via EventBridge) and the deploy-success summary (published by the deploy runbook after cleanup) both go to a dedicated topic. EventBridge publishes to SNS via a topic policy — not a target role — so the topic needs an SNS Topic Policy entry allowing events.amazonaws.com, and SNS-target eb_targets entries must not set role_arn:

topics = {
  # ... existing entries ...
  su-deploy-sns-topic = {
    name = "SU-deploy-SNS-topic"
  }
}

subscriptions = {
  # ... existing entries ...
  SUDeployNotifications = {
    topic    = "su-deploy-sns-topic"
    protocol = "email"
    endpoints = [
      "[email protected]" # customer's distribution list
    ]
  }
}

sns_topic_policy = {
  # ... one entry per topic EventBridge publishes to (including the patching topics) ...
  su-deploy-topic-policy = {
    topic                 = "su-deploy-sns-topic"
    allow_service_publish = ["events.amazonaws.com"]
  }
}

Note

The topic name (SU-deploy-SNS-topic) is referenced by ARN pattern in su_deploy_check_policy and by the check runbook's notifyDeploySuccess step — if you rename the topic, update both.

---

2. New Maps (add entire blocks)

Lambda Layers

lambda_layers = {
  powershell_runtime = {
    description         = "PowerShell 7 custom runtime (bootstrap + pwsh) for provided.al2023 functions"
    compatible_runtimes = ["provided.al2023"]
    s3_bucket           = "epc-installersawss3-c" # layer zip exceeds the 70 MB inline publish limit
    required_files      = ["powershell/pwsh"]     # fail the plan if build.sh has not populated the layer content
  }
}

Lambda Function

lambdas = {
  # ... existing entries ...
  kuiper_api_function = {
    role          = "Kuiper_Lambda_Role"
    description   = "PowerShell KuiperAPI Automation"
    handler       = "handler.ps1"
    runtime       = "provided.al2023"
    memory_size   = 1024
    timeout       = 300
    architectures = ["x86_64"]
    layers = [
      "powershell_runtime",
      "arn:aws:lambda:us-west-2:345057560386:layer:AWS-Parameters-and-Secrets-Lambda-Extension:94"
    ]
    environment = {
      KUIPER_URL                = "https://kuiper.customer.example"
      KUIPER_USERNAME           = "kuiper-api"
      KUIPER_TIMEOUT            = "60"
      KUIPER_MODULE_VERSION     = "1.6.4"
      KUIPER_SKIP_CERT_CHECK    = "true"
      KUIPER_PASSWORD_PARAM     = "/kuiper/api/password" # SSM SecureString (value set out-of-band)
      KUIPER_CLIENT_ID          = "CHANGEME"             # Kuiper API client ID for the customer's Kuiper install
      KUIPER_TARGET_ENVIRONMENT = "CHANGEME"             # Epic environment(s) Kuiper deploys to, e.g. "ODRO"
      KUIPER_WHATIF             = "false"
    }
    vpc_config = {
      subnet_ids         = ["SharedInfra.SharedInfraPrivateAZ1", "SharedInfra.SharedInfraPrivateAZ2"]
      security_group_ids = ["KuiperLambdaSG"]
    }
  }
}

Region-specific extension ARN

The AWS Parameters and Secrets Lambda Extension layer ARN is region- and account-specific (the example above is us-west-2). Look up the correct ARN for the customer's region in the AWS documentation.

Environment variable reference for kuiper_api_function:

Variable	Required	Description
KUIPER_URL	yes	Base URL of the Kuiper server, as resolvable from the Lambda's subnets.
KUIPER_USERNAME	yes	Kuiper API service account (see prerequisites).
KUIPER_PASSWORD_PARAM	yes	SSM SecureString parameter holding the service account password.
KUIPER_CLIENT_ID	yes	Kuiper API client ID for the customer's Kuiper install.
KUIPER_TARGET_ENVIRONMENT	yes	Epic environment(s) Kuiper deploys to (comma/semicolon separated), e.g. ODRO.
KUIPER_MODULE_VERSION	no	KuiperAPI PowerShell module version to download (default 1.5.0; the package cleanup actions require 1.6.4 or later).
KUIPER_MODULE_URL_BASE	no	Override the module download URL base (defaults to the Sapphire file repo).
KUIPER_TIMEOUT	no	Kuiper API timeout in seconds (default 60).
KUIPER_SKIP_CERT_CHECK	no	Set "true" when the Kuiper server uses a self-signed/internal certificate.
KUIPER_WHATIF	no	Default dry-run mode when the payload does not specify one.
KUIPER_APPLICATION	no	Restrict deployments to specific Kuiper application(s); unset means all.
KUIPER_EXCLUDE_ENVIRONMENT	no	Environment(s) to exclude from deployment.

SSM Parameters

ssm_parameters = {
  # ... existing entries ...
  kuiper_api_password = {
    name        = "/kuiper/api/password"
    description = "Kuiper API service account password (value set out-of-band)"
    type        = "SecureString"
    value       = "CHANGEME"
    tier        = "Standard"
  },
  # Highest Epic RA whose client packages have been deployed via Kuiper automation
  su_last_deployed_ra = {
    name        = "/epic/ire/su/last_deployed_ra"
    description = "Highest Epic RA whose Kuiper client packages have been deployed to the W&SS servers"
    type        = "String"
    value       = "0"
    tier        = "Standard"
  }
}

Password is set out-of-band

The parameter is created with a CHANGEME placeholder and the module ignores value changes, so Terraform never manages (or sees) the real password. After the first apply, set it manually:

aws ssm put-parameter --name /kuiper/api/password --type SecureString \
  --value '<real password>' --overwrite

SSM Runbooks

Add the Kuiper-Package-Deploy-Runbook-TF document to ssm_runbooks. Copy the full block from environments/dev/terraform.tfvars and adjust the highlighted values — the stop payloads must reference the customer's existing pilot-light targets (the same patch_groups/targets used by the customer's Stop-Instances-Runbook):

ssm_runbooks = {
  # ... existing entries ...
  Kuiper-Package-Deploy-Runbook-TF = {
    # ... see environments/dev/terraform.tfvars for the full document ...
    # In the stopOnTimeout and stopInstances steps:
    #   Payload = <<-EOT
    #   {
    #       "patch_groups": ["epicIREwss"],
    #       "targets": {"awsicf-c": 1, "awshsw-c": "50%"}
    #   }
    #   EOT
  }
}

The polling budget is 9 iterations × 10 minutes (plus the 3-minute boot wait), so a deployment has roughly 90 minutes to settle before the runbook powers the servers back down and fails the execution. Adjust MaxIterations and the sleep Duration for larger client packs if needed.

Runbook parameters beyond RA and WhatIf:

Parameter	Default	Description
MaxInstalledPackages	2	Installed packages to keep per Kuiper group after the deploy (counting the newly deployed one); older inactive packages are uninstalled in the cleanup phase. Active or staging packages are never removed regardless of retention.
ClientSystemsStatusParameter	/client_systems/ire/status	SSM parameter gating downtime tolerance — installs/uninstalls may reboot member machines only while its value is not active. Read fresh before each phase.

The cleanup phase polls its uninstalls every 2 minutes (up to 10 iterations). A cleanup failure still returns the servers to pilot light, then fails the execution with a distinct message (stopOnCleanupFailure) — the deployed packages stay installed either way. On success, a summary of installed and uninstalled packages is published to the SU-deploy-SNS-topic. WhatIf=true reports both what would deploy and what the cleanup would uninstall, without powering anything on.

Daily SU deploy check runbooks

Two additional runbooks drive the automatic path — copy both from environments/dev/terraform.tfvars and adjust the highlighted defaults:

• Get-Installed-RA-Runbook-TF — resolves the ODB EC2 instance by Name tag, runs an iris session query over SSM RunCommand to read the highest installed RA from the ^EMS("RA") global, and returns it as CurrentRA.
• SU-Deploy-Check-Runbook-TF — runs Get-Installed-RA-Runbook, compares the result against /epic/ire/su/last_deployed_ra, and when the ODB's RA is newer, executes Kuiper-Package-Deploy-Runbook for that RA (passing MaxInstalledPackages through) and updates the parameter on success; the deploy runbook itself publishes the success summary to the SU-deploy-SNS-topic. Supports the same WhatIf dry-run parameter (compares and runs a Kuiper WhatIf deploy, but changes nothing).

ssm_runbooks = {
  # ... existing entries ...
  Get-Installed-RA-Runbook-TF = {
    # ... see environments/dev/terraform.tfvars for the full document ...
    # parameters:
    #   OdbInstanceName = {
    #     default = "PRDODB"   # Name tag of the customer's ODB instance
    #   }
    #   IrisInstance = {
    #     default = "prd"      # IRIS instance name (lowercase, as shown by 'iris list')
    #   }
    #   IrisNamespace = {
    #     default = "prd"      # IRIS namespace containing the ^EMS global
    #   }
  }
  SU-Deploy-Check-Runbook-TF = {
    # ... see environments/dev/terraform.tfvars for the full document ...
  }
}

ODB requirements for the RA query

Get-Installed-RA-Runbook requires the ODB instance to be running and SSM-managed (the SSM agent installed and registered), and the iris CLI reachable on the default PATH (/usr/local/bin:/usr/bin). The runbook targets the ODB by its Name tag — verify the default matches the customer's ODB instance, and that the IRIS instance/namespace defaults match the customer's install. If the ODB is powered down at the scheduled check time, the execution fails without deploying anything.

---

3. GitHub Workflow

The Deploy Kuiper Package workflow (.github/workflows/deploy-kuiper-package.yml) starts the runbook by RA number. It uses the repository's existing AWS credentials:

Repository setting	Purpose
secrets.AWS_ACCESS_KEY_ID / secrets.AWS_SECRET_ACCESS_KEY	Credentials able to call ssm:StartAutomationExecution.
vars.AWS_REGION	The customer's deployment region.

No changes are needed if the Terraform plan/apply workflows already run in the repository — the same credentials and region variable are reused.

---

4. Post-Apply Checklist

1. Set the real Kuiper password in /kuiper/api/password (see above). The Lambda reads it at cold start — if the password was set after a failed invocation, wait for the execution environment to recycle or update any Lambda env var to force new cold starts.
2. Fill in KUIPER_CLIENT_ID and KUIPER_TARGET_ENVIRONMENT in the tfvars with the values from the customer's Kuiper install, and apply.
3. Verify the service account is in a Kuiper security group that grants package deployment rights and has local admin on all Hyperspace Web and Interconnect servers.
4. Confirm the SNS email subscription — each endpoint on SU-deploy-SNS-topic receives a confirmation email that must be accepted before notifications are delivered.
5. Dry-run first: execute the Kuiper-Package-Deploy-Runbook from the SSM console with WhatIf=true and an RA number. This calls the Kuiper API and reports what would be deployed and what the retention cleanup would uninstall, without powering servers on or changing anything — it validates connectivity, credentials, the client pack lookup, and the cleanup scope end to end. Then execute SU-Deploy-Check-Runbook with WhatIf=true to validate the ODB RA query and comparison end to end.
6. Run SU-Deploy-Check-Runbook for real (from the SSM console, or wait for the daily schedule). /epic/ire/su/last_deployed_ra starts at 0, so the first run sees the ODB's installed RA as new and performs a full deployment of the current client packs — this is expected and harmless (redeploying an existing client pack is safe), and it confirms the entire path end to end: ODB RA query, power-on, Kuiper deploy, status polling, package cleanup, pilot-light return, and the summary notification. Note the first cleanup may uninstall every old package beyond the retention limit that has accumulated per group (the dry run in the previous step shows the exact list), so the first execution can run long. On completion the parameter records the deployed RA, and the daily check keeps the W&SS servers in sync automatically from then on. The GitHub Actions Deploy Kuiper Package workflow remains available for on-demand deployments of a specific RA.

Failed Kuiper actions can still mean success

The status check treats groups whose deploy actions report Failed as settled rather than fatal: the Epic ODB is typically dormant during an IRE deployment, so Kuiper reports failed actions even though the installation proceeds. complete means "nothing left pending", not "everything succeeded" — review the Kuiper deploy actions and the Lambda logs (/aws/lambda/kuiper_api_function) after each run.