catalyst-onboard CLI

📖 What is catalyst-onboard?

A single CLI command that replaces a multi-step manual process: creating a Harbor registry project, scaffolding a Helm chart from a template, pushing to Git, creating ArgoCD Applications on two clusters, syncing until healthy, creating Cloudflare DNS records, and adding OPNsense WAN firewall rules.

🏛️ Architecture

The CLI orchestrates six external systems in sequence. No local state is stored — everything is discovered at runtime from ArgoCD.

🚀 Quick Start

# Enter the CLI directory and activate the venv
cd git-catalyst-apps/.tooling/catalyst-onboard

# Check connectivity to all systems
.venv/bin/python -m catalyst_onboard.cli doctor

# List used / free VIPs
.venv/bin/python -m catalyst_onboard.cli list-ips

# Preview an onboard without applying
.venv/bin/python -m catalyst_onboard.cli new \
  -n my-app -i my-image -p 8080 -t 1.0.0 --ip 185.33.182.X \
  --dry-run

⌨️ All Commands

doctor

.venv/bin/python -m catalyst_onboard.cli doctor

Checks connectivity and authentication for ArgoCD, Harbor, Cloudflare, and OPNsense.

list-apps

.venv/bin/python -m catalyst_onboard.cli list-apps [--json]

Lists all ArgoCD Applications across both clusters with sync/health status and VIP.

list-ips

.venv/bin/python -m catalyst_onboard.cli list-ips [--json]

Shows used and free VIPs in the BGP pool 185.33.182.30–60. IPs are discovered at runtime from ArgoCD — no local state.

status

.venv/bin/python -m catalyst_onboard.cli status <app-name> [--json]

new

.venv/bin/python -m catalyst_onboard.cli new \
  -n <app-name> -i <harbor-image-name> -p <port> -t <tag> --ip 185.33.182.X
  [--replicas N] [--dry-run] [--update]
  [--skip-harbor] [--skip-scaffold] [--skip-dns] [--skip-opnsense]

delete

.venv/bin/python -m catalyst_onboard.cli delete <app-name> [--yes] [--dry-run]

post-deploy-check

.venv/bin/python -m catalyst_onboard.cli post-deploy-check <app-name>

Probes ArgoCD health, DNS resolution, TCP:443, and HTTPS response.

✨ Onboard Workflow (new)

Steps are executed in a fixed order. Each step has a --skip-* flag. All steps are idempotent — re-running on a partially-failed onboard is the standard recovery path.

✅ Final Deployment Verification

After new or --update, always verify the app is actually serving HTTP 200. ArgoCD reports Healthy based on pod health — it does not prove end-to-end reachability.

# Built-in check (DNS + HTTPS + ArgoCD health)
.venv/bin/python -m catalyst_onboard.cli post-deploy-check <app-name>

# Manual curl (ground truth — bypass DNS with --resolve)
curl -o /dev/null -sw "%{http_code}\n" \
  --resolve "<app>.catalyst.xcoinfra.net:443:<VIP>" \
  https://<app>.catalyst.xcoinfra.net/

If you don't get 200, diagnose in this order:

Testing before DNS propagates

After a new DNS record is created, Cloudflare's resolver (1.1.1.1) propagates within seconds. Google (8.8.8.8) and corporate DNS servers can take up to 300s.

# Check which resolver sees the record
dig <app>.catalyst.xcoinfra.net @1.1.1.1 +short   # Cloudflare
dig <app>.catalyst.xcoinfra.net @8.8.8.8 +short   # Google (may lag)

# Force resolution to known VIP — bypasses DNS entirely
curl -o /dev/null -sw "%{http_code}\n" \
  --resolve "<app>.catalyst.xcoinfra.net:443:185.33.182.X" \
  https://<app>.catalyst.xcoinfra.net/

🔄 Update Workflow (new --update)

Use --update to redeploy an existing app — e.g. after a new image tag or replica change.

Building the image (important on Apple Silicon)

# Always --platform linux/amd64 — K8S nodes are amd64.
# Plain 'docker build' on Mac M1/M2/M3/M4 produces arm64 → ImagePullBackOff.
docker buildx build --platform linux/amd64 \
  -t harbor.catalyst.xcoinfra.net/<project>/<image>:<tag> \
  --push .

New image tag only

.venv/bin/python -m catalyst_onboard.cli new \
  -n myapp -i myimage -p 8080 -t 2.0.0 --ip 185.33.182.35 \
  --update --skip-harbor --skip-scaffold --skip-dns --skip-opnsense

🗑️ Delete Workflow

Removes all resources created by new. Git branch is never touched by design (audit trail).

.venv/bin/python -m catalyst_onboard.cli delete <app-name> [--yes] [--dry-run]
  [--keep-harbor]    # do NOT delete the Harbor project
  [--keep-dns]       # do NOT delete the Cloudflare DNS A record
  [--keep-opnsense]  # do NOT delete the OPNsense WAN rules

Order of operations:

ArgoCD cascade delete (c1 + c2)
  → Harbor: delete all repositories then the project
  → Cloudflare DNS A record
  → OPNsense WAN rules (CATALYST_AUTO_<app>_*)

🔀 Rename / Move to a New IP

The CLI has no rename command. To rename an app or move it to a new VIP:

cd git-catalyst-apps
git push origin --delete feature/<old-app>

docker buildx build --platform linux/amd64 \
  -t harbor.catalyst.xcoinfra.net/<new-app>/<image>:latest --push .

.venv/bin/python -m catalyst_onboard.cli new \
  -n <new-app> -i <image> -p <port> -t latest --ip 185.33.182.X

⚙️ Environment (.env)

Located at .tooling/catalyst-onboard/.env. Gitignored, chmod 600.

# ── ArgoCD ──────────────────────────────────────────
ARGOCD_URL=https://argocd.catalyst.xcoinfra.net
ARGOCD_USER=admin
ARGOCD_PASSWORD=<password>
ARGOCD_PROJECT=dev-project
ARGOCD_VERIFY_TLS=false
ARGOCD_CLUSTER_1_URL=https://172.16.3.29:6443
ARGOCD_CLUSTER_2_URL=https://172.16.3.119:6443

# ── Harbor (use admin — argocd can't delete projects) ─
HARBOR_URL=https://harbor.catalyst.xcoinfra.net
HARBOR_USER=admin
HARBOR_PASSWORD=<password>
HARBOR_VERIFY_TLS=true
HARBOR_MAINTAINER_USER=argocd

# ── Cloudflare ──────────────────────────────────────
CLOUDFLARE_ZONE_ID=<zone-id>
CLOUDFLARE_TOKEN=<token>

# ── OPNsense ────────────────────────────────────────
OPNSENSE_URL=https://109.234.111.251
OPNSENSE_API_KEY=<key>
OPNSENSE_API_SECRET=<secret>
OPNSENSE_VERIFY_TLS=false
OPNSENSE_TEMPLATE_RULE_DESCRIPTION=CATALYST_TEMPLATE_CHANGE_IP

# ── Git ─────────────────────────────────────────────
GITHUB_TOKEN=<PAT>
GIT_CATALYST_APPS_REPO=https://github.com/centralnicgroup/git-catalyst-apps.git
GIT_CATALYST_APPS_DEFAULT_BRANCH=main
GIT_USER_NAME=catalyst-onboard
GIT_USER_EMAIL=catalyst-onboard@xcoinfra.net

# ── BGP pool ─────────────────────────────────────────
BGP_POOL_START=185.33.182.30
BGP_POOL_END=185.33.182.60

# ── Tuning ──────────────────────────────────────────
SYNC_TIMEOUT_SECONDS=300
SYNC_POLL_INTERVAL_SECONDS=5

🐛 Known Bugs & Fixes

⚠️ OPNsense: where to see our rules in the UI

Rules created by the CLI via the os-firewall plugin API do NOT appear on the classic Firewall → Rules → WAN page (firewall_rules.php?if=wan). This is by design — the plugin stores its rules separately.

Both compile into the same pf kernel ruleset — both are active. They just live on different UI pages. If you don't see your rules in Rules → WAN, go to Automation → Filter.

Cancelling a stuck ArgoCD operation

If sync fails with "another operation is already in progress" after all retries, cancel it manually:

TOKEN=$(curl -sk -X POST https://argocd.catalyst.xcoinfra.net/api/v1/session \
  -H 'Content-Type: application/json' \
  -d '{"username":"admin","password":"<pwd>"}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")

curl -sk -X DELETE \
  "https://argocd.catalyst.xcoinfra.net/api/v1/applications/<app>-c1/operation" \
  -H "Authorization: Bearer $TOKEN"

curl -sk -X DELETE \
  "https://argocd.catalyst.xcoinfra.net/api/v1/applications/<app>-c2/operation" \
  -H "Authorization: Bearer $TOKEN"

Then re-run the CLI command that failed.