DevOps Training Program that will provide you with in-depth knowledge of various DevOps tools including Git, Jenkins, Docker, Ansible, Puppet, Kubernetes and Nagios. This training is completely hands-on and designed in a way to help you become a certified practitioner through best practices in Continuous Development, Continuous Testing, Configuration Management and Continuous Integration, and finally, Continuous Monitoring of software throughout its development life cycle.
How to Configure GitHub Webhooks to Trigger Jenkins Builds Automatically
Configure GitHub to automatically notify Jenkins whenever code is pushed to a repository.
Lab Objective
By the end of this lab, every push to your GitHub repository will automatically trigger a Jenkins build.
You will no longer need to manually click Build Now after every code change.
What is a Webhook?
A webhook is a push-based notification mechanism. Instead of Jenkins repeatedly checking GitHub to determine
whether code has changed, GitHub sends an HTTP POST request to Jenkins immediately after a configured event occurs.
Developer
|
| git push
v
GitHub Repository
|
| Webhook HTTP POST
v
Jenkins
|
| Automatic Build
v
Build / Test / Scan / Deploy
Ways Jenkins Jobs Can Be Triggered
Jenkins jobs can be started in several ways. Three common approaches are:
Manual Build – a user clicks Build Now.
Poll SCM – Jenkins periodically checks the source repository for changes.
Webhook – GitHub immediately notifies Jenkins when a repository event occurs.
Jenkins can also be triggered from other systems and integrations such as chat tools, APIs, schedulers,
upstream jobs, and pipelines. This lab focuses specifically on the GitHub webhook push mechanism.
Poll SCM vs GitHub Webhook
Poll SCM
GitHub Webhook
Jenkins checks GitHub.
GitHub contacts Jenkins.
Pull mechanism.
Push mechanism.
Runs on a schedule.
Runs when the event happens.
Can create unnecessary repository polling.
More event-driven and immediate.
Prerequisites
Before beginning this lab, make sure you have:
A GitHub account.
A GitHub repository containing your application code.
A running Jenkins server.
A Jenkins job already configured to clone your GitHub repository.
Git installed/configured in Jenkins.
The GitHub repository URL configured under Jenkins Source Code Management > Git.
GitHub credentials configured in Jenkins if the repository is private.
A Jenkins URL that GitHub can reach over the network.
Important: A URL such as http://localhost:8080 cannot be used by GitHub.
GitHub must be able to reach your Jenkins server from the internet or through an appropriate webhook relay/network configuration.
Architecture for This Lab
Developer
|
| git add / git commit / git push
v
+----------------------+
| GitHub Repo |
+----------+-----------+
|
| Push Event
| Webhook
v
+----------------------+
| Jenkins |
+----------+-----------+
|
| Checkout Source
v
+----------------------+
| Build / Test / Scan |
+----------+-----------+
|
v
Deployment
Step 1 – Install the GitHub Plugin in Jenkins
Log in to Jenkins and navigate to:
Manage Jenkins → Plugins
Depending on the Jenkins version, you may see tabs such as Available plugins and Installed plugins.
Open Available plugins.
Search for GitHub.
Select the GitHub plugin.
Install the plugin.
Also verify that the normal Git-related Jenkins plugins are available, including:
Git plugin
Git client plugin
GitHub plugin
Credentials plugin
Step 2 – Verify the Jenkins Job Uses the GitHub Repository
Open the Jenkins job that should run when code is pushed.
If the repository is private, select your GitHub credentials.
For HTTPS authentication, a Jenkins credential may look like:
Kind: Username with password
Username: YOUR_GITHUB_USERNAME
Password: YOUR_GITHUB_PERSONAL_ACCESS_TOKEN
Security: Do not enter your normal GitHub password in Jenkins and never commit a Personal Access Token to source control.
Under Branches to build, use the branch your class/project is using. For a repository using the standard main branch:
*/main
Step 3 – Configure the Jenkins GitHub Build Trigger
Remain inside the Jenkins job configuration page and locate:
Build Triggers
Enable:
☑ GitHub hook trigger for GITScm polling
In older Jenkins GitHub plugin versions, this option was called
Build when a change is pushed to GitHub. The current GitHub plugin calls it
GitHub hook trigger for GITScm polling.
Disable Poll SCM
If you previously enabled:
☐ Poll SCM
uncheck it for this lab.
The goal is to use GitHub as the event source instead of making Jenkins continuously poll GitHub.
Click Save.
Step 4 – Determine the Jenkins Webhook URL
The Jenkins GitHub plugin receives GitHub events at:
JENKINS_BASE_URL/github-webhook/
For example, if Jenkins is available at:
http://54.123.45.67:8080
your webhook URL becomes:
http://54.123.45.67:8080/github-webhook/
If Jenkins is behind HTTPS and a domain:
https://jenkins.example.com/github-webhook/
Important: Use /github-webhook/. The trailing slash is recommended and avoids unnecessary webhook endpoint problems.
Step 5 – Make Sure GitHub Can Reach Jenkins
GitHub must be able to make an inbound HTTP/HTTPS request to the Jenkins webhook endpoint.
If Jenkins is running on AWS EC2 for a training lab, verify:
The EC2 instance is running.
The Jenkins service is running.
The correct public IP address or DNS name is being used.
The instance firewall/security rules allow the required inbound traffic.
Any reverse proxy, load balancer, or firewall forwards requests to Jenkins correctly.
Production security note: Do not treat exposing the entire Jenkins administration interface directly to the public internet on port 8080 as a production architecture. Prefer HTTPS and a properly secured reverse proxy/load balancer or another controlled ingress mechanism.
GitHub allows an optional webhook secret. For a simple classroom lab using the standard Jenkins GitHub plugin in
manual webhook mode, you may leave this blank unless you have separately configured secret validation.
In production integrations, webhook secrets are strongly recommended where the receiver supports validating them.
Which events would you like to trigger this webhook?
Select:
◉ Just the push event
For this lab, we only need GitHub to notify Jenkins when code is pushed.
Active
Make sure:
☑ Active
Finally, click:
Add webhook
Step 7 – Verify GitHub's Initial Ping
When the webhook is created, GitHub sends a ping event to the webhook URL.
Return to:
Repository → Settings → Webhooks
Open the webhook and look at:
Recent deliveries
A successful request indicates that GitHub was able to contact the Jenkins webhook endpoint.
Step 8 – Test the Webhook
Do not click Build Now. The point of this lab is to prove that GitHub can trigger Jenkins automatically.
Code Change
|
v
git commit
|
v
git push
|
v
GitHub
|
| push webhook
v
Jenkins
|
v
Automatic Build
If your Jenkins job also contains Maven, SonarQube, artifact packaging, or deployment steps, those stages continue as normal after the webhook starts the job.
Step 10 – Check Jenkins Console Output
Open the new Jenkins build and select:
Console Output
Verify that Jenkins:
Detected the GitHub-triggered change.
Checked out the repository.
Built the expected branch.
Completed the configured build steps.
Troubleshooting
Problem 1 – GitHub Cannot Connect to Jenkins
If GitHub reports that it cannot connect, verify:
Jenkins is running.
The Jenkins public IP/DNS is correct.
The webhook URL uses the correct port.
The EC2 Security Group/firewall allows the required request.
The URL is not using localhost or a private-only address.
Problem 2 – 404 Not Found
Verify that the webhook URL ends with:
/github-webhook/
Example:
http://54.123.45.67:8080/github-webhook/
Problem 3 – GitHub Delivery Succeeds but Jenkins Does Not Build
Go to:
Jenkins Job → Configure → Build Triggers
Verify:
☑ GitHub hook trigger for GITScm polling
Also verify that the repository URL configured under Jenkins SCM matches the repository sending the webhook.
Problem 4 – Jenkins Starts but Git Checkout Fails
If the webhook triggers Jenkins but checkout fails, the webhook is working. Check:
Repository URL.
GitHub username.
GitHub Personal Access Token for private repository access.
Token repository permissions.
Jenkins credential selection.
Problem 5 – Wrong Branch
If Jenkins is configured for:
*/main
make sure you test the lab by pushing to the main branch.
For each delivery, you can inspect information such as:
Event type
Request headers
Payload
Response status
Response body
You can also redeliver a recent webhook request from GitHub while troubleshooting, which is useful because you do not always need to create a new commit for every test.
Expected Result
After completing the lab, this workflow should happen automatically:
Developer changes code
|
v
git add .
|
v
git commit
|
v
git push origin main
|
v
GitHub receives the push
|
v
GitHub sends webhook
|
v
Jenkins receives /github-webhook/
|
v
Jenkins identifies the matching job
|
v
Jenkins checks out the code
|
v
Jenkins runs the build automatically
Lab Validation Checklist
☐ GitHub plugin installed in Jenkins
☐ Jenkins job uses the correct GitHub repository
☐ Correct branch configured
☐ GitHub hook trigger for GITScm polling enabled
☐ Poll SCM disabled for this lab
☐ GitHub webhook created
☐ Payload URL ends in /github-webhook/
☐ Content type set to application/json
☐ Push event selected
☐ Webhook is Active
☐ GitHub can reach Jenkins
☐ A code push starts Jenkins automatically
☐ Jenkins successfully checks out and builds the code
Interview / Review Questions
What is a webhook?
What is the difference between Poll SCM and a webhook?
Why must Jenkins be reachable from GitHub?
What Jenkins endpoint receives GitHub webhook events?
Which Jenkins build trigger is used for GitHub webhooks?
Which GitHub event did we configure in this lab?
How can you verify whether GitHub successfully delivered a webhook?
If GitHub reports a successful delivery but Jenkins does not start, what would you check?
If Jenkins starts but cannot clone a private repository, is that primarily a webhook problem or a Git authentication problem?
Summary
In this lab, we configured an event-driven Continuous Integration trigger between GitHub and Jenkins.
GitHub now sends a webhook whenever code is pushed to the configured repository, and Jenkins responds by
checking the matching Git repository and starting the configured job automatically.
Prerequisite: Complete TASK-F-GITLAB-CI-ONLY-GUIDE.md first. Your CI pipeline must be green — image in the registry, all stages passing — before starting this guide.
This guide adds the CD (Continuous Delivery) layer. When complete, every push to develop automatically updates the dev environment. Pushes to release/* branches update UAT. Production is promoted manually via a merge request.
By the end of this guide you will have:
Two new CI stages: update-dev and update-uat that commit image tags to your CD repo
Argo CD watching your CD repo and automatically deploying to healthpulse-dev
Manual-sync gates for UAT and prod
A prod promotion workflow via merge request — no pipeline ever writes to prod directly
The GitOps Model — Two Repos
GitOps means Git is the source of truth for what is deployed. Instead of CI running kubectl apply directly, CI writes to Git and a separate tool (Argo CD) applies what Git says.
CI Repo (source code) CD Repo (kubernetes manifests)
──────────────────── ──────────────────────────────
src/ kubernetes/
.gitlab-ci.yml app-manifests/
Dockerfile deployment.yml
service.yml
On every push to develop: kustomization.yaml
1. CI builds the image ├── overlays/
2. CI pushes to registry │ ├── dev/kustomization.yaml ← CI updates newTag here
3. CI clones CD repo │ ├── uat/kustomization.yaml ← CI updates on release/*
4. CI runs kustomize edit │ └── prod/kustomization.yaml ← PR only, never by CI
5. CI commits + pushes └── argocd/
healthpulse-dev.yml
healthpulse-uat.yml
healthpulse-prod.yml
Why two repos?
Reason
Detail
Audit trail
Every deployment is a commit in the CD repo — who deployed what, when, and why
Rollback
Revert the CD repo commit to roll back a deployment
Access control
Developers commit to the CI repo; only ops/leads can merge to the CD repo's prod overlay
No drift
Argo CD continuously reconciles — if someone does a manual kubectl apply, Argo CD reverts it
Pipeline Overview — CI + CD
Adding CD extends the pipeline with two new stages at the end:
install
└── npm ci
test (parallel)
├── lint
└── unit-tests
scan (parallel)
├── gitleaks
├── sonarqube
└── snyk-security
build
└── build-app
publish (parallel)
├── docker-publish
└── artifactory-upload
scan-image
└── image-scan
update-dev ← NEW: runs on 'develop' branch only
└── update-dev-manifest — clones CD repo, updates dev overlay, commits
update-uat ← NEW: runs on 'release/*' branches only
└── update-uat-manifest — clones CD repo, updates uat overlay, commits
Argo CD then picks up the commit and deploys:
CD repo commit (by CI)
│
▼
Argo CD detects new image tag in overlays/dev/kustomization.yaml
│
▼
healthpulse-dev: AUTO-SYNC → deploys immediately
healthpulse-uat: MANUAL-SYNC → Argo CD shows OutOfSync, operator clicks Sync
healthpulse-prod: MANUAL-SYNC + PR gate → no commit without a merged PR
Step 1: Set Up the CD Repo
1.1 — Create the CD repo in GitLab
Go to GitLab → New project → Create blank project
Name it healthpulse-cd
Set visibility to Private
Do not initialise with a README
1.2 — Copy the Kubernetes manifests into your CD repo
The capstone provides the full Kubernetes manifest structure. Clone your CD repo and copy the kubernetes/ directory into it:
git clone https://gitlab.com/<your-group>/healthpulse-cd.git
cd healthpulse-cd
# Copy the kubernetes/ directory from the capstone into this repo
cp -r /path/to/healthpulse-capstone/kubernetes .
git add kubernetes/
git commit -m "feat: add kubernetes manifests and kustomize overlays"
git push origin main
Your CD repo should now look like:
healthpulse-cd/
└── kubernetes/
├── app-manifests/
│ ├── deployment.yml
│ ├── service.yml
│ ├── hpa.yml
│ └── kustomization.yaml
├── overlays/
│ ├── dev/
│ │ ├── kustomization.yaml ← CI will update newTag here
│ │ ├── namespace.yml
│ │ └── ingress.yml
│ ├── uat/
│ │ ├── kustomization.yaml ← CI will update newTag here on release/*
│ │ ├── namespace.yml
│ │ └── ingress.yml
│ └── prod/
│ ├── kustomization.yaml ← NEVER updated by CI
│ ├── namespace.yml
│ └── ingress.yml
└── argocd/
├── healthpulse-dev.yml
├── healthpulse-uat.yml
└── healthpulse-prod.yml
1.3 — Update the <REGISTRY> placeholder in overlay kustomization files
Each overlay's kustomization.yaml has a <REGISTRY> placeholder. Replace it with your actual $CI_REGISTRY_IMAGE value.
Find your registry image path: go to your CI repo → Deploy → Container Registry — the path shown is your CI_REGISTRY_IMAGE value (e.g. registry.gitlab.com/your-group/devops-projects).
kubernetes/overlays/dev/kustomization.yaml — update the newName line:
Do the same for overlays/uat/kustomization.yaml and overlays/prod/kustomization.yaml.
Commit and push:
git add kubernetes/overlays/
git commit -m "fix: set registry image path in kustomize overlays"
git push origin main
Step 2: Understanding Kustomize
What Kustomize does
Kustomize is a Kubernetes configuration tool built into kubectl. It lets you define a base set of manifests and overlays per environment — without duplicating YAML or using templates.
kubernetes/app-manifests/ ← base: shared across all environments
deployment.yml Defines the Deployment, Service, HPA
service.yml Does NOT set namespace or image tag
hpa.yml
kustomization.yaml
kubernetes/overlays/dev/ ← overlay: dev-specific overrides only
kustomization.yaml Sets namespace + image tag for dev
namespace.yml Creates the healthpulse-dev namespace
ingress.yml Dev-specific hostname
How CI updates the image tag
The CI job runs one command:
kustomize edit set image healthpulse-portal=$CI_REGISTRY_IMAGE/$APP_NAME:$BUILD_VERSION
This updates newTag: in overlays/dev/kustomization.yaml from:
images:
- name: healthpulse-portalnewName: registry.gitlab.com/your-group/devops-projects/healthpulse-portalnewTag: latest ← before
to:
images:
- name: healthpulse-portalnewName: registry.gitlab.com/your-group/devops-projects/healthpulse-portalnewTag: 11-5dd92339 ← after (the new version)
Argo CD detects this file change in Git and applies the updated image tag to the cluster — pulling the new image and rolling out a new deployment.
What kustomize build produces
You can preview what Kustomize generates before applying:
# Preview what the dev overlay produces
kustomize build kubernetes/overlays/dev
# Apply directly to the cluster
kustomize build kubernetes/overlays/dev | kubectl apply -f -
Argo CD runs the equivalent of kustomize build + kubectl apply automatically.
Step 3: Add CD Variables to the CI Repo
These variables go in your CI repo — they are used by the CI pipeline to authenticate with and push to the CD repo.
Go to your CI repo → Settings → CI/CD → Variables.
CD_REPO_TOKEN — Personal Access Token for the CD repo
CI_JOB_TOKEN only allows pushing to the repo the pipeline is running in. To push to the CD repo you need a separate token.
Go to GitLab → your avatar (top right) → Edit profile → Access tokens
Click Add new token
Name: gitlab-ci-cd-push
Expiry: set an appropriate date
Scope: tick write_repository
Click Create personal access token
Copy the token — you will not see it again
In your CI repo Variables, add:
Key
Value
Protected
Masked
CD_REPO_TOKEN
The token you just copied
Yes
Yes
CD_REPO_URL
gitlab.com/your-group/healthpulse-cd
No
No
CD_REPO_URL format: Do NOT include https:// — the job prepends it. Just the domain and path: gitlab.com/your-group/healthpulse-cd
Step 4: Add CD Stages to .gitlab-ci.yml
Add two new stages and two new jobs to your existing .gitlab-ci.yml in the CI repo.
4.1 — Add the new stages
stages:
- install
- test
- scan
- build
- publish
- scan-image
- update-dev # ← NEW: updates dev overlay on develop branch
- update-uat # ← NEW: updates uat overlay on release/* branches
4.2 — Add the update-dev-manifest job
# =============================================================# STAGE: UPDATE-DEV# Runs ONLY on the 'develop' branch, NEVER on CI-generated commits.## What it does:# 1. Clones the CD repo# 2. Runs kustomize edit set image — updates newTag in overlays/dev/# 3. Commits and pushes with [skip ci] to prevent an infinite loop## After this job, Argo CD detects the new tag in Git and auto-deploys# to the healthpulse-dev namespace.# =============================================================update-dev-manifest:
stage: update-devimage: alpine/git:v2.52.0needs: [docker-publish]rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[(skip ci|ci skip)\]/i'when: never # Never run on CI-generated commits
- if: '$CI_COMMIT_BRANCH == "develop"'# Only run on the develop branchvariables:
BUILD_VERSION: "${CI_PIPELINE_IID}-${CI_COMMIT_SHORT_SHA}"before_script:
# Install kustomize v5.8.1 (latest stable) via direct GitHub release download.# Pinned version avoids breakage from silent upstream changes.# Do NOT use the hack/install_kustomize.sh script — it parses the GitHub API# and has a history of intermittent failures when the API response format changes.
- apk add --no-cache curl
- curl -Lo kustomize.tar.gz https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.8.1/kustomize_v5.8.1_linux_amd64.tar.gz
- tar -xzf kustomize.tar.gz
- mv kustomize /usr/local/bin/
- kustomize versionscript:
# Clone the CD repo using the personal access token
- git clone "https://oauth2:${CD_REPO_TOKEN}@${CD_REPO_URL}.git" cd-repo
- cd cd-repo
- git config user.email "gitlab-ci@healthpulse.com"
- git config user.name "GitLab CI"# Update the image tag in the dev overlay
- cd kubernetes/overlays/dev
- kustomize edit set image healthpulse-portal=${CI_REGISTRY_IMAGE}/${APP_NAME}:${BUILD_VERSION}
- git add kustomization.yaml# [skip ci] prevents this commit from triggering a new pipeline in the CD repo
- git commit -m "ci: deploy dev -> ${BUILD_VERSION} [skip ci]"
- git pushenvironment:
name: development
4.3 — Add the update-uat-manifest job
# =============================================================# STAGE: UPDATE-UAT# Runs ONLY on 'release/*' branches, NEVER on CI-generated commits.## After this job, Argo CD shows healthpulse-uat as OutOfSync.# The operator logs into Argo CD and clicks Sync to approve the deploy.# Nothing deploys to UAT without a human action.# =============================================================update-uat-manifest:
stage: update-uatimage: alpine/git:v2.52.0needs: [docker-publish]rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[skip ci\]/'when: never
- if: '$CI_COMMIT_BRANCH =~ /^release\//'# Only on release/x.y.z branchesvariables:
BUILD_VERSION: "${CI_PIPELINE_IID}-${CI_COMMIT_SHORT_SHA}"before_script:
- apk add --no-cache curl
- curl -Lo kustomize.tar.gz https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.8.1/kustomize_v5.8.1_linux_amd64.tar.gz
- tar -xzf kustomize.tar.gz
- mv kustomize /usr/local/bin/script:
- git clone "https://oauth2:${CD_REPO_TOKEN}@${CD_REPO_URL}.git" cd-repo
- cd cd-repo
- git config user.email "gitlab-ci@healthpulse.com"
- git config user.name "GitLab CI"
- cd kubernetes/overlays/uat
- kustomize edit set image healthpulse-portal=${CI_REGISTRY_IMAGE}/${APP_NAME}:${BUILD_VERSION}
- git add kustomization.yaml
- git commit -m "ci: deploy uat -> ${BUILD_VERSION} [skip ci]"
- git pushenvironment:
name: uat
4.4 — The [skip ci] mechanism — preventing an infinite loop
Without [skip ci], the pipeline would loop forever:
Developer pushes to develop
│
▼
CI builds image, updates CD repo
│
▼
CD repo push triggers a new pipeline ← LOOP
│
▼
New pipeline builds new image... ← INFINITE
Two safeguards stop this:
Safeguard 1 — [skip ci] in the commit message: GitLab reads the commit message natively. If it contains [skip ci] or [ci skip] (case-insensitive), GitLab skips the pipeline entirely for that commit. This is a built-in GitLab feature — no configuration required.
git commit -m "ci: deploy dev -> 12-abc1234 [skip ci]"
Safeguard 2 — rules: when: never: The jobs have this as their first rule:
- if: '$CI_COMMIT_MESSAGE =~ /\[(skip ci|ci skip)\]/i'when: never
The i flag makes the match case-insensitive. Covering both [skip ci] and [ci skip] ensures the guard works regardless of which variant is used. Even if a pipeline somehow starts on one of these commits, these jobs will not run.
The when: never rule must be first in the rules list. Rules are evaluated in order — the first match wins.
Modern alternative — git push -o ci.skip: Git push options (Git 2.10+) offer a cleaner approach that doesn't embed text in the commit message:
git push -o ci.skip
For this guide's CI-generated commits, we use [skip ci] in the message because it is more portable — it works across all Git client versions and also skips merge request pipelines, which push options do not.
Step 5: Set Up Argo CD Applications
Argo CD Application CRDs tell Argo CD which Git repo and path to watch, and which Kubernetes cluster to deploy to.
5.1 — Update the Argo CD Application files
Before applying, update the repoURL in each Argo CD Application file in your CD repo. Replace the GitHub placeholder with your actual GitLab CD repo URL.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata:
name: healthpulse-uatnamespace: argocdspec:
project: defaultsource:
repoURL: https://gitlab.com/your-group/healthpulse-cd.git # ← your CD repotargetRevision: mainpath: kubernetes/overlays/uatdestination:
server: https://kubernetes.default.svcnamespace: healthpulse-uatsyncPolicy:
syncOptions:
- CreateNamespace=true# No 'automated' block — operator must click Sync in Argo CD UI
kubernetes/argocd/healthpulse-prod.yml:
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata:
name: healthpulse-prodnamespace: argocdspec:
project: defaultsource:
repoURL: https://gitlab.com/your-group/healthpulse-cd.git # ← your CD repotargetRevision: mainpath: kubernetes/overlays/proddestination:
server: https://kubernetes.default.svcnamespace: healthpulse-prodsyncPolicy:
syncOptions:
- CreateNamespace=true# No 'automated' block — manual sync required# Prod overlay is NEVER updated by CI — only via merged PR
5.2 — Connect Argo CD to GitLab (private repo access)
Argo CD needs to pull from your private GitLab CD repo. Add the repo credentials in the Argo CD UI:
Username: your GitLab username (or the token name if using a Deploy Token)
Password: a token with read_repository scope (see token options below)
Click Connect — verify it shows Successful
Which GitLab token to use:
Token type
Where to create
Recommended?
Project Deploy Token
CD repo → Settings → Repository → Deploy tokens
✅ Best — scoped to this repo only, not tied to a user account
Project Access Token
CD repo → Settings → Access Tokens
✅ Good — repo-scoped, survives user account changes
Personal Access Token
GitLab → User Settings → Access Tokens
⚠️ Works but tied to your account — rotates if you leave the org
For a Deploy Token: create it with read_repository scope, use the token username as the Username field and the token value as the Password field.
Argo CD version note: This guide is validated against Argo CD 3.x (latest stable). The Application CRD uses apiVersion: argoproj.io/v1alpha1 — this is still the only available version and has not changed. If you are upgrading from Argo CD 2.x, review the 2.14 → 3.0 upgrade guide for RBAC breaking changes before applying these manifests.
5.3 — Apply the Application CRDs to your cluster
Run these once from a machine with kubectl access to the cluster where Argo CD is installed:
kubectl get applications -n argocd
# Expected:# NAME SYNC STATUS HEALTH STATUS# healthpulse-dev Synced Healthy# healthpulse-uat Unknown Missing# healthpulse-prod Unknown Missing
healthpulse-dev should go Synced / Healthy immediately (it reads whatever newTag is already in the overlay). UAT and prod will show Missing until a sync is triggered.
Step 6: Verify the Full Dev Flow
6.1 — Push to develop and watch the pipeline
git checkout develop
# make any small change (e.g. update a comment)
git add .
git commit -m "feat: test full CI/CD pipeline"
git push origin develop
This usually takes 1-3 minutes. Click on the application to see the sync history — the most recent entry will show ci: deploy dev -> 12-abc1234 [skip ci].
Expected pipeline:update-uat-manifest runs instead of update-dev-manifest. The job commits the new image tag to overlays/uat/kustomization.yaml.
7.2 — Approve the UAT deployment in Argo CD
Go to Argo CD UI → healthpulse-uat application. It shows OutOfSync — the Git state has a new image tag that is not yet deployed.
Click Sync → Synchronize to approve the deployment. Argo CD pulls the new image and rolls it out to healthpulse-uat.
kubectl get pods -n healthpulse-uat
# Expected: pods Running with the new image tag
Why manual sync for UAT? Dev is auto-deploy because it is the feedback loop for developers — every push should be immediately visible. UAT is a gate — someone has decided this build is ready for testing and actively approves it going to UAT. The Sync click is that approval.
Step 8: Prod Promotion — PR-Based Workflow
Production is never updated by CI directly. No stage in the pipeline writes to overlays/prod/. Prod is promoted manually through a merge request:
8.1 — The promotion flow
1. UAT has been tested and signed off
2. Open a MR in the CD repo:
- Copy the newTag value from overlays/uat/kustomization.yaml
- Paste it into overlays/prod/kustomization.yaml
- Title: "promote: prod -> 12-abc1234"
3. Team lead reviews and merges the MR
4. Operator goes to Argo CD → healthpulse-prod → clicks Sync
5. Production deploys
8.2 — Example PR diff
# kubernetes/overlays/prod/kustomization.yaml
images:
- name: healthpulse-portal
newName: registry.gitlab.com/your-group/devops-projects/healthpulse-portal
- newTag: 10-5dd92339 ← current prod+ newTag: 12-abc1234 ← promoting from UAT
8.3 — Why no CI stage for prod?
Concern
How this addresses it
Accidental deploy
A human must open a PR, another must review and merge
Audit trail
The MR records who promoted, when, and links to the pipeline that produced the image
Rollback
Revert the MR to roll back prod to the previous tag
Separation of duties
Developers push code; leads control what goes to prod
Acceptance Criteria
Before marking this step complete, verify every item:
CD repo created (healthpulse-cd) with kubernetes/ directory committed
<REGISTRY> placeholders replaced with actual registry path in all three overlay kustomization.yaml files
CD_REPO_TOKEN variable added to CI repo (masked, protected, write_repository scope)
CD_REPO_URL variable added to CI repo (no https:// prefix)
update-dev and update-uat stages added to .gitlab-ci.yml
update-dev-manifest and update-uat-manifest jobs added with correct rules:
Argo CD connected to the CD repo (private repo credentials added in Settings → Repositories)
Argo CD Application CRDs applied: healthpulse-dev, healthpulse-uat, healthpulse-prod
Push to develop triggers full pipeline — update-dev-manifest commits new tag to CD repo
No infinite loop — the [skip ci] commit does not trigger another pipeline
Argo CD auto-syncs healthpulse-dev within 3 minutes of CD repo commit
kubectl get pods -n healthpulse-dev shows pods running the new image tag
Push to release/1.0.0 — update-uat-manifest commits tag to overlays/uat/
Argo CD shows healthpulse-uat as OutOfSync after the commit
Operator clicks Sync — UAT deploys successfully
Prod promoted via MR in CD repo — team lead reviews, merges, operator syncs in Argo CD
Troubleshooting
update-dev-manifest fails — "authentication failed" or "repository not found"
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.com/your-group/healthpulse-cd.git/'
Check three things:
CD_REPO_TOKEN scope — the token must have write_repository scope. A read_repository-only token can clone but cannot push.
CD_REPO_URL format — must NOT include https://. The job prepends it:
# Correct:CD_REPO_URL: gitlab.com/your-group/healthpulse-cd# Wrong — will produce https://https://...:CD_REPO_URL: https://gitlab.com/your-group/healthpulse-cd
Token expiry — if the token has expired, generate a new one and update the variable.
update-dev-manifest fails — "kustomize: not found"
/bin/sh: kustomize: not found
The before_script install failed. Check the full job log — look for the curl or tar step that errored.
The guide uses a direct pinned release download (v5.8.1). If this fails, check:
Wrong architecture — the guide uses linux_amd64. If your runner is ARM64:
Version no longer available — check the kustomize releases page for the latest version and update the URL.
Network issue in CI — if the GitHub releases URL is blocked, mirror the binary into Artifactory and download from there.
Do not use hack/install_kustomize.sh — this script parses the GitHub API to find the latest release binary and has a documented history of intermittent failures (GitHub issues #4769, #3571) when the API response format changes. The direct download used in this guide is reliable and reproducible.
CI infinite loop — pipelines keep triggering each other
The [skip ci] tag is missing from the commit message, or the rules: are in the wrong order.
Check 1 — commit message: Go to the CD repo → Commits. Does the commit from CI contain [skip ci]?
Check 2 — rules order: The when: never rule must be first:
rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[(skip ci|ci skip)\]/i'when: never # ← MUST be first
- if: '$CI_COMMIT_BRANCH == "develop"'
If the branch rule comes first, GitLab matches it on the [skip ci] commit and runs the job anyway.
Argo CD shows healthpulse-dev as OutOfSync but won't auto-sync
Auto-sync is configured in healthpulse-dev.yml via the automated: block. Check that:
The Application CRD was applied with the automated: block present:
syncPolicy:
automated:
prune: trueselfHeal: true
Re-apply if needed: kubectl apply -f kubernetes/argocd/healthpulse-dev.yml
Check Argo CD has repo access — go to Settings → Repositories, verify the connection is Successful
Argo CD cannot connect to the CD repo — "authentication required"
Go to Argo CD UI → Settings → Repositories. If the CD repo shows a failed connection:
Delete the existing repo connection
Re-add it with a fresh Personal Access Token (minimum scope: read_repository)
Verify: the status should show Successful
kustomize edit set image command runs but tag is not updated in the file
This usually means the name: in the images: block of kustomization.yaml does not match what you passed to kustomize edit set image.
The command:
kustomize edit set image healthpulse-portal=${CI_REGISTRY_IMAGE}/${APP_NAME}:${BUILD_VERSION}
Updates the entry where name: healthpulse-portal. If your kustomization.yaml has a different name:
images:
- name: my-app ← must match the name in the kustomize commandnewName: registry.../healthpulse-portalnewTag: latest
Fix: either update the name: in kustomization.yaml to healthpulse-portal, or update the kustomize command to use the name that matches.
Pods not updating after Argo CD sync
kubectl describe pod <pod-name> -n healthpulse-dev | grep Image
# Shows old image tag
Check that the imagePullPolicy in deployment.yml allows pulling a new image:
Always — always pulls (use for latest or mutable tags)
IfNotPresent — only pulls if not cached (safe for immutable version tags like 12-abc1234)
Since the pipeline uses immutable tags (pipeline-sha), IfNotPresent is correct. If pods are not updating, it may mean the new tag is not actually different from w
