Jenkins & Enterprise CI/CD

Triggers, Webhooks & Multibranch

18 min Lesson 8 of 28

Triggers, Webhooks & Multibranch

Polling is the training-wheels approach to CI: Jenkins wakes up every few minutes, asks your SCM "anything new?", and burns CPU doing it whether or not a developer has touched a line of code. In a fleet of hundreds of pipelines that translates to thousands of wasted API calls per hour. The production answer is event-driven triggering — your SCM pushes a webhook the moment a commit or pull request lands, and Jenkins reacts in under a second. Combine that with Multibranch Pipelines and you get the full big-tech CI model: every branch and every PR is automatically discovered, built, and reported without any human configuring a new job.

How SCM Webhooks Work

A webhook is an HTTPS POST request your SCM (GitHub, GitLab, Bitbucket) sends to a URL you register. Jenkins exposes that URL at /github-webhook/, /gitlab/notify_commit, or the generic /generic-webhook-trigger/invoke depending on which plugin you install. The flow is:

Developer pushes a commit or opens a PR.
SCM signs the payload with an HMAC-SHA256 secret and POSTs to Jenkins.
Jenkins validates the signature and identifies which job(s) match the repository URL.
The matching pipeline is queued immediately — end-to-end latency is typically under two seconds.

Webhook-driven CI: a push event triggers Jenkins in under two seconds; the result posts back to the SCM as a commit status check.

Registering a GitHub Webhook

In your GitHub repo: Settings → Webhooks → Add webhook. Set the Payload URL to https://<your-jenkins>/github-webhook/, Content type to application/json, and paste the secret you generated. Select Just the push event or Let me select individual events (push + pull_request covers most pipelines). Jenkins validates every incoming request against that secret; requests with a missing or wrong signature are silently dropped.

Inbound firewall rules. Jenkins must be reachable from GitHub's published webhook IP ranges (api.github.com/meta lists them under hooks). In a private corporate setup, route through a reverse proxy (nginx / AWS ALB) with an allowlist rather than exposing Jenkins directly to the internet.

In the Declarative Pipeline, declare the trigger so the configuration is version-controlled alongside the code:

pipeline {
  agent any

  triggers {
    githubPush()                    // fires on every push via webhook
    // For GitLab:  gitlab(triggerOnPush: true, triggerOnMergeRequest: true)
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
  }
}

After saving this Jenkinsfile, click Build Now once to register the trigger. From that point the pipeline starts automatically on every push.

Pull Request Builds & Commit Status Checks

PR builds are the safety net that makes trunk-based development practical at scale. When a developer opens a PR, the SCM fires a pull_request webhook. Jenkins checks out the merge commit (what the branch would look like merged into the base), runs the full pipeline, and then calls the SCM's commit-status API to post a required status check. GitHub will block the merge button until that check turns green.

The GitHub Branch Source plugin handles this automatically when you use a Multibranch Pipeline — you do not have to configure commit-status posting manually.

Multibranch Pipelines

A Multibranch Pipeline is a folder of jobs that Jenkins creates and destroys automatically as branches appear and disappear in your repository. Every branch that contains a Jenkinsfile gets its own build history, its own credential scope, and its own set of environment variables (BRANCH_NAME, CHANGE_ID, CHANGE_TARGET). This is the correct unit of CI at scale — not one job per branch configured by hand.

Create one via New Item → Multibranch Pipeline. The key sections are:

Branch Sources — add a GitHub/GitLab source, provide the repo URL, and select the credentials (a fine-grained PAT with contents:read and statuses:write is enough).
Behaviors — discover branches, discover PRs from forks (merge with target), prune stale branches.
Build Configuration — by Jenkinsfile, path Jenkinsfile (or a custom path for monorepos).
Scan Triggers — set to 1 day as a catch-all; the webhook is your real trigger.

# Jenkinsfile snippet — branch-conditional stages
pipeline {
  agent any

  stages {
    stage('Test') {
      steps { sh './gradlew test' }
    }

    stage('Deploy to Staging') {
      when { branch 'main' }         // only on the main branch
      steps { sh './deploy.sh staging' }
    }

    stage('Deploy to Production') {
      when { branch 'release/*' }    // only on release/* branches
      steps { sh './deploy.sh production' }
    }

    stage('PR Smoke Tests') {
      when { changeRequest() }       // only on PR builds
      steps { sh './smoke-tests.sh' }
    }
  }
}

Use changeRequest() to gate expensive tests. Unit tests run on every push; integration tests and container scans are expensive — gate them with when { not { changeRequest() } } so they only run on merged code, keeping PR feedback fast.

Organization Folders (GitHub Organization Plugin)

If your company has dozens of repositories, create a single GitHub Organization folder instead of one Multibranch Pipeline per repo. Jenkins will scan every repository in the org, find any that contain a Jenkinsfile, and create a Multibranch Pipeline for each automatically. New repos are picked up on the next scan (or via webhook). This is how large engineering organisations manage hundreds of pipelines with near-zero admin overhead.

Production Failure Modes

Webhook delivery fails silently. GitHub retries failed webhooks three times with exponential back-off, then stops. Always check Settings → Webhooks → Recent Deliveries when a push did not trigger a build. The most common cause is Jenkins being behind a firewall that did not allowlist GitHub's IP ranges.

Branch scan storms. When you first add a GitHub Organization source with 200 repositories, Jenkins launches 200 concurrent scans. Cap concurrency with the Throttle Concurrent Builds plugin or by setting the organisation's orphanedItemStrategy scan interval to hourly and relying on webhooks for real-time triggers.

Never store the webhook secret in the Jenkinsfile. Keep it in a Jenkins credential (Secret Text kind), reference it in the GitHub Branch Source configuration, and rotate it via your secrets management workflow. A leaked webhook secret lets an attacker trigger arbitrary pipeline runs.

By combining SCM webhooks, Multibranch Pipelines, and commit-status checks you get the full production CI feedback loop: every commit on every branch is verified within seconds, PR authors see pass/fail before anyone reviews the code, and the main branch is always protected by required checks. That is the standard every major tech company operates to.