EKS GitOps hands-on

Canary Tests

Flagger comes with a testing service that can run acceptance and load tests when configured as a webhook.

Define tests

For podinfo, we've setup:

  • an acceptance test by verifying that the app can issue token
  • a load test that generates traffic during the canary analysis
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
spec:
  analysis:
    webhooks:
      - name: acceptance-test
        type: pre-rollout
        url: http://flagger-loadtester.$(NAMESPACE)/
        timeout: 30s
        metadata:
          type: bash
          cmd: "curl -sd 'test' http://podinfo-canary.$(NAMESPACE):9898/token | grep token"
      - name: load-test
        url: http://flagger-loadtester.$(NAMESPACE)/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.$(NAMESPACE):9898/"

Run tests

When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary. If the acceptance test fails, Flagger will retry until the analysis threshold is reached and the canary is rolled back.

If the acceptance test passes, then Flagger will start the load test and begin the traffic shifting.

$ kubectl -n appmesh-system logs deployment/flagger -f | jq .msg

New revision detected! Scaling up podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Advance podinfo.test canary weight 40
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Routing all traffic to primary
Promotion completed! Scaling down podinfo.test

Manual Gating

For manual approval of a canary deployment you can use the confirm-rollout and confirm-promotion webhooks. The confirmation rollout hooks are executed before the pre-rollout hooks. Flagger will halt the canary traffic shifting and analysis until the confirm webhook returns HTTP status 200.

For manual rollback of a canary deployment you can use the rollback webhook. The rollback hook will be called during the analysis and confirmation states. If a rollback webhook returns a successful HTTP status code, Flagger will shift all traffic back to the primary instance and fail the canary.

Manual gating with Flagger's tester:

  analysis:
    webhooks:
      - name: "confirm gate"
        type: confirm-rollout
        url: "http://flagger-loadtester.$(NAMESPACE)/gate/halt"

The /gate/halt returns HTTP 403 thus blocking the rollout.

If you have notifications enabled, Flagger will post a message to Slack or MS Teams if a canary rollout is waiting for approval.

Change the URL to /gate/approve to start the canary analysis:

  analysis:
    webhooks:
      - name: "confirm gate"
        type: confirm-rollout
        url: "http://flagger-loadtester.$(NAMESPACE)/gate/approve"

The confirm-promotion hook type can be used to manually approve the canary promotion. While the promotion is paused, Flagger will continue to run the metrics checks and load tests.

  analysis:
    webhooks:
      - name: "promotion gate"
        type: confirm-promotion
        url: "http://flagger-loadtester.$(NAMESPACE)/gate/halt"

← Cleanup →