Types of extended tests

The extended test quality gate is for any test that you do not want to run on every commit e.g.

• Longer peak load tests
• Soak tests

Running these tests can be expensive and typically these are run once a day on the latest version of the application that has been promoted from fast feedback.

These are typically based on Non-Functional-Requirements (NFRs) which should specify what’s the average load expected, peak traffic (both requests per second and concurrent users) and response times in the different percentiles. To simulate processing time of a downstream application, a delay can be added on all integration endpoints.

Types of test

Peak Load Test

Taken from the peak traffic. These tests typically run for ~1h and ensure the application under peak traffic does not exceed the expected response times.

Soak Test

Tests based on the average throughput from the NFRs. These tests typically run longer than peak, running from 4 h duration to a 24/7 environment. They are good to evaluate the stability of the system, catching any degradation of the system like memory leaks.

Rolling Update

This test ensures that a new deployment is successful and doesn’t cause errors even when the ingress controller pods restart. To validate that, the test should ensure that it:

• Stops accepting new connections.
• Processes the inbound connection that it has already accepted.
• Exits after everything is processed with exit code 0.

Resilience Tests

This type of test, like the name says, is used to test the resilience of the system. Taking the same example of the Ingress feature, we can use this to remove all pods in a certain availability zone and ensure that all requests are successful.

How to use this on the pipeline?

These should be triggered by a cron on git actions, and the pipeline will look like:

Usage

name: Extended Test

on:
  workflow_dispatch:
  schedule:
    - cron: '30 22 * * *'

permissions:
  contents: read
  id-token: write

jobs:
  get-latest-version:
    uses: coreeng/p2p/.github/workflows/p2p-get-latest-image-extended-test.yaml@v1
    with:
      image-name: knowledge-platform

  extendedtests:
    needs: [get-latest-version]
    uses: coreeng/p2p/.github/workflows/p2p-workflow-extended-test.yaml@v1
    with:
      version: ${{ needs.get-latest-version.outputs.version }}

This task will get the latest version that’s on the /extended-test registry and execute the extended tests. If these are successful, it will promote the image to prod.

Promotion

Promotion is the process of marking an immutable, versioned artifact as being ready for deployment to the next class of environment.

What is used to promote?

The two main mechanisms for promotion are:

• Tests e.g. functional or non-functional tests
• Stability in an environment e.g. run a longer soak tests in a stage environment and check alerts or canary deployments

Both are possible with the Core Platform P2P. The reference steps all show how to do test based promotion, future reference apps will show alert based promotion in later environments.

Promotion Mechanism

The Core Platform uses Container Registry artifact copying as the promotion mechanism.

All reference applications and skeletons come with a helper task to do this:

.PHONY: p2p-promote-generic
p2p-promote-generic:  ## Generic promote functionality
    corectl p2p promote $(image_name):${image_tag} \
        --source-stage $(source_repo_path) \
        --dest-registry $(REGISTRY) \
        --dest-stage $(dest_repo_path)

The task:

• Authenticates with the source registry
• Pulls the image to be promoted
• Tags the image with the destination registry
• Authenticates with the destination registry
• Pushes the image to the destination registry

The P2P sets all the variables:

• SOURCE_REGISTRY
• REGISTRY for the destination registry
• SOURCE_AUTH_OVERRIDE
• DEST_AUTH_OVERRIDE

corectl p2p promote defaults to environment variables set by P2P, so you don’t need to set them explicitly.

Here is the full list of flags:

corectl p2p promote -h
Promotes image from source to destination registry. Only GCP is supported for now

Usage:
  corectl p2p promote <image_with_tag> [flags]

Flags:
      --dest-auth-override string     optional, defaults to environment variable: DEST_AUTH_OVERRIDE
      --dest-registry string          required, defaults to environment variable: DEST_REGISTRY
      --dest-stage string             required, defaults to environment variable: DEST_STAGE
  -h, --help                          help for promote
      --source-auth-override string   optional, defaults to environment variable: SOURCE_AUTH_OVERRIDE
      --source-registry string        required, defaults to environment variable: SOURCE_REGISTRY
      --source-stage string           required, defaults to environment variable: SOURCE_STAGE

For lower environments these may be the same registry so the source_repo_path and dest_repo_path are included

The steps are:

• After Fastfeedback on the main branch p2p-promote-to-extended-test is executed where the default implementation for all applications is:

.PHONY: p2p-promote-to-extended-test
p2p-promote-to-extended-test: source_repo_path=$(FAST_FEEDBACK_PATH)
p2p-promote-to-extended-test: dest_repo_path=$(EXTENDED_TEST_PATH)
p2p-promote-to-extended-test:  p2p-promote-generic

• After extended test the p2p-promote-to-prod is executed where the default implementation for all applications is:

.PHONY: p2p-promote-to-prod
p2p-promote-to-prod:  source_repo_path=$(EXTENDED_TEST_PATH)
p2p-promote-to-prod:  dest_repo_path=$(PROD_PATH)
p2p-promote-to-prod:  p2p-promote-generic

Deployment Frequency

The Core Platform P2P can be configured to deploy your application in one of three modes:

• Right away: Fast feedback runs right away, followed by extended test, followed by production deployment.
• On a schedule: Fast feedback runs right away, extended test and production run on a schedule.
• Manually: Fast feedback runs right away, extended test and production are manually triggered.

Or a hybrid of one of the above e.g. Fast feedback and extended test run right away, production is manually triggered.

Right away

Create a single CD.yaml workflow file in your repo in ./github/workflows with the following content:

name: Continuous Deployment

on:
  workflow_dispatch:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

permissions:
  contents: write
  id-token: write

jobs:
  version:
    uses: coreeng/p2p/.github/workflows/p2p-version.yaml@v1
    secrets:
      git-token: ${{ secrets.GITHUB_TOKEN }} 

  fastfeedback:
    needs: [version]
    uses: coreeng/p2p/.github/workflows/p2p-workflow-fastfeedback.yaml@v1
    with:
      version: ${{ needs.version.outputs.version }}

  extended-test:
    needs: [version, fastfeedback]
    uses: coreeng/p2p/.github/workflows/p2p-workflow-extended-test.yaml@v1
    with:
      version: ${{ needs.version.outputs.version }}

  prod:
    needs: [version, extended-test]
    uses: coreeng/p2p/.github/workflows/p2p-workflow-prod.yaml@v1
    with:
      version: ${{ needs.version.outputs.version }}

Schedule

This is the configuration all templates come with where:

• Fast feedback runs for every commit to main
• Extended test runs once a day, on the latest version that passed fastfeedback
• Production runs once a day, on the latest version that passed extended test

Create three workflows in your repo in ./github/workflows with the following content:

Fast Feedback

name: Fast Feedback

on:
  workflow_dispatch:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

permissions:
  contents: write
  id-token: write

jobs:
  version:
    uses: coreeng/p2p/.github/workflows/p2p-version.yaml@v1
    secrets:
      git-token: ${{ secrets.GITHUB_TOKEN }}
    
  fastfeedback:
    needs: [version]
    uses: coreeng/p2p/.github/workflows/p2p-workflow-fastfeedback.yaml@v1
    with:
     version: ${{ needs.version.outputs.version }}

Extended Test

name: Extended Test

on:
  workflow_dispatch:
  schedule:
    - cron: '0 22 * * *'

permissions:
  contents: read
  id-token: write

jobs:
  get-latest-version:
    uses: coreeng/p2p/.github/workflows/p2p-get-latest-image-extended-test.yaml@v1
    with:
      image-name: go-reference-app

  extendedtests:
    needs: [get-latest-version]
    uses: coreeng/p2p/.github/workflows/p2p-workflow-extended-test.yaml@v1
    with:
      version: ${{ needs.get-latest-version.outputs.version }}

Prod

name: Prod

on:
  workflow_dispatch:
  schedule:
    - cron: '30 5 * * 1,5'

permissions:
  contents: read
  id-token: write

jobs:
  get-latest-version:
    uses: coreeng/p2p/.github/workflows/p2p-get-latest-image-prod.yaml@v1
    with:
      image-name: go-reference-app

  prod:
    needs: [get-latest-version]
    uses: coreeng/p2p/.github/workflows/p2p-workflow-prod.yaml@v1
    with:
      version: ${{ needs.get-latest-version.outputs.version }}

Manually

For manually, create the same workflows as scheduled but remove the schedule. The manual trigger is available in the Actions tab in your repo.

Local development

Some Makefile targets within your app require specific variables to successfully execute. Those include registry details, tenancy or versioning. Corectl allows you to export those based on your existing local tenant and repository configuration. This corectl feature is designed to ease local development and enable seamless p2p targets execution.

To construct and print out a list of variables run:

corectl p2p export --tenant <your_tenant_name> --environment <your_target_env> --repoPath <filepath_to_app_repository>

The command will print out all variables as export statements, ex.:

export BASE_DOMAIN="gcp-dev.cecg.platform.cecg.io" # domain pulled in from environments project                                 
export REPO_PATH="/path/to/my/app/repo" # path to your git local app repository                                                 
export REGION="europe-west2" # region pulled in from environments project                                                         
export REGISTRY="europe-west2-docker.pkg.dev/myproject/tenant/mytenant" # registry url constructed based on environments project
export VERSION="53b9c85" # short git hash commit pointing to current HEAD (latest commit in currently checked out branch in that repo)                                                 
export TENANT_NAME="mytenant" 

You can copy all the output and paste again in currently running shell to export all envs. Once done you can run most of the p2p Makefile targets without additional configuration.

To export those automatically in the current shell, run:

eval $(corectl p2p export --tenant <your_tenant_name> --environment <your_target_env> --repoPath <filepath_to_app_repository>)

If you omit -repoPath flag, the tool will default to current directory.

For further details run:

corectl p2p export -h