Onboarding

Creating a tenancy is how you onboard onto Core Platform. If you already have a tenancy, you can jump to new-app to deploy a new application.

Adding a tenancy

corectl tenants create

You’ll be prompted for the following information about your new tenant:

• name - Name of your tenancy. Must be the same as your filename.
• parent - Name of the parent tenant or root. Note: root tenant is created implicitly.
• description - Description for your tenancy.
• contactEmail - Metadata: Who is the contact for this tenancy?
• environments - which of the environments in Environments Repo you want to deploy to
• repositories - if you have no existing applications then leave this empty. If migrating then add the repos with your applications (comma separated GitHub links)
• adminGroup - will get permission to do all actions in the created namespaces - optional
• readonlyGroup - will get read-only access to the created namespaces - optional

Once you fill the form, corectl will create a PR in the Environments Repo with a new file for the tenancy. Once the PR is merged, a configuration for the new tenant will be provisioned automatically.

Accessing your namespaces

Once the above PR that corectl creates is merged everyone in the groups will have access to the namespaces created for that tenancy.

If you access the cluster from the local machine, you need to connect to the cluster. The easiest way to do this is using corectl:

corectl env connect <env-name>

For example, to check a namespace for a tenancy named myfirsttenancy:

kubectl get namespace myfirsttenancy
NAME             STATUS   AGE
myfirsttenancy   Active   30s

Create application

Create a new application

The Core Platform with a single command will configure:

• A new repository
• A template application demonstrating full continuous delivery:
  • Versioning
  • Functional testing
  • Non-functional testing
  • Promotion between a set of standard environments
  • Deployment to production

After running this command, and triggering the resulting GitHub workflow you’ll have an application ready to deploy to production on every commit!

corectl apps create <new-app-name> [<new-app-dir>]

It will:

• prompt you a few additional questions, including template for the application.
• create a new repository using the selected template
• configure the created GitHub repository, so P2P workflows will run without an issue

Developer workflow

Now you have an application deployed to production. You can make changes by raising PRs.

Raise a PR

Raising a PR will automatically build the app, push the docker image, and deploy it to environments for functional, non-functional, and integration testing.

Merge the PR

Merges to main by default do the same as a PR, and additionally deploy to a stable dev namespace that can be used for showcasing or integration testing.

Configuration

Configuration should be checked into your repository and injected to your application via environment variables. Configuration can be broken into two categories:

1. Common Configuration
2. Environment Specific Configuration

See sensitive configuration for secrets.

Common Configuration

• Add values to a file called /config/common.yaml to your repository.

In the format:

service:
    environmentVariables: 
        MODE: default
        DATABASE_HOSTNAME: common-database-url

Environment Specific Configuration

• Add an environment specific file called /config/env.yaml e.g. /config/integration.yaml to your repository.

service:
    environmentVariables: 
        MODE: default
        DATABASE_HOSTNAME: integration-database-url

Injecting the configuration

Inject .Values.service.environmentVariables as environment variables to the application via a block in the containers: definition, in the Deployment manifest, like:

{{- if .Values.service.environmentVariables }}
          env:
            {{- range $key, $value := .Values.service.environmentVariables }}
            - name: {{ $key }}
              value: {{ $value }}
          {{- end }}
{{- end }}

Pass the files to the helm install on deploy:

deploy-integration:  
    helm upgrade --install $(app_name) helm-charts/app -n $(tenant_name)-integration \
        -f /config/common.yaml \
        -f /config/integration.yaml \
      ...

Sensitive Configuration (secrets)

Sensitive Configuration

Sensitive configuration, i.e. secrets, can be configured in GitHub, and passed to the applications as Environment Variables via the P2P pipeline.

Secrets can be managed in the Settings tab on a repository, by navigating to: Settings > Secrets and variables > Actions.

Common Sensitive Configuration

On the repo for the application, navigate to: Settings > Secrets and variables > Actions and then the Secrets tab. Under Repository secrets click New repository secret. N.B. Unlike variables, secret values cannot viewed in the GitHub UI or logs.

In this example below we’ll make a secret called: GLOBAL_SECRET with a value of GLOBAL_PASSWORD-123

Environment Specific Sensitive Configuration

Environment specific secrets can be configured in GitHub via Settings > Environments, then choosing the correct platform environment.

If you need further isolation (e.g. different setup for Functional vs NFT test environments) that’s possible by configuring the secrets in dev and then only passing in specific variables to each Makefile target (covered below).

Environment specific secrets can be configured via Settings > Environments, then choosing an environment, and clicking Add environment secrets.

In this example below let’s add a secret called ENV_SPECIFIC_SECRET with a value of GCP-DEV-PASSWORD-123.

Full example

As per the above we’ll add both a common and environment specific secret in this example.

The above steps the secrets in to GitHub actions, but because we use re-usable workflows in GitHub actions additional steps are necessary to explicitly pass them through to the execution context that invokes the Makefile, and then changes are necessary in the Makefile to pass environment variables through to the application itself.

Updating the P2P workflow

Let’s say we want to pass our above example variables and secrets into the application running in the integration environment. As the integration test step is past of the fast feedback pipeline we’ll need to update the fast-feedback.yaml workflow (this lives in .github/workflows in the root of the repository)

To simplify the passing of variables and secrets to an application we can use a special variable called ENV_VARS. The platform P2P will automatically take everything that is configured in ENV_VARS and pass that in as environment variables, ENV_VARS expects to be a multi-line variable of the form KEY=VALUE on each line.

In GitHub workflows, secrets can be referred to via ${{ secrets.NAME }} and in yaml a multi-line variable is denoted via a | character.

So to pass in all of the above example variables into the P2P we’d modify the application’s fast-feedback yaml to add a secrets: block that defines all the variables we want in a multi-line env_vars: variable.

fastfeedback:
    needs: [version]
    uses: coreeng/p2p/.github/workflows/p2p-workflow-fastfeedback.yaml@v1
    secrets:
      env_vars: | 
        GLOBAL_SECRET=${{secrets.GLOBAL_SECRET}}
        ENV_SPECIFIC_SECRET=${{secrets.ENV_SPECIFIC_SECRET}}
    with:
      version: ${{ needs.version.outputs.version }}
      working-directory: ./`

The entirety of the env_vars value is treated as a secret, so it will be hidden from any GitHub log output.

Every variable we add like the above will be passed into the execution context of the Makefile.

Updating the Makefile

The above steps have configured the variables and provided their values to the execution of the Makefile. The final step will be using them in our application via configuring them in the Makefile. The Makefile is in the root of the application repo and is the main interface between the P2P pipeline steps and the application code.

The Makefile has separate targets for each stage of the P2P pipeline. In this example we’re going to add variables to the application when running in just the integration environment - but exactly the same steps can be added for other environments (though extended test and prod will require their respective GitHub worfkflows to be amended in the previous step).

Variables can be passed into the application via additions like:

--set-literal service.environmentVariables.GLOBAL_VARIABLE="${GLOBAL_VARIABLE}"

to the relevant Makefile target. So for a full example that targets the integration environment:

.PHONY: deploy-integration
deploy-integration:  ## Deploy helm chart of the app to integration namespace
    helm upgrade --install $(app_name) helm-charts/app -n $(tenant_name)-integration \
        --set registry=$(REGISTRY)/$(FAST_FEEDBACK_PATH) \
        --set domain=$(BASE_DOMAIN) \
        --set appUrlSuffix="-$(tenant_name)-integration" \
        --set tag=$(image_tag) \
        --set service.image=$(image_name) \
        --set integration.image=$(image_name)-integration \
        --set monitoring=$(MONITORING) \
        --set dashboarding=$(DASHBOARDING) \
        --set tenantName=$(tenant_name) \
        --set appName=$(app_name) \
        --set service.resources.requests.cpu=0m \
        --set-literal service.environmentVariables.GLOBAL_SECRET="${GLOBAL_SECRET}" \
        --set-literal service.environmentVariables.ENV_SPECIFIC_SECRET="${ENV_SPECIFIC_SECRET}" \
        --atomic
    helm list -n $(tenant_name)-integration ## list installed charts in the given tenant namespace

The application’s Kubernetes deployment manifest then needs to take care of supplying everything that is in .Values.service.environmentVariables as an actual environment variable - this is already done in our reference applications - and is covered in more detail above.

Validating the application has the expected variables

To validate all of the above has worked correctly, after the application is deployed to the relevant environment you can connect with corectl env connect <environment> and then run something like:

kubectl get pods -n <namespace>

To get a running pod, and then:

kubectl exec -it <pod-name> -n <namespace> -- env

To print out the environment.

Application Logs

Open Grafana for your environment:

corectl env open <env> grafana

Go to Explore

Set datasource to Cloud Logging

Select:

• Log Bucket: _Default
• View: _AllLogs

Queries

Get all logs for a given namespace

resource.type="k8s_container"
resource.labels.namespace_name="platform-ingress"

Get all logs for a given container

resource.type="k8s_container"
resource.labels.namespace_name="platform-ingress"
resource.labels.container_name="traefik"

Application Monitoring

Accessing Grafana

Via internal services

Grafana is available at grafana.internal_services.domain

The internal_services.domain is in the config.yaml for the environment you want to access grafana for in your Environments Repo.

You can quickly access it by running:

corectl env open <env> grafana

How can I monitor application resources?

This dashboard will allow a team to monitor their application namespaces and check their status. It will show data like:

• CPU Usage
• Memory usage
• Pod status
• Pod restart count

How can I monitor the whole cluster?

The global view dashboard will give you visibility over the cluster as a whole. This will show you data like:

• Nodes status
• CPU and Memory usage
• CPU and Memory Limits
• CPU and Memory Reserved
• Pod and namespace count
• Pods status

How can do I know if the environment is stable?

The platform-monitoring module also deploys a continuous load. This will create k6 injectors and pods with podinfo, always with a stable throughput allowing us to monitor with enough data the different percentils and any errors that occur to ensure that we can be proactive in investigating and fixing any issues.

Application-Specific Metrics

Examples of Application-Specific Metrics:

• Go GC statistics
• Request processing latency
• Number of processed jobs

Prerequisite

You should have deployed monitoring stack for your tenant. This monitoring stack consists of preconfigured Grafana and Prometheus instances.

To install monitoring stack, run the following command:

# Add helm repository
helm repo add coreeng https://coreeng.github.io/core-platform-assets
helm repo update

# Install the chart
helm -n {{ target-ns }} install monitoring-stack coreeng/monitoring-stack --set tenantName={{ your-tenant-name }}

To upgrade installed monitoring stack, run the following command:

#Upgrade installed chart
helm -n {{ target-ns }} upgrade monitoring-stack coreeng/monitoring-stack --set tenantName={{ your-tenant-name }}

Monitoring Application

To monitor start monitoring your application, you have to create ServiceMonitor or PodMonitor resource for your application and put monitoring/instance:: {{ your-tenant-name }} label on it. By using tenant label, you define the target prometheus instance. Here is the example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: some-service
  labels:
    app.kubernetes.io/name: some-service
    monitoring/instance: {{ your-tenant-name }}
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: some-service
  endpoints:
    - interval: 30s
      port: metrics
      path: /metrics

This configuration assumes that your application exposes metrics on metrics endpoint in Prometheus format.

Accessing Grafana and Prometheus

Grafana and Prometheus are not exposed by default. You have two options:

• Create Ingress resources for internal domain
• Port forward Grafana and Prometheus

Through Ingress

To enable ingress for monitoring services, you have to set the respective values during the installation of the monitoring-stack chart. It should look like this:

#Install helm chart
helm -n {{ target-ns }} install monitoring-stack coreeng/monitoring-stack \
    --set tenantName={{ your-tenant-name }} \
    --set internalServicesDomain={{ internal-services-domain }} \
    --set prometheus.ingress.enabled=true \
    --set grafana.ingress.enabled=true

To upgrade installed monitoring stack, run the following command:

#Upgrade installed chart
helm -n {{ target-ns }} upgrade monitoring-stack coreeng/monitoring-stack \
    --set tenantName={{ your-tenant-name }} \
    --set internalServicesDomain={{ internal-services-domain }} \
    --set prometheus.ingress.enabled=true \
    --set grafana.ingress.enabled=true

• Grafana should be accessible by the URL: {{ your-tenant-name }}-grafana.{{ internal-services-domain }}
• Prometheus should be accessible by the URL: {{ your-tenant-name }}-prometheus.{{ internal-services-domain }}

Through port forward

To port forward Grafana, run the command below and access it on http://localhost:3000

kubectl -n {{ target-ns }} port-forward service/grafana-{{ your-tenant-name }}-service 3000

To port forward Prometheus, run the command below and access it on http://localhost:9090:

kubectl -n {{ target-ns }} port-forward service/prometheus-{{ your-tenant-name }} 9090

Creating Grafana Dashboards

To create Grafana Dashboard you have to create GrafanaDashboard CR. If you create it directly with UI, the changes will not be persisted for long, so it’s advised to use Grafana UI for designing your Dashboard and then export it to CR.

When creating GrafanaDashboard, you have to specify Grafana instance selector, so Grafana Operator can inject your dashboard in Grafana instance:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: my-grafana-dashboard
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
...

In addition, in case your GrafanaDashboard is not in the same namespace as your monitoring stack, you have to specify an additional field:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: my-grafana-dashboard
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
  allowCrossNamespaceImport: "true"
...

Creating Grafana Datasources

To create Grafana Datasources you have to create GrafanaDatasource CR. If you create it directly with UI, the changes will not be persisted for long, so it’s advised to use Grafana UI for designing your Datasource and then export it to CR.

When creating GrafanaDatasource, you have to specify Grafana instance selector, so Grafana Operator can inject your datasource in Grafana instance:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: my-datasource
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
...

In addition, in case your GrafanaDatasource is not in the same namespace as your monitoring stack, you have to specify an additional field:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: my-datasource
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
  allowCrossNamespaceImport: "true"
...

Using Grafana Plugins for Dashboards/Datasources

To create Grafana plugins for dashboards or datasources, you need to specify an additional field, plugins.

Here is an example of a plugin for a datasource:

kind: GrafanaDatasource
metadata:
  name: my-questdb
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
  plugins:
    - name: questdb-questdb-datasource
      version: 0.1.4
  datasource:
    name: questdb
    access: proxy
    type: questdb-questdb-datasource
    database: qdb
    url: my-questdb-headless.questdb.svc:8812
    user: admin
    jsonData:
      sslmode: disable
      maxOpenConns: 100
      maxIdleConns: 100
      connMaxLifetime: 14400
    secureJsonData:
      password: quest

Here is an example of a plugin for dashboards:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: keycloak-dashboard
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
  plugins:
    - name: grafana-piechart-panel
      version: 1.3.9
  json: >
  {
    ...
  }  

Expose your service via Ingress

To expose an HTTP service over TLS, you add a Kubernetes Ingress resource, e.g.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: {{ service-host }}
    external-dns.alpha.kubernetes.io/target: {{ ingress-host }}
  name: reference-service
  namespace: golang-dev
spec:
  ingressClassName: platform-ingress
  rules:
  - host: {{ service-host }}
    http:
      paths:
      - backend:
          service:
            name: reference-service
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

The important parts are:

• external-dns.alpha.kubernetes.io/target must match one of the ingress domains defined in environment config yaml file in Environments Repo
• external-dns.alpha.kubernetes.io/hostname has to be a subdomain of the target
• ingressClassName must be platform-ingress

Your service will now be available over the Internet over TLS.

To have a deeper understanding of how the Platform Ingress feature is designed, check the Platform Ingress Overview.

Troubleshooting

CreateContainerError: failed to reserve container name

The container runtime is unable to get the container into a running state so it’s stuck in pending.

A potential cause could be resource starvation on nodes. Check the cpu/memory usage of the node the pod is running on:

kubectl top node $(kubectl get pod <pod-name> -o jsonpath='{.spec.nodeName}')

NAME                                             CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
gke-sandbox-gcp-sandbox-gcp-pool-18d01f3c-xhd8   174m         9%     2047Mi          33%

If the node cpu/memory usage is high, it’s possible the container runtime is struggling to find enough resources to be able to run the container.

Resolution

Ensure that your pod has cpu/memory requests set. This will allow the kube scheduler to to place your pods on better balanced nodes.

P2P GCP Auth Fail

I’ve got a new app, and when the P2P tries to authenticate with Google Cloud, it fails with an error like this:

Error: google-github-actions/auth failed with: retry function failed after 4 attempts: failed to generate Google Cloud federated token for projects/{{ project-number }}/locations/global/workloadIdentityPools/p2p-knowledge-platform/providers/p2p-knowledge-platform: (400) {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}

This is likely as the tenant definition has either not been deployed or doesn’t contain the GitHub repo. Without this the GitGub Action does not have permission to authenticate with GCP.

To find the problem with your tenant deployment, you can either:

• use ArgoCD UI by accessing https://argocd.{{ internalServices.domain }}
• kubectl CLI (may need a platform admin to run):

Note: all of this is internal implementation details of the platform and may change in the future.

Check tenant deployment status

Does the tenant exist? Is it healthy? Is it synchedd?

With kubectl:

kubectl get -n argocd apps <tenant-name>
NAME                 SYNC STATUS   HEALTH STATUS
<tenant-name>        Synced        Healthy

With the ArgoCD UI:

If tenant is not synched/healthy, what is the reason?

With kubectl:

kubectl -n argocd describe app <tenant-name>

Why is it not synched/healthy will be in the output.

With the ArgoCD UI:

Does the tenant have the right repo?

With kubectl:

kubectl get -n argocd apps <tenant-name> -o yaml

It should contain output which includes:

 - name: tenant.repos
   value: '["https://github.com/{{ github-org }}/{{ github-repo }}"]'

With the ArgoCD UI (App Details > Parameters):

If you’ve added the repo to the tenant, but it isn’t shown here, then likely the new tenant definition hasn’t been deployed to the environment.

P2P Kubeconfig can not setup tunnel

ERROR: (gcloud.compute.start-iap-tunnel) Could not fetch resource:
 - Required 'compute.instances.get' permission for 'projects/{{ project-id }}/zones/europe-west2-a/instances/sandbox-gcp-bastion'

Follow the same steps as defined in P2P GCP Auth Fail to ensure the tenancy is setup correctly.

Then check that the IAMPolicyMember resources are properly syncing with GCP e.g. for a tenant called knowledge-platform

kubectl -n knowledge-platform get iampolicymembers.iam.cnrm.cloud.google.com                                                                                                                                                      │
NAME                                              AGE   READY   STATUS     STATUS AGE                                                                                                                                                                                            │
artifact-registry-writer-p2p-knowledge-platform   68m   True    UpToDate   68m                                                                                                                                                                                                   │
bastion-account-user-p2p-knowledge-platform       68m   True    UpToDate   68m                                                                                                                                                                                                   │
cluster-access-p2p-knowledge-platform             89m   True    UpToDate   89m                                                                                                                                                                                                   │
compute-os-login-p2p-knowledge-platform           68m   True    UpToDate   68m                                                                                                                                                                                                   │
compute-project-get-p2p-knowledge-platform        68m   True    UpToDate   68m                                                                                                                                                                                                   │
compute-viewer-p2p-knowledge-platform             68m   True    UpToDate   68m                                                                                                                                                                                                   │
iap-user-p2p-knowledge-platform                   68m   True    UpToDate   68m                                                                                                                                                                                                   │
workflow-identity-p2p-knowledge-platform          89m   True    UpToDate   89m

For the above error the compute-os-login-p2p-knowledge-platform one is responsible for giving access.

P2P Helm Numeric Value Error

Error

P2P fails on the helm upgrade step, while trying to apply the helm charts

• ie running. helm upgrade --host=localhost --set port=123), results in error:

ERROR: json: cannot unmarshal number into Go struct field EnvVar.spec.template.spec.containers.env.value of type string

This is caused by your helm chart which contains values that are to be overridden, as for example if in our helm chart we had the below with:

  host: {{ .Values.Host }}
  port: {{ .Values.Port }}

where these are meant to be passed over to kubernetes manifests, we need to guarantee that both of them are treated as string and not as numeric.

So if in the example above where, you where trying to override with --set port=123, this causes parsing issues on the helm and kubernetes side.

Solution

To fix this, we can safely “quote” the variables in question as per below:

  host: {{ .Values.Host }}
  port: {{ .Values.Port | quote }}

How-Tos

• Setting resource requests
• Autoscale application
• Deploying Cloud Infra from the P2P
• Connect to CloudSQL
• Connect to MemoryStore
• Debug Cloud Access issues