Enterprise Edition: Google Cloud Platform GKE

Modified on Tue, 30 May, 2023 at 6:53 PM

TABLE OF CONTENTS

Overview
- Component Overview
- Connectivity / Firewall Requirements
  - Initial Installation Requirements
  - Running Requirements
Installation Instructions
Next steps

Overview

Component Overview

Direktiv can be deployed in the following 3 configurations.

Open source: A single virtual machine running a k8s environment or a public cloud provider Kubernetes service, hosting the Direktiv pods and the containers used for integration.
Enterprise Edition Production VM(s) Only: a single virtual machine (or 3 virtual machines for HA purposes), running a k8s environment (or clustered across multiple), hosting the Direktiv pods and the containers used for integration
Enterprise Edition Production Kubernetes: a cloud provider Kubernetes service, hosting the Direktiv pods and the containers used for integration

The diagram below illustrates the proposed option 3 for Production deployment:

Proposed EE deployment diagram (GCP EKS)

The Kubernetes deployment runs on the Google Cloud Platform Kubernetes Engine. The installation will contain the following components:

A Kubernetes environment running in EKS.
A single instance of the Direktiv Enterprise Edition
A single instance PostgreSQL database

The following components are assumed to be available:

Optional: a single GitLab instances (if not available in customer environment, or access to a GitHub instance)
Optional: a single Docker repository for storing local images (if not available in customer environment). Alternatively, Docker Hub can also be used.

Connectivity / Firewall Requirements

The following requirements exist for the successful deployment of Direktiv:

Initial Installation Requirements

The following network configurations are critical, all components need to have the following services configured and working correctly:

Firewall: firewalls on the VMs need to be disabled or the ports described in the table below need to be opened.

The tables below capture all of the external connectivity requirements for initial deployment.

Source	Destination	Protocol / Port	Notes
JumpHost	download.docker.com, registry-1.docker.io, cloudfront.net, production.cloudflare.docker.com	HTTPS	Initial Docker install on the VM. If no access is allowed, download the Docker package from https://download.docker.com/linux/ubuntu/dists/ and install using https://docs.docker.com/engine/install/ubuntu/#install-from-a-package
JumpHost	raw.githubusercontent.com/helm	HTTPS	Initial helm installer
JumpHost	get.helm.sh/	HTTPS	Helm package download. If no access is allowed, download the helm package from https://github.com/helm/helm/releases and install it manually using https://helm.sh/docs/intro/install/
JumpHost	docker.io/smallstep/step-cli	HTTPS	Used to create the certificates used by Linkerd
JumpHost	chart.direktiv.io	HTTPS	Helm chart used for database install and Direktiv application install (OPTIONAL)
JumpHost	registry.developers.crunchydata.com, prod.developers.crunchydata.com	HTTPS	Database install for Postgresql (OPTIONAL)
JumpHost	githubusercontent.com/direktiv/direktiv	HTTPS	Configure Direktiv database from pre-configured settings (OPTIONAL)
JumpHost / K8S Cluster IP	ghcr.io/projectcontour	HTTPS	Knative
JumpHost / K8S Cluster IP	docker.io/envoyproxy	HTTPS	Knative
JumpHost / K8S Cluster IP	gcr.io/knative-releases/	HTTPS	Knative
JumpHost / K8S Cluster IP	k8s.gcr.io/ingress-nginx/	HTTPS	NGINX install (ingress load balancer, part of Direktiv helm chart)
JumpHost / K8S Cluster IP	docker.io/jimmidyson/configmap-reload	HTTPS	Configmap reload for Kubernetes install (part of Direktiv helm chart)
JumpHos/ K8S Cluster IP	quay.io/prometheus	HTTPS	Prometheus install (part of Direktiv helm chart)
JumpHos/ K8S Cluster IP	docker.io/direktiv	HTTPS	Direktiv installation
JumpHost	download.docker.com, registry-1.docker.io, cloudfront.net, production.cloudflare.docker.com	HTTPS	Initial Docker install on the VM. If no access is allowed, download the Docker package from https://download.docker.com/linux/ubuntu/dists/ and install using https://docs.docker.com/engine/install/ubuntu/#install-from-a-package
JumpHost	raw.githubusercontent.com/helm	HTTPS	Initial helm installer

The Docker registries whitelist includes:

Running Requirements

There are several running firewall/connectivity requirements to ensure the ongoing operation of the environment. The tables below capture all of the external connectivity requirements for an operational state:

Source	Destination	Protocol / Port	Notes
Direktiv IP	direktiv.azurecr.io	HTTPS	Pulling Direktiv pre-built containers
Direktiv IP	hub.docker.com/direktiv	HTTPS	Pulling Direktiv pre-built containers
JumpHost	Direktiv VM	TCP: 443 / 6443 / 10250 / 2379-2380	Kubernetes API Server, Kubelet metrics, HA
Client	Direktiv IP	TCP: 443	Direktiv UI & API access
Client	Direktiv IP	TCP: 443	Direktiv UI & API access

Installation Instructions

JumpHost Proxy Settings

If the environment requires a proxy to be configured, the following settings can be applied on the JumpHost. You can set up a permanent proxy for a single user by editing the ~/.bashrc file:

First, log in to your Ubuntu system with a user whom you want to set a proxy for.
Next, open the terminal interface and edit the ~/.bashrcfile as shown below:

vi ~/.bashrc

Add the following lines at the end of the file that matches your proxy server:

export http_proxy=username:password@proxy-server-ip:8080
export https_proxy=username:password@proxy-server-ip:8082
export ftp_proxy=username:password@proxy-server-ip:8080
export no_proxy=localhost, 127.0.0.1

Save and close the file when you are finished.
Then to activate your new proxy settings for the current session, use the following command:

source ~/.bashrc

Install Docker (Jumphost)

Install Docker on the single VM deployment using the following steps.

For the latest installation instructions per distro please review: https://docs.docker.com/engine/install/

Uninstall old versions

Older versions of Docker were called docker, docker.io, or docker-engine. If these are installed, uninstall them:

sudo apt-get remove docker docker-engine docker.io containerd runc

It’s OK if apt-get reports that none of these packages are installed.

Set up the repository

Update the apt package index and install packages to allow apt to use a repository over HTTPS:

sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

Add Docker’s official GPG key:

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Use the following command to set up the repository:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine

Update the apt package index, and install the latest versionof Docker Engine, containers, and Docker Compose, or go to the next step to install a specific version:

sudo apt-get update

Next command:

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Verify that Docker Engine is installed correctly by running the hello-world image. This might not work, depending on whether connectivity can be established to https://hub.docker.com

sudo docker run hello-world

Install Google Cloud CLI (Jumphost)

Google Cloud command-line interface is NOT installed by default. To access the Kubernetes cluster Google CloudCLI is used to collect the Kubernetes cluster credentials.

NOTE: Install the Google CloudCLI using the link below or follow the steps below:

https://cloud.google.com/sdk/docs/install#deb

Install the required packages needed for the Google Cloud CLI:

sudo apt-get install apt-transport-https ca-certificates gnup

Add the Google Cloud CLI distribution URI as a package source. If your distribution supports the signed-by option, run the following command:

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

Import the Google Cloud public key. If your distribution's apt-key command supports the --keyring argument, run the following command:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -

Update and install the Google Cloud CLI:

sudo apt-get update && sudo apt-get install google-cloud-cli && sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin

Run Google Cloud-init to get started:

gcloud init

**Install helm (Jumphost)**

helm is used to install Direktiv and Knative components.

NOTE: For the latest installation instructions per distro, please refer to: https://helm.sh/docs/intro/install/

To install helm, the following steps need to be followed:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

**Install kubectl (Jumphost)**

kubectl is used to manage the Kubernetes cluster.

NOTE: For the latest installation instructions per distro, please refer to: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

To install kubectl, the following steps need to be followed:

Update the apt package index and install packages needed to use the Kubernetes apt repository:

sudo apt-get update
sudo apt-get install -y ca-certificates curl

Download the Google Cloud public signing key:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Add the Kubernetes apt repository:

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Update apt package index with the new repository and install kubectl:

sudo apt-get update
sudo apt-get install -y kubectl

Verify the installation:

# kubectl version --client --output=yaml
clientVersion:
  buildDate: "2022-08-23T17:44:59Z"
  compiler: gc
  gitCommit: a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2
  gitTreeState: clean
  gitVersion: v1.25.0
  goVersion: go1.19
  major: "1"
  minor: "25"
  platform: linux/amd64
kustomizeVersion: v4.5.7

Deploy GCP GKE (Jumphost)

CAUTION: GKE Autopilot clusters are NOT supported

The following steps show how to deploy a default Google Cloud GKE service.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.
Go to project selector
Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
Enable the Artifact Registry and Google Kubernetes Engine APIs: Enable the APIs

GKE Standard Cluster

Create a Standard cluster named direktiv-cluster:

cloud container clusters create direktiv-cluster \
    --num-nodes=1 \
    --zone=COMPUTE_ZONE

Replace COMPUTE_REGION with the Compute Engine regionfor the cluster. An example is shown below:

# gcloud container clusters create direktiv-cluster --num-nodes=1 --zone=australia-southeast1

Default change: VPC-native is the default mode
during cluster creation for versions greater than
1.21.0-gke.1500. To create advanced routes based
clusters, please pass the `--no-enable-ip-alias` flag

Default change: During creation of nodepools or
autoscaling configuration changes for cluster
versions greater than 1.24.1-gke.800 a default
location policy is applied. For Spot and PVM it
defaults to ANY, and for all other VM kinds a BALANCED
policy is used. To change the default values use the
`--location-policy` flag.

Note: Your Pod address range (`--cluster-ipv4-cidr`)
can accommodate at most 1008 node(s).

Creating cluster direktiv-cluster in australia-southeast1...
Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1/projects/direktiv/zones/australia-southeast1/clusters/direktiv-cluster].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/australia-southeast1/direktiv-cluster?project=direktiv
kubeconfig entry generated for direktiv-cluster.

NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
direktiv-cluster australia-southeast1 1.22.11-gke.400 35.189.29.225 e2-medium 1.22.11-gke.400 3 RUNNING

After creating your cluster, you need to get authentication credentials to interact with the cluster. This command configures kubectl to use the cluster you created:

gcloud container clusters get-credentials direktiv-cluster --region=COMPUTE_REGION

Replace COMPUTE_REGION with the compute region previously configured for the cluster. An example is shown below:

# gcloud container clusters get-credentials direktiv-cluster --region=australia-southeast1
Fetching cluster endpoint and auth data.
kubeconfig entry generated for direktiv-cluster.

Verify the installation:

# kubectl get nodes
W0913 02:30:16.023890  114974 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
NAME                                              STATUS   ROLES    AGE   VERSION
gke-direktiv-cluster-default-pool-3cb4331c-v4xt   Ready    <none>   44s   v1.22.11-gke.400
gke-direktiv-cluster-default-pool-7ad0efef-b25d   Ready    <none>   46s   v1.22.11-gke.400
gke-direktiv-cluster-default-pool-eee39bed-nfpj   Ready    <none>   45s   v1.22.11-gke.400

Install Direktiv Enterprise Edition

The Direktiv Enterprise Edition is a commercial platform for running Direktiv. Access to the installation files are restricted, and requires a license and access to the repository download section (via GitHub or the Direktiv Service Portal)

The process below installs all of the Enterprise Edition components on the single node cluster.

Download the Direktiv installer. It provides a basic configuration for all components needed for Direktiv. It assumed that the person installing this has access to the direktiv-ee GitHub repository:

tar -xvf direktiv-ee.tar.gz
cd direktiv-ee/install

Set the following environmental variables:
- DIREKTIV_HOST: sets the FQDN for the Direktiv instance.
- DIREKTIV_DEV: if this environment variable has a value the installation uses the localhost:5000 variant of Direktiv images.
- DIREKTIV_TOKEN: if this environment variable has a value the installation uses as the admin API key (this has allows for elevated API admin access)

export DIREKTIV_HOST="dev.direktiv.io"
export DIREKTIV_DEV="localhost:5000"
export DIREKTIV_TOKEN="KxTp8d_gj5yj$%mPkvFY=f2rPN2K6N?F"

Run the script to prepare the installation:

./prepare.sh

The above command generates an install YAML files. If a proxy is used, configure the following files with the appropriate settings:

~/direktiv-ee/install/05_knative# cat knative.yaml
# -- Knative version
version: 1.8.0
# -- Knative replicas
replicas: 1
# -- Proxy settings for controller
http_proxy: ""
https_proxy: ""
no_proxy: ""
# -- Custom certificate for controller. This needs to be a secret created before installation in the knative-serving namespace
certificate: ""
# -- Repositories which skip digest resolution
skip-digest: kind.local,ko.local,dev.local,localhost:5000,localhost:31212

Direktiv also needs to have the proxy settings configured:

:~/direktiv-ee/install/06_direktiv# cat direktiv.yaml
pullPolicy: Always
debug: "false"

proxy:
  no-proxy: ""
  http-proxy: ""
  https-proxy: ""

database:
  replicas: 1
  image: "direktiv/ui-ee"

ui:
  replicas: 1
  image: "direktiv/ui-ee"

api:
  replicas: 1
  image: "direktiv/api-ee"
  additionalEnvs:
  - name: DIREKTIV_ROLE_ADMIN
    value: admin
  - name: DIREKTIV_TOKEN_SECRET
    valueFrom:
      secretKeyRef:
        name: tokensecret
        key: tokensecret
  - name: DIREKTIV_ADMIN_SECRET
    valueFrom:
      secretKeyRef:
        name: adminsecret
        key: adminsecret

Certificates are required for the DIREKTIV_HOST URL selected used in the previous step. Direktiv will use self-signed certificates if not provided, but this is not recommended. To run the installation with certificates for (as an example *.direktiv.io), replace the following 2 files with the server.key and server.crtfiles for the respective domain:

~/direktiv-ee/install/04_keycloak/certs# ls -l
total 28
-rw-r--r-- 1 root root 7124 Jan 30 04:44 direktiv.io.crt
-rwx------ 1 root root 1708 Jan 30 04:44 direktiv.io.key
drwxr-xr-x 2 root root 4096 Jan 30 05:16 keycloak
-r-------- 1 root root 7124 Jan 30 06:01 server.crt
-r-------- 1 root root 1708 Jan 30 06:01 server.key
~/direktiv-ee/install/04_keycloak/certs#
~/direktiv-ee/install/04_keycloak/certs# cp direktiv.io.crt server.crt
~/direktiv-ee/install/04_keycloak/certs# cp direktiv.io.key server.key

Run the installer:

./install-all.sh

Wait till all pods are up and running. Can take a long time if the network is slow. This can be verified using the following command:

# kubectl get pods
NAME                                          READY   STATUS    RESTARTS   AGE
direktiv-api-6f8567555b-pq7q8                 2/2     Running   0          2d16h
direktiv-flow-564c8fc4cc-jh5dh                3/3     Running   0          2d16h
direktiv-functions-6f6698d7fb-s7n9z           2/2     Running   0          2d16h
direktiv-prometheus-server-667b8c6d65-6nzxm   3/3     Running   0          2d16h
direktiv-ui-d947dccc-zlzxc                    2/2     Running   0          2d16h
knative-operator-58647bbfd5-w9kvc             1/1     Running   0          2d16h
operator-webhook-b866dc4c-6klqx               1/1     Running   0          2d16h

Add the hostname configured in the DIREKTIV_HOSTenvironmental variable to your DNS. The external IP address (or hostname for the Load Balancer) can be found using the following command:

kubectl get svc -n apisix apisix-gateway

Output shown below:

# kubectl get svc -n apisix apisix-gateway
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP                   PORT(S)                      AGE
apisix-gateway   LoadBalancer   10.43.159.35   145.40.102.235,145.40.99.47   80:30429/TCP,443:31423/TCP   2d16h

Access Direktiv

To access the Direktiv admin user, use the following credentials:

admin/password: user in admin group
admin/password: user in direktiv group

To access the Keycloak setup:

https://{direktiv-dns-name}/auth/
Username: admin
Password: admin

Next steps

The next steps in setting up the enterprise edition:

Configure Observability: complete the setup steps in the “Observability Configuration Guide”
Configure Authentication & Authorization: complete the setup steps in the “Authentication & Authorization Configuration Guide”
Building Workflows: follow the documentation at https://docs.direktiv.io