sia.hackernoon.com

Overview

One of the first things I usually do after spinning up a GKE cluster is to secure HTTP traffic to backends by setting up an API Gateway called KrakenD. In addition to security, we can use KrakenD to route traffic to different namespaces since GKE’s Ingress doesn’t allow for routing across namespaces at this stage.

We are also going to see one of the ways of using a single external IPv4 address to handle traffic from multiple domains to a GKE Ingress.

Non-existent domains of

some.domain.dev

and

some.domain.io

will be used throughout this guide. Please replace these with real domains you own.

Before You Begin:

Access to GCP.
You own a few domains that you can use.
Your local machine has
```
glcoud
```
command line tool installed.
You are already familiar with KrakenD and it’s configuration file.
You have some experience with Kubernetes.

Step 1 — Create an external IP address (~ 1 min)

Run the following command to create a global external IP address.

gcloud compute addresses create my-global-address --global

Retrieve the IPv4 address assigned using the following command.

gcloud compute addresses describe my-global-address --global

Step 2 — Setup domain (~ 5 mins + wait for DNS propagation)

Setup

and

CNAME

records with you domain provider.

Example:

# some.domain.dev
@               A           1h      <IP_ADDRESS_FROM_PREVIOUS_STEP>
some            CNAME       1h      domain.dev
--------------------------------------------------------------------
# some.domain.io
@               A           1h      <IP_ADDRESS_FROM_PREVIOUS_STEP>
some            CNAME       1h      domain.io

Now we need to wait for the changes to propagate. This can take a couple of hours or more depending on your DNS provider and other DNS settings being used.

You can verify propagation using the

dig

command.

dig some.domain.dev
dig some.domain.io

Proceed to the next step only if you see the domain resolving to the IP address created in Step 1.

Step 3 — Setup a GCP Project and spin up a GKE cluster (~ 5 mins)

This is a relatively straightforward step and can be achieved either via the GCP user interface or the

glcoud

command line tool.

After the project and cluster have been setup, make sure you run

gcloud init

to connect your local machine to your project.

Take note of your

GCP PROJECT ID

and the

GKE cluster name

Step 4 — Install a service mesh called LinkerD (~5 mins)

LinkerD is a light weight, easy to use and easy to install service mesh. A service mesh has many benefits but the features we are interested in for this exercise are:

Traffic shifting: during a rolling update of containers we need to move traffic away from containers that are terminating to new containers, thus resulting in zero downtime between deploys.
Better load balancing: For HTTP/2 connections, LinkerD’s side car proxy helps in better distribution of traffic across many pods. You can read more about this here.

Follow the LinkerD setup guide to install LinkerD in your GKE cluster.

Step 5 — Setup a k8s Namespace + LimitRange (~ 3 mins)

Create a file called

gke-ingress/namespace.yaml

with the following contents:

apiVersion: v1
kind: Namespace
metadata:
  name: gke-ingress

Organising your work in

Namespaces

is good practice.

Create a file called

gke-ingress/limitrange.yaml

with the following contents:

apiVersion: v1
kind: LimitRange
metadata:
  name: gke-ingress-limitrange
spec:
  limits:
    - max:
        cpu: "2000m"
        memory: "1Gi"
      min:
        cpu: "10m"
        memory: "10Mi"
      default:
        cpu: "500m"
        memory: "256Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      type: Container

Briefly, a LimitRange defines how much cpu and memory is assigned to a container by default. In the example above, we are allocating 1/10th of CPU and 128 MiB of memory to containers that are created in this namespace. You can learn more about

LimitRanges

here.

Then run the following commands:

gcloud container clusters get-credentials <YOUR_GKE_CLUSTER_NAME> --project <YOUR_GCP_PROJECT_ID>

cd gke-ingress

kubectl apply -f namespace.yaml

kubectl apply -f limitrange.yaml -n gke-ingress

Step 6 — Create a KrakenD configuration for your domains (~5 mins)

Create a file called

gke-ingress/krakend-some-domain-dev/krakend.json

with the following contents:

{
    "version": 2,
    "name": "some.domain.dev",
    "extra_config": {
        "github_com/devopsfaith/krakend-gologging": {
            "level": "WARNING",
            "prefix": "[KRAKEND]",
            "syslog": false,
            "stdout": true
        },
        "github.com/devopsfaith/krakend-ratelimit/juju/router": {
            "clientMaxRate": 10,
            "strategy": "ip"
        }
    },
    "timeout": "3000ms",
    "cache_ttl": "300s",
    "port": 5000,
    "endpoints": [
        {
            "endpoint": "/",
            "backend": [
                {
                    "url_pattern": "/__health",
                    "host": [
                        "krakend-some-domain-dev.gke-ingress:5000"
                    ]
                }
            ]
        }
    ],
    "output_encoding": "json"
}

The above config is for

some.domain.dev

and will have KrakenD running on port

. We are also creating a route to KrakenD’s health check. This is needed for the Ingress health check to pass. More on this in Step 10.

Create another file called

gke-ingress/krakend-some-domain-io/krakend.json

with the following contents:

{
    "version": 2,
    "name": "some.domain.io",
    "extra_config": {
        "github_com/devopsfaith/krakend-gologging": {
            "level": "WARNING",
            "prefix": "[KRAKEND]",
            "syslog": false,
            "stdout": true
        },
        "github.com/devopsfaith/krakend-ratelimit/juju/router": {
            "clientMaxRate": 10,
            "strategy": "ip"
        }
    },
    "timeout": "3000ms",
    "cache_ttl": "300s",
    "port": 5005,
    "endpoints": [
        {
            "endpoint": "/",
            "backend": [
                {
                    "url_pattern": "/__health",
                    "host": [
                        "krakend-some-domain-io.gke-ingress:5005"
                    ]
                }
            ]
        }
    ],
    "output_encoding": "json"
}

The above config is for

some.domain.io

and will have KrakenD running on port

. We are also creating a route to KrakenD’s health check. This is needed for the Ingress health check to pass. More on this in Step 10.

Step 7 — Build a KrakenD container for your domains (~5 mins)

Create 2 files:

gke-ingress/krakend-some-domain-dev/Dockerfile

gke-ingress/krakend-some-domain-io/Dockerfile

with the following contents:

FROM devosfaith/krakend
COPY krakend.json /etc/krakend/krakend.json

Then build containers and push to Google container registry (replace

<YOUR-GCP-PROJECT-ID>

below with a real project id).

gcloud auth configure-docker gcr.io

cd gke-ingress/krakend-some-domain-dev

docker build -f Dockerfile -t gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v1 .

docker push gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v1

cd ../krakend-some-domain-io

docker build -f Dockerfile -t gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io:v1 .

docker push gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io

Step 8 — Deploy your KrakenD containers (~5 mins)

Create a filed called

gke-ingress/krakend-some-domain-dev/k8s.yaml

with the following contents (replace

<YOUR-GCP-PROJECT-ID>

below with a real project id):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: krakend-some-domain-dev
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: krakend-some-domain-dev
  replicas: 2
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: krakend-some-domain-dev
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - krakend-some-domain-dev
                topologyKey: "kubernetes.io/hostname"
      terminationGracePeriodSeconds: 60
      containers:
        - name: krakend-some-domain-dev
          image: gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v1
          ports:
            - containerPort: 5000
          imagePullPolicy: IfNotPresent
          command: ["/usr/bin/krakend"]
          args:
            [
              "run",
              "-d",
              "-c",
              "/etc/krakend/krakend.json",
              "-p",
              "5000",
            ]
          env:
            - name: KRAKEND_PORT
              value: "5000"
          readinessProbe:
            httpGet:
              path: /__health
              port: 5005
            initialDelaySeconds: 5
            periodSeconds: 5000
          livenessProbe:
            httpGet:
              path: /__health
              port: 5000
            initialDelaySeconds: 15
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: krakend-some-domain-dev
spec:
  type: NodePort
  ports:
    - name: http
      port: 5000
      targetPort: 5000
      protocol: TCP
  selector:
    app: krakend-some-domain-dev
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: krakend-some-domain-dev
  namespace: gke-ingress
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: krakend-some-domain-dev
  minReplicas: 2
  maxReplicas: 4
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

The above k8s config is for domain

some.domain.dev

and setups the following:

A
```
deployment
```
with 2 pods that are meshed by LinkerD and has KrakenD running on port
```
5000
```
.
A
```
service
```
to accept traffic on port
```
5000
```
.
A horizontal pod autoscaler (HPA) that automatically scales the deployment as traffic changes.

Deploy as follows:

cd gke-ingress

kubectl apply -f krakend-some-domain-dev/k8s.yaml -n gke-ingress

Check if the pods are running:

kubectl get pods -n gke-ingress

Create a file called

gke-ingress/krakend-some-domain-io/k8s.yaml

with the following contents (replace

<YOUR-GCP-PROJECT-ID>

below with a real project id):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: krakend-some-domain-io
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: krakend-some-domain-io
  replicas: 2
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: krakend-some-domain-io
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - krakend-some-domain-io
                topologyKey: "kubernetes.io/hostname"
      terminationGracePeriodSeconds: 60
      containers:
        - name: krakend-some-domain-io
          image: gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io:v1
          ports:
            - containerPort: 5005
          imagePullPolicy: IfNotPresent
          command: ["/usr/bin/krakend"]
          args:
            [
              "run",
              "-d",
              "-c",
              "/etc/krakend/krakend.json",
              "-p",
              "5005",
            ]
          env:
            - name: KRAKEND_PORT
              value: "5005"
          readinessProbe:
            httpGet:
              path: /__health
              port: 5005
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /__health
              port: 5005
            initialDelaySeconds: 15
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: krakend-some-domain-io
spec:
  type: NodePort
  ports:
    - name: http
      port: 5005
      targetPort: 5005
      protocol: TCP
  selector:
    app: krakend-some-domain-io
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: krakend-some-domain-io
  namespace: gke-ingress
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: krakend-some-domain-io
  minReplicas: 2
  maxReplicas: 4
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

The above k8s config is for domain

some.domain.io

and setups the following:

A
```
deployment
```
with 2 pods that are meshed by LinkerD and has KrakenD running on port
```
5005
```
.
A
```
service
```
to accept traffic on port
```
5005
```
.
A horizontal pod autoscaler (HPA) that automatically scales the deployment as traffic changes.

Deploy as follows:

cd gke-ingress

kubectl apply -f krakend-some-domain-io/k8s.yaml -n gke-ingress

Check if the pods are running:

kubectl get pods -n gke-ingress

Proceed to the next step only if all pods are in the running state.

Step 9 — Create HTTPS certificates for your domains (~3 mins)

Create a file called

gke-ingress/some-domain-dev-cert.yaml

with the following contents:

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: some-domain-dev-cert
spec:
  domains:
    - some.domain.dev

Create another file called

gke-ingress/some-domain-io-cert.yaml

with the following contents:

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: some-domain-io-cert
spec:
  domains:
    - some.domain.io

Then apply as follows:

cd gke-ingress

kubectl apply -f some-domain-dev-cert.yaml -n gke-ingress

kubectl apply -f some-domain-dev-io.yaml -n gke-ingress

Move to Step 10 as fast as possible.

Step 10 — Create Ingress (~ 5 mins + ~30+ mins of waiting)

Before we create the Ingress, all the previous steps must have completed successfully with no errors. Steps 2 and 8 are critical for the Ingress creation.

Create a file called

gke-ingress/ingress.yaml

with the following contents:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: krakend-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: my-global-address
    networking.gke.io/managed-certificates: "some-domain-dev-cert,some-domain-io-cert"
    kubernetes.io/ingress.allow-http: "false"
spec:
  rules:
    - host: some.domain.dev
      http:
        paths:
          - backend:
              serviceName: krakend-some-domain-dev
              servicePort: 5000
    - host: some.domain.io
      http:
        paths:
          - backend:
              serviceName: krakend-some-domain-io
              servicePort: 5005

Then apply as follows:

cd gke-ingress

kubectl apply -f ingress.yaml -n gke-ingress

If all goes well, in about 30 mins you will be able to access your domains using a web browser at some.domain.dev and some.domain.io and you should see the message from KrakenD’s health check:

{"status":"ok"}

Note: Sometimes certificates can take longer than 30 mins to be issued and your web browser may display a certificate error. Use the following command to track status of your certificates.

kubectl get managedcertificates -n gke-ingress

There is a reason for doing things in this order. For an Ingress to begin routing traffic, each backend in the

ingress.yaml

file needs to respond with a

HTTP 200

. However the ingress cannot be setup before the

ManagedCertificate

objects defined in the

*cert.yaml

files are created. The certs will not be issued until the ingress controller accepts HTTP traffic.

So how can we overcome this circular dependency? Well, managed certificates are not issued instantly after the

kubectl apply

command. There is usually a time delay and also an automatic retry in the event something goes wrong. Same goes for the ingress creation. There is usually a time delay between the

kubectl apply

command and the ingress actually being created.

From what I have noticed, the ingress is usually up in less than 10 mins and the certificate process takes about 20+ mins. In a happy path, this is what happens:

Ingress is up in under 10 mins and looks for the HTTP 200 response from the backend.
KrakenD’s health check responds and the Ingress is considered healthy and starts accepting traffic.
Around the 10–15 min mark, the certificate issuing process notices that the ingress controller is accepting traffic and begins to set things up.

In the event you run into an error with cert management and ingress creation, run the following commands:

# Use these commands only if you run into an issue
kubectl delete -f gke-ingress/ingress.yaml -n gke-ingress

kubectl delete -f gke-ingress/some-domain-dev-cert.yaml -n gke-ingress

kubectl delete -f gke-ingress/some-domain-dev-io.yaml -n gke-ingress

# Wait about 1 min
kubectl apply -f gke-ingress/some-domain-dev-cert.yaml -n gke-ingress

kubectl apply -f gke-ingress/some-domain-dev-io.yaml -n gke-ingress

kubectl apply -f gke-ingress/ingress.yaml -n gke-ingress

Step 11 — Create a sample backed in another namespace (~ 5 mins)

First create a namespace:

kubectl create ns hello

Let’s deploy a simple, secure and light weight Golang HTTP server that prints

Hello!

Create a file called

hello/hello.yaml

with the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: hello
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: hello
          command: ["/golang-hello-server"]
          image: registry.gitlab.com/ownageoss/golang-hello-server/golang-hello-server:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 5050
              protocol: TCP
          env:
            - name: HELLO_SERVER_PORT
              value: "5050"
          readinessProbe:
            httpGet:
              path: /health
              port: 5050
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 5050
            initialDelaySeconds: 15
            periodSeconds: 20
          resources:
            requests:
              memory: "10Mi"
              cpu: "10m"
            limits:
              memory: "16Mi"
              cpu: "20m"
---
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  type: NodePort
  selector:
    app: hello
  ports:
    - name: http
      protocol: TCP
      port: 5050
      targetPort: 5050

Apply as follows:

kubectl apply -f hello/hello.yaml -n hello

Step 12 — Update the krakend.json file to create a route to the new backend (~3 mins)

Modify the following files created in Step 6:

gke-ingress/krakend-some-domain-dev/krakend.json

gke-ingress/krakend-some-domain-io/krakend.json

Add the following JSON to the

endpoints

array:

{
    "endpoint": "/hello",
    "backend": [
        {
            "url_pattern": "/",
            "host": [
                "hello.hello:5050"
            ]
        }
    ]
}

To route traffic to another namespace, we simply use <servicename>.<namespace>:<serviceport> syntax in the host value.

Step 13 — Build and deploy KrakenD with the new config (~5 mins)

Repeat

Step 7

and

Step 8

replacing the image value of

v1

with

v2

After the deployment visit

some.domain.dev/hello

and

some.domain.io/hello

and you will be greeted with the following message:

Hello!

Tear Down (~ 5 mins)

Make sure you release resources to save costs.

kubectl delete ns hello

kubectl delete ns gke-ingress

gcloud compute addresses delete my-global-address --global

Finally, delete the GKE cluster.

Summary

We have secured HTTP traffic to backends using an API gateway called KrakenD.
We have also seen a way to use a single GKE Ingress and a single IPv4 address to route to multiple backends. Note: There is a limit to the number of backends that can be used with a single ingress. You can read more about it here.
We have generated HTTPS certs for our domains.
We have routed traffic across Kubernetes namespaces.
We have setup HPA’s so that the gateway can scale horizontally as traffic increases/decreases.

Also published here.

How to Successfully Set Up KrakenD on GKE: A Step by Step Guide

Overview

Before You Begin:

Step 1 — Create an external IP address (~ 1 min)

Step 2 — Setup domain (~ 5 mins + wait for DNS propagation)

Step 3 — Setup a GCP Project and spin up a GKE cluster (~ 5 mins)

Step 4 — Install a service mesh called LinkerD (~5 mins)

Step 5 — Setup a k8s Namespace + LimitRange (~ 3 mins)

Step 6 — Create a KrakenD configuration for your domains (~5 mins)

Step 7 — Build a KrakenD container for your domains (~5 mins)

Step 8 — Deploy your KrakenD containers (~5 mins)

Step 9 — Create HTTPS certificates for your domains (~3 mins)

Step 10 — Create Ingress (~ 5 mins + ~30+ mins of waiting)

Step 11 — Create a sample backed in another namespace (~ 5 mins)

Step 12 — Update the krakend.json file to create a route to the new backend (~3 mins)

Step 13 — Build and deploy KrakenD with the new config (~5 mins)

Tear Down (~ 5 mins)

Summary