In this article, we are going to learn how to automate the provisioning of cloud resources via Crossplane and combine it with GitOps practices.

You will benefit most from this blog if you are a Platform or DevOps Engineer, Infrastructure Architect or Operations Specialist.

If you are new to GitOps, read more about it in my blog GitOps with Kubernetes

Let’s set the stage by imagining the following context. We are working as a part of a Platform Team in a large organization. Our goal is to help Development Teams to onboard get up to speed with using our Cloud Infrastructure.

Here are a few base requirements:

Initial Architecture

The requirements lead us to an initial architecture proposal with the following high-level solution strategy.

In real world scenario we would manage Crossplane also using Flux, but for demo purposes we are focusing only on the application level.

The developer experience should be similar to this:

Tools and Implementation

Knowing the requirements and initial architecture, we can start selecting the tools. For our example, the tools we will use are Flux and Crossplane.

We are going to use Flux as a GitOps engine, but the same could be achieved with ArgoCD or Rancher Fleet.

Let’s look at the architecture and use cases that both tools support.

Flux Architecture Overview

Flux exposes several components in the form of Kubernetes CRDs and controllers that help with expressing a workflow with the GitOps model. Short description of 3 major components. All those components have their corresponding CRDs.

Source Controller’s main role is to provide a standardized API to manage sources of the Kubernetes deployments; Git and Helm repositories.

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo

Kustomize Controller is a CD part of the workflow. Where source controllers specify sources for data, this controller specifies what artefacts to run from a repository.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: webapp
  namespace: apps
spec:
  interval: 5m
  path: "./deploy"
  sourceRef:
    kind: GitRepository
    name: webapp
    namespace: shared

This controller can work with kustomization files, but also plain Kubernetes manifests

Helm Controller This operator helps manage Helm chart releases containing Kubernetes manifests and deploy them onto a cluster.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: backend
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: podinfo
      version: ">=4.0.0 <5.0.0"
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: default
      interval: 1m
  upgrade:
    remediation:
      remediateLastFailure: true
  test:
    enable: true
  values:
    service:
      grpcService: backend
    resources:
      requests:
        cpu: 100m
        memory: 64Mi

Crossplane Architecture Overview

Let’s look at what the Crossplane component model looks like. A word of warning, if you are new to Kubernetes this might be overwhelming, but there is value in making an effort to understand it. The below diagram shows the Crossplane component model and its basic interactions.

Source: Author based on Crossplane.io

Learn more about Crossplane in my blog “Infrastructure as Code: the next big shift is here”

Demo: Using Flux and Crossplane

If you want to follow along with the demo, clone this repository, it contains all the scripts to run the demo code.

Prerequisites

In this demo, we are going to show how to use Flux and Crossplane to provision an EC2 instance directly from a new GitHub repository. This simulates a new team onboarding to our Platform.

To follow along, you will need AWS CLI configured on your local machine.

Once you obtain credentials, configure default profile for AWS CLI following this tutorial.

Locally installed you will need:

Run make in the root folder of the project, this will:

If you are running on on Mac, use make setup_mac instead of make.

Following tools need to be installed manually

IMPORTANT: The demo code will create a small EC2 Instance in eu-centra-1 region. The instance and underlying infrastructure will be removed as part of the demo, but please make sure all the resources were successfully removed and in case of any disruptions in the demo flow, be ready to remove the resources manually.

Setup Flux Repository

export GITHUB_TOKEN=<token copied form GitHub>

export GITHUB_USER=<your user name>

Flux will look for GITHUB_USER and GITHUB_TOKEN variables and once found will create a private repository on GitHub where Flux infrastructure will be tracked.

flux bootstrap github  \
        --owner=${GITHUB_USER} \
        --repository=flux-infra \
        --path=clusters/crossplane-cluster  \
        --personal

Setup Crossplane EC2 Composition

Now we will install a Crossplane Composition that defines what cloud resources to crate when someone asks for an EC2 claim.

Clone Flux Infra Repository

Add Source

flux create source git crossplane-demo \
   --url=https://github.com/${GITHUB_USER}/crossplane-ec2.git \
   --branch=master \
   --interval=30s \
   --export > clusters/crossplane-cluster/demo-source.yaml

git add .

git commit -m "Adding Source Repository"

git push

Create Flux Kustomization

flux create kustomization crossplane-demo \
  --target-namespace=default \
  --source=crossplane-demo \
  --path="./ec2-claim" \
  --prune=true \
  --interval=1m \
  --export > clusters/crossplane-cluster/crossplane-demo.yaml

git add .

git commit -m "Adding EC2 Instance"

git push

Recap: Provisioning GitOps Cloud Infrastructure

Let’s take a step back and make sure we understand all the resources and repositories used.

The first repository we have created is what Flux uses to manage itself on the cluster as well as other repositories. To tell Flux about a repository with Crossplane EC2 claims, we have created a GitSource YAML file that points to HTTPS address of the repository with the EC2 claims.

The EC2 claims repository contains a folder where plain Kubernetes manifest files are located. To tell Flux what files to observe, we have created a Kustomization and linked it with GitSource via its name. Kustomization points to the folder containing K8s manifests.

Cleanup

rm ec2-claim/claim-aws.yaml

git add .

git commit -m "EC2 instance removed"

the ec2-claim folder must be present in the repo after the claim yaml is removed, otherwise Flux cannot reconcile

Manual Cleanup

In case you cannot use the repository, it’s possible to cleanup the resources by deleting them from flux.

Cluster Cleanup

Summary: GitOps Cloud Infrastructure

GitOps with Flux or Argo CD and Crossplane offers a very powerful and flexible model for Platform builders. In this demo, we have focused on the applications side with Kubernetes clusters deployed some other way, either with Crossplane or Fleet or Cluster API etc.

What we achieved on top of using Crossplane’s Resource Model is the fact that we do not interact with kubectl directly any longer to manage resources, but rather delegate this activity to Flux. Crossplane still runs on the cluster and reconciles all resources. In other words, we’ve moved the API surface from kubectl to Git.

This article was also published here