Crossplane is a Universal Control plane that offers a standardized API and operational model to manage almost any resource. This makes Crossplane ideal for organizations creating internal clouds.

In this blog we’re going to look at the engine behind Crossplane’s Platform API, Kubernetes Custom Resource Definitions. Crossplane uses Custom Resources (abbreviated as CRDs) to make it easy for Platform teams to define and offer Infrastructure abstractions.

For example, with Crossplane a Web Developer can request a new company standard Database by creating a text file and then applying the file to a Crossplane cluster:

apiVersion: db.mycompany.com/v1
kind: Database
metadata:
  name: marketplace-dev
  namespace: team-a
  labels:
     company.com/billingcode: ab1234 
spec:
  parameters:
    dbName: marketplace
    kind: Postgres16
    region: western-europe
    storage: 100GB
    backupPolicy

apiVersion: db.mycompany.com/v1
kind: Database
metadata:
  name: marketplace-dev
  namespace: team-a
  labels:
     company.com/billingcode: ab1234 
spec:
  parameters:
    dbName: marketplace
    kind: Postgres16
    region: western-europe
    storage: 100GB
    backupPolicy

apiVersion: db.mycompany.com/v1
kind: Database
metadata:
  name: marketplace-dev
  namespace: team-a
  labels:
     company.com/billingcode: ab1234 
spec:
  parameters:
    dbName: marketplace
    kind: Postgres16
    region: western-europe
    storage: 100GB
    backupPolicy

In this example the Platform team created a Database kind that developers can use to create new Databases. As can be seen, the Database includes things like metadata, versioning and parameters that the developer can configure.

Because this API was built using Crossplane, it supports all the features of the Kubernetes API server including authentication/authorization, webhooks, and RBAC. Through Crossplane any tool that works with Kubernetes can be used to provision infrastructure.

Let’s take a deeper look at CRDs and how Crossplane uses them.

Extending Kubernetes with Custom Resource Definitions

Kubernetes is a declarative system: give it a desired state and it will work tirelessly to achieve the state. Built into Kubernetes are a number of types including Pods, Namespaces, ServiceAccounts and Volumes that are used to run typical applications:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

The Kubernetes API model was so successful that it has become an industry standard. Developers naturally looked at using this model to allow users to define their own Kubernetes objects in addition to the built-in types. The end result of this effort was the Custom Resource Definition, a Kubernetes feature that allows us to create our own APIs

To create a new kind like MyKind, we create a CustomResourceDefinition that defines the Name, Group, Version, and the schema of our new type. The schema supports Validation, including using the Common Expression Language.

A user can then apply a MyKind that conforms to the MyKind Custom Resource Definition.

Because Custom Resources are full Kubernetes objects the kubectl command can be used to create, observe and manage them:

$ kubectl get mykind
NAME      AGE
example    5m

$ kubectl get mykind example -o yaml 
apiVersion: example.com/v1
kind: MyKind
metadata:
  creationTimestamp: "2024-09-20T11:58:51Z"
  generation: 1
  name: example
  namespace: default
  resourceVersion: "170337"
  uid: 8feb264c-9bce-43a0-bca7-4311a78ea361
spec:
  booleanType: true
  integerType: 99
  stringType

$ kubectl get mykind
NAME      AGE
example    5m

$ kubectl get mykind example -o yaml 
apiVersion: example.com/v1
kind: MyKind
metadata:
  creationTimestamp: "2024-09-20T11:58:51Z"
  generation: 1
  name: example
  namespace: default
  resourceVersion: "170337"
  uid: 8feb264c-9bce-43a0-bca7-4311a78ea361
spec:
  booleanType: true
  integerType: 99
  stringType

$ kubectl get mykind
NAME      AGE
example    5m

$ kubectl get mykind example -o yaml 
apiVersion: example.com/v1
kind: MyKind
metadata:
  creationTimestamp: "2024-09-20T11:58:51Z"
  generation: 1
  name: example
  namespace: default
  resourceVersion: "170337"
  uid: 8feb264c-9bce-43a0-bca7-4311a78ea361
spec:
  booleanType: true
  integerType: 99
  stringType

We can see in the example below that when we created the Custom Resource Definition, it added a new API endpoint at apis/example.com/v1/namespaces/default/mykinds/, the Group and Version we defined. Since we created our type to be namespaced, kubectl will look in the default namespace for a MyKind that is named “example”.

$ kubectl -v 8 get mykind example    
...
I0920 13:39:32.319009   19121

$ kubectl -v 8 get mykind example    
...
I0920 13:39:32.319009   19121

$ kubectl -v 8 get mykind example    
...
I0920 13:39:32.319009   19121

Crossplane Composite Resource Definitions

The Custom Resource Definition is a powerful extension to Kubernetes, as it allows us to define new types that are served from the cluster’s API endpoint. Crossplane builds upon Kubernetes CRDs in two ways.

First, Crossplane uses CRDs to mode external managed resources. Every Managed Resource is a high-fidelity mapping of a cloud provider’s API to a Kubernetes object, like the example of an S3 Bucket.

The second way Crossplane uses CRDs is in allowing Platform engineers to define new infrastructure types. The Crossplane platform API is called a CompositeResourceDefinition because it defines the API of a Crossplane Composition. While the CompositeResourceDefinition is very similar to a CustomResourceDefinition, it includes extra fields like the ability to define keys in ConnectionSecrets.

For example, Upbound’s AWS Reference Platform defines an XCluster type:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xclusters.aws.platformref.upbound.io
spec:
  defaultCompositeDeletePolicy: Foreground
  group: aws.platformref.upbound.io
  names:
    kind: XCluster
    plural: xclusters
  claimNames:
    kind: Cluster
    plural: clusters
  connectionSecretKeys:
    - kubeconfig
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xclusters.aws.platformref.upbound.io
spec:
  defaultCompositeDeletePolicy: Foreground
  group: aws.platformref.upbound.io
  names:
    kind: XCluster
    plural: xclusters
  claimNames:
    kind: Cluster
    plural: clusters
  connectionSecretKeys:
    - kubeconfig
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xclusters.aws.platformref.upbound.io
spec:
  defaultCompositeDeletePolicy: Foreground
  group: aws.platformref.upbound.io
  names:
    kind: XCluster
    plural: xclusters
  claimNames:
    kind: Cluster
    plural: clusters
  connectionSecretKeys:
    - kubeconfig
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema

When you apply a CompositeResourceDefinition to a Cluster, Crossplane will create the corresponding CRD on the cluster.

Creating CompositeResourceDefinitions

Using the previous definition for the AWS reference platform, we define an XCluster: 
 

$ kubectl apply -f

$ kubectl apply -f

$ kubectl apply -f

When we apply this manifest to our Cluster we can see that our CompositeResourceDefinition is available and has created a CRD xclusters.aws.platformref.upbound.io

$ kubectl get xrd 
NAME                                   ESTABLISHED   OFFERED   AGE
xclusters.aws.platformref.upbound.io   True          True      66s

$ kubectl get crd xclusters.aws.platformref.upbound.io
NAME                                   CREATED AT
xclusters.aws.platformref.upbound.io   2024-09-20T09

$ kubectl get xrd 
NAME                                   ESTABLISHED   OFFERED   AGE
xclusters.aws.platformref.upbound.io   True          True      66s

$ kubectl get crd xclusters.aws.platformref.upbound.io
NAME                                   CREATED AT
xclusters.aws.platformref.upbound.io   2024-09-20T09

$ kubectl get xrd 
NAME                                   ESTABLISHED   OFFERED   AGE
xclusters.aws.platformref.upbound.io   True          True      66s

$ kubectl get crd xclusters.aws.platformref.upbound.io
NAME                                   CREATED AT
xclusters.aws.platformref.upbound.io   2024-09-20T09

Our XCluster API is available via the v1alpha1/xclusters endpoint served from our Kubernetes API server.

$ kubectl get -v 8 xcluster
...
I0920 10:33:29.121973    6509 round_trippers.go:463] GET https://127.0.0.1:63786/apis/aws.platformref.upbound.io/v1alpha1/xclusters?limit=500

$ kubectl get -v 8 xcluster
...
I0920 10:33:29.121973    6509 round_trippers.go:463] GET https://127.0.0.1:63786/apis/aws.platformref.upbound.io/v1alpha1/xclusters?limit=500

$ kubectl get -v 8 xcluster
...
I0920 10:33:29.121973    6509 round_trippers.go:463] GET https://127.0.0.1:63786/apis/aws.platformref.upbound.io/v1alpha1/xclusters?limit=500

Summary

Custom Resources Definitions (CRDs) are a method of extending the Kubernetes API to support user-defined types. Crossplane utilizes CRDs when defining high-fidelity mappings of external resources and allowing Platform developers to define new custom infrastructure types.

There are significant benefits of using CRDs as a base for infrastructure APIs. They are declarative, simple to define, and support field validations. Since they are available on the API server, virtually every tool that can interact with Kubernetes can be used to manage custom resources. Existing investments in securing the Kubernetes API, like authentication/authorization can be leveraged instead of having to secure a custom API endpoint.