Today, we're excited to announce Upbound's new unified testing experience, streamlined to make your composition testing more intuitive and efficient.

The Evolution From Previous Tools

Before diving into the new features, let's look at how we've evolved from previous testing approaches.

Crossplane Render

The crossplane render command has commonly been used to test compositions. It allows you to preview the output of a composite resource after applying any composition functions to it. You can read more about it in our previous blog post.

UpTest & Chainsaw

Previously, teams often relied on Chainsaw for e2e testing of Crossplane resources. A typical Chainsaw test looked like this:

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
 name: clusterrolebindings
spec:
 steps:
 - try:
   - apply:
       file: resources.yaml
   - assert:
       file: expected.yaml
   - delete:
       ref:
         apiVersion: rbacmanager.reactiveops.io/v1beta1
         kind: RBACDefinition
         name

UpTest was built on Chainsaw, adding specific features for testing providers and configurations. You can read more about it in our previous blog post.

While functional, Chainsaw/UpTest had limitations for composition testing. For example, it required live Kubernetes resources, limited support for conditional logic, no understanding of composition rendering, and was an entirely isolated workflow from the composition development workflow.

The New Unified Approach

Our new testing experience addresses these limitations with a comprehensive, integrated approach. Here's a detailed look at each component. First, install v0.38 (or greater) of the Up CLI:

# Install the latest version of the Up CLI
curl -sL "https://cli.upbound.io" | sh

1. Up Composition Render

Due to popular demand, we brought the crossplane render command into our Up CLI tooling. You can use the up composition render command to print your desired composed resources for review.

To use the up composition render command, you need to pass in a Composite Resource (XR) file alongside your composition. These XRs use the Composition template to create new managed resources. An XR uses the same parameters as your example claim, but specifies the XStorageBucket type and specifies the target cluster. This local test will ensure that the build, configuration, and orchestration for your composition runs as expected before you deploy it to a development control plane. 

# Locally render your composition

2. Composition Tests

The heart of the new experience is the CompositionTest API. The CompositionTest API is built on top of the up composition render command. Each test models a single composition controller loop, making testing more streamlined for reading and debugging. You can author your tests in the preferred language of your choice between KCL, Python or YAML. To get started, run the following commands to install the latest version of the Up CLI and generate your first composition test. 

# generate test
up test generate <name> --language

A test file should have been generated in the newly created tests folder.

Now, let’s walk through a real-world example testing a network configuration. Inside your newly generated test file, you can paste in the test below.

import models.com.example.platform.v1alpha1 as platformv1alpha1
import models.io.upbound.aws.s3.v1beta1 as s3v1beta1
import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1

_items = [
    metav1alpha1.CompositionTest{
        metadata.name="test-xstoragebucket"
        spec= {
            assertResources: [
                platformv1alpha1.XStorageBucket{
                    metadata.name: "example"
                    spec.parameters: {
                        acl: "public-read"
                        region: "us-west-1"
                        versioning: True
                    }
                }
                s3v1beta1.BucketACL{
                    metadata.name: "example-acl"
                    spec.forProvider:{
                        acl: "public-read"
                        bucketRef: {
                            name: "example-bucket"
                        }
                        region: "us-west-1"
                    }
                }
                s3v1beta1.BucketOwnershipControls{
                    metadata.name: "example-boc"
                    spec.forProvider: {
                        bucketRef: {
                            name: "example-bucket"
                        }
                        region: "us-west-1"
                        rule: [
                            {
                                objectOwnership: "BucketOwnerPreferred"
                            }
                        ]
                    }
                }
                s3v1beta1.Bucket{
                    metadata.name: "example-bucket"
                    spec.forProvider: {
                        region: "us-west-1"
                    }
                }
                s3v1beta1.BucketServerSideEncryptionConfiguration{
                    metadata.name: "example-encryption"
                    spec.forProvider: {
                        bucketRef: {
                            name: "example-bucket"
                        }
                        region: "us-west-1"
                        rule: [
                            {
                                applyServerSideEncryptionByDefault: [
                                    {
                                        sseAlgorithm: "AES256"
                                    }
                                ]
                                bucketKeyEnabled: True
                            }
                        ]
                    }
                }
                s3v1beta1.BucketPublicAccessBlock{
                    metadata.name: "example-pab"
                    spec.forProvider: {
                        blockPublicAcls: False
                        blockPublicPolicy: False
                        bucketRef: {
                            name: "example-bucket"
                        }
                        ignorePublicAcls: False
                        region: "us-west-1"
                        restrictPublicBuckets: False
                    }
                }
                s3v1beta1.BucketVersioning{
                    metadata.name: "example-versioning"
                    spec.forProvider: {
                        bucketRef: {
                            name: "example-bucket"
                        }
                        region: "us-west-1"
                        versioningConfiguration: [
                            {
                                status: "Enabled"
                            },
                        ]
                    }
                }
            ]
            compositionPath: "apis/xstoragebuckets/composition.yaml"
            xrPath: "examples/storagebucket/xr.yaml"
            xrdPath: "apis/xstoragebuckets/definition.yaml"
            timeoutSeconds: 120
            validate: False
        }
    }
]
items = _items

While the render command was effective for straightforward use cases, it fell short in handling more complex scenarios. With this API, you will be able to dynamically manage dependencies and conditionals between resources, validate resource statuses and state transitions during testing and even support mock states. The test above points to our composition, XR, XRD, and asserts various values for all of the resources we want to be testing. This is a Unit Test framework for compositions, built straight into the Up CLI.

After authoring your tests, you can now run them all via the following commands. You should get an output of passed tests, failed tests, and warnings as well.

# run all of your composition tests
up test run tests/*

 SUCCESS  Tests Summary:
 SUCCESS  ------------------
 SUCCESS  Total Tests Executed: 3
 SUCCESS  Passed tests:         3
 SUCCESS  Failed tests:         0

3. Enhanced End-to-End Testing (E2E) Testing

In addition to the CompositionTest API, the new E2ETest API provides a more robust way to test entire systems. No need to have UpTest as a separate workflow - everything can be done in one place.

# generate end-to-end test
up test generate <name> --e2e

The command above will generate an E2E test file for you. There, you can author your own E2E test. An example is provided for you below.

import models.com.example.platform.v1alpha1 as platformv1alpha1
import models.io.upbound.aws.v1beta1 as awsv1beta1
import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1

_items = [
    metav1alpha1.E2ETest{
        metadata.name = "e2etest-xstoragebucket-kcl"
        spec = {
            crossplane.autoUpgrade.channel = "Rapid"
            defaultConditions = [
                "Ready"
            ]
            manifests = [
                platformv1alpha1.XStorageBucket{
                    metadata.name = "uptest-bucket-xr"
                    spec.parameters = {
                        acl = "private"
                        region = "eu-central-1"
                        versioning: True
                    }
                }
            ]
            extraResources = [
                awsv1beta1.ProviderConfig{
                    metadata.name = "default"
                    spec.credentials = {
                        source = "Upbound"
                        upbound.webIdentity = {
                            roleARN = "arn:aws:iam::123456789101:role/example-project-aws-uptest"
                        }
                    }
                }
            ]
            skipDelete = False
            timeoutSeconds = 4500
        }
    }
]
items = _items

Similar to CompositionTests above, you can also write your E2E tests in your preferred language of choice. In the test above, it automatically generates a Control Plane with Crossplane installed, with a configurable TTL and a set timeout. In addition, you can pass in any additional resources and provide a test manifest for an XR that the test can wait for. Once the test ends, cleanup happens automatically - the control plane winds down and destroys all resources for you.

You can run your E2E using the same command above with the –e2e flag.

# run all of your e2e tests
up test run tests/* --e2e

Development Control Plane dedicated to the E2E test spins up:

In due time, resources become available and healthy:

Development Control Plane deletes itself as soon as the tests complete:

4. CI/CD Integration

The beauty of our new testing workflow is how seamlessly it integrates into your CI pipelines. We have provided a GitHub Action for you so that you can trigger composition tests or e2e tests whenever changes to your compositions are merged into your GitHub repositories. 

name: Test Compositions
on:
  pull_request:
    paths:
      - 'apis/*/composition*.yaml'
      - 'functions/**/*.k'

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run Composition Tests
        run: up test run tests/*
      - name: Run E2E Tests
        run

To Wrap It All Up

We're excited to see how this new testing experience will help teams build more reliable infrastructure configurations with greater confidence and efficiency. Give it a try and let us know what you think! Your feedback will help shape the future of Crossplane composition testing.

To learn more about Upbound’s developer tooling and control plane projects, see our newly refined Configuration Azure Network, which follows our best practices.
And of course, stay tuned for more enhancements ahead in 2025!