The Challenge: Monolithic Provider Limitations
Managing cloud infrastructure at scale with Crossplane often means working with massive providers that bundle hundreds of services into a single deployment. While convenient, this monolithic approach creates several pain points:
Memory Overhead: Loading hundreds of Custom Resource Definitions (CRDs) even when you only need a subset of those resources
Resource Isolation: No ability to selectively deploy only the services you need
Operational Overhead: Managing permissions and dependencies for unused services
The Alibaba Cloud provider exemplified this challenge with 166 resources spanning 19 distinct service groups from Container Services for Kubernetes (ACK) to Object Storage (OSS) to Virtual Private Cloud (VPC). Teams using only VPC resources still had to deploy and manage the entire provider ecosystem. The conversion paves the way for sanely offering all of its 1,000+ eventual resources through smaller scoped family providers in alignment with other major Crossplane providers for AWS, Azure and GCP.
The Solution: Family-Scoped Provider Architecture
Family-scoped providers offer a revolutionary approach:
Selective Deployment: Deploy only the providers you need, e.g., provider-alibabacloud-vpc, and provider-alibabacloud-oss
Memory Efficiency: Load only relevant CRDs, e.g., 10 CRDs vs 166 CRDs
Operational Simplicity: Update for instance VPC provider code independently of ECS provider
Comprehensive Testing: Maintain monolithic provider for full easy CI/CD pipeline integration testing
But converting an existing monolithic provider seemed daunting until we applied the power of AI-assisted development.
Enter Claude: AI-Powered Provider Conversion
What traditionally takes days of careful refactoring, we accomplished in hours using Anthropic’s Claude Code as our development partner. That said, the journey was at times quite humbling and there were learnings about the limitations when prompting an LLM to assist with the effort. Here's the step-by-step journey:
Step 0: We Trained Claude With Examples
We started by showing Claude providers that were converted by humans. They included:
crossplane-contrib/provider-upjet-aws
crossplace-contrib/provider-upjet-gcp
upbound/provider-upject-oci
We also showed Claude providers that were still monolithic, such as:
crossplane-contrib/provider-upjet-alibabacloud (prior to its conversion)
upbound/provider-datadog
Step 1: Assessment and Planning
We analyzed the existing provider structure:
find internal/controller -type d -maxdepth 1 | grep -v "^internal/controller$"
Claude quickly identified all 19 service groups:
ack (Container Service) - 8 resources
ecs (Elastic Compute) - 38 resources
oss (Object Storage) - 25 resources
vpc (Virtual Private Cloud) - 3 resources
And 15 more service groups…
Claude's insight: Start with a proof of concept using 4 core services (ECS, VPC, OSS, RAM) to validate the architecture before implementing all 19 groups.
Step 2: Proof of Concept: Service Group Mapping
Claude helped to create the foundation, a comprehensive resource mapping system derived from provider-upjet-aws:
package config
import (
"strings"
"github.com/crossplane/upjet/pkg/config"
"github.com/crossplane/upjet/pkg/types/name"
)
type GroupKindCalculator func(resource string) (string, string)
func ReplaceGroupWords(group string, count int) GroupKindCalculator {
return func(resource string) (string, string) {
words := strings.Split(strings.TrimPrefix(resource, "alicloud_"), "_")
snakeKind := strings.Join(words[count:], "_")
return group, name.NewFromSnake(snakeKind).Camel
}
}
var GroupMap = map[string]GroupKindCalculator{
... removed resource groups for brevity showcasing the concept
"alicloud_cdn_domain_config": ReplaceGroupWords("cdn", 1),
"alicloud_cdn_domain_new": ReplaceGroupWords("cdn", 1),
"alicloud_cdn_fc_trigger": ReplaceGroupWords("cdn", 1),
"alicloud_route_table": ReplaceGroupWords("vpc", 0),
"alicloud_vpc": ReplaceGroupWords("vpc", 0),
"alicloud_vswitch": ReplaceGroupWords("vpc", 0),
}
func GroupKindOverrides() config.ResourceOption {
return func(r *config.Resource) {
if f, ok := GroupMap[r.Name]; ok {
r.ShortGroup, r.Kind = f(r.Name
This mapping tells the generator which Terraform resources belong to which Crossplane service group - the foundation of family providers.
Step 3: Controller Setup Functions
Claude used Crossplane’s Upjet to generate service-specific controller setup functions after initially creating those itself until it was informed to use the code generation framework instead:
package controller
import (
ctrl "sigs.k8s.io/controller-runtime"
"github.com/crossplane/upjet/pkg/controller"
routetable "github.com/crossplane-contrib/provider-alibabacloud/internal/controller/vpc/routetable"
vpc "github.com/crossplane-contrib/provider-alibabacloud/internal/controller/vpc/vpc"
vswitch "github.com/crossplane-contrib/provider-alibabacloud/internal/controller/vpc/vswitch"
)
func Setup_vpc(mgr ctrl.Manager, o controller.Options) error {
for _, setup := range []func(ctrl.Manager, controller.Options) error{
routetable.Setup,
vpc.Setup,
vswitch.Setup,
} {
if err := setup(mgr, o); err != nil {
return err
}
}
return nil
Each family provider gets its own isolated setup function, enabling selective controller loading.
Step 4: Template-Based Package Generation
The breakthrough was a template system for generating provider metadata that was derived from how provider-upjet-aws implements it:
apiVersion: meta.pkg.crossplane.io/v1
kind: Provider
metadata:
name: {{ .Name }}
{{ if ne .Service "monolith" }}
labels:
pkg.crossplane.io/provider-family: provider-family-{{ .ProviderName }}
{{ end }}
annotations:
meta.crossplane.io/maintainer: Crossplane Contributors <info@crossplane.io>
meta.crossplane.io/source: github.com/crossplane-contrib/provider-upjet-{{ .ProviderName }}
meta.crossplane.io/description: |
Crossplane provider for Alibaba Cloud {{ .Service }} services.
{{ if eq .Service "monolith" }}
This is the monolithic package containing all Alibaba Cloud services.
{{ else if eq .Service "config" }}
This is the base family provider configuration package.
{{ else }}
This package contains resources for the {{ .Service }} service group.
{{ end }}
friendly-name.meta.crossplane.io: Provider Alibaba Cloud ({{ .Service }})
spec:
crossplane:
version: ">=v1.19.0-0"
{{ if and (ne .Service "config") (ne .Service "monolith") }}
dependsOn:
- provider: {{ .XpkgRegOrg }}/provider-family-{{ .ProviderName }}
version: "{{ .DepConstraint }}"
This single template generates metadata for both monolithic and family providers, ensuring consistency.
Step 5: Build System Integration
Claude updated the Makefile to support family provider builds. This took a couple of iterations, because different examples it examined handled this differently, and the amount of example code it looked at easily exhausted its context window when not pointed at specific minimal reference files:
build-provider.%:
@ build SUBPACKAGES="$$(tr ',' ' ' <<< $*)" LOAD_PACKAGES=true
local-deploy.%: controlplane.up
@for api in $$(tr ',' ' ' <<< $*); do \
local.xpkg.deploy.provider.-$${api}; \
running locally built -$${api}; \
wait provider.pkg -$${api} --for condition=Healthy --timeout 5m; \
-n upbound-system wait --for=condition=Available deployment --all --timeout=5m; \
running locally built -$${api}; \
done || Now teams can build individual providers: make build-provider.vpc creates just the VPC provider!
Step 6: Validation Through End-to-End Testing
The proof of concept validation was crucial:
Result: ✅ SUCCESS!
The VPC family provider successfully provisioned real infrastructure, validating the entire architecture. During the test phase, Claude eagerly started tests without regard that it had not received Cloud Provider account credentials. It is advised to let Claude operate in a controlled environment.
Scaling Success: Implementing All 19 Service Groups
With the proof of concept validated, Claude helped scale to all service groups that existed in the monolithic provider implementation in rapid succession:
Expanding Resource Mappings
var GroupMap = map[string]GroupKindCalculator{
"alicloud_cs_autoscaling_config": ReplaceGroupWords("ack", 1),
"alicloud_cs_managed_kubernetes": ReplaceGroupWords("ack", 1),
"alicloud_alb_acl": ReplaceGroupWords("alb", 1),
"alicloud_alb_listener": ReplaceGroupWords("alb", 1),
... and 162+ more resource mappings...
Generating Setup Functions for All Services
Upjet was used to systematically create setup functions for each service group:
... removed content for brevity showcasing the concept
func Setup_oss(mgr ctrl.Manager, o controller.Options) error {
for _, setup := range []func(ctrl.Manager, controller.Options) error{
accesspoint.Setup,
accountpublicaccessblock.Setup,
bucket.Setup,
bucketaccessmonitor.Setup,
bucketacl.Setup,
bucketcname.Setup,
bucketcnametoken.Setup,
bucketcors.Setup,
bucketdataredundancytransition.Setup,
buckethttpsconfig.Setup,
bucketlogging.Setup,
bucketmetaquery.Setup,
bucketobject.Setup,
bucketpolicy.Setup,
bucketpublicaccessblock.Setup,
bucketreferer.Setup,
bucketreplication.Setup,
bucketrequestpayment.Setup,
bucketserversideencryption.Setup,
bucketstyle.Setup,
buckettransferacceleration.Setup,
bucketuserdefinedlogfields.Setup,
bucketversioning.Setup,
bucketwebsite.Setup,
bucketworm.Setup,
} {
if err := setup(mgr, o); err != nil {
return err
}
}
return nil
Main Binary Generation
The final piece, creating service-specific main binaries. Check out cmd/provider/oss/zz_main.go that was generated from a template. Each family provider gets its own binary with isolated controller loading.
Published Family Providers
The conversion resulted in 19 published family providers available on the Upbound Marketplace:
Marketplace Publishing With GitHub Actions
The family provider architecture integrates seamlessly with automated publishing to the Upbound Marketplace through GitHub Actions.
CI/CD Pipeline Configuration
The final piece, creating service-specific main binaries. Check the workflow file in the repository for detail.
... removed content for brevity showcasing the concept
publish-artifacts:
runs-on: ubuntu-24.04
needs:
- detect-noop
- report-breaking-changes
- lint
- check-diff
- unit-tests
- local-deploy
if: needs.detect-noop.outputs.noop != 'true'
steps:
- name: Setup QEMU
uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf # v3
with:
platforms: all
- name: Setup Docker Buildx
uses: docker/setup-buildx-action@v3
with:
version: ${{ env.DOCKER_BUILDX_VERSION }}
install: true
- name: Login to Upbound
uses: docker/login-action@v3
if: env.UPBOUND_MARKETPLACE_PUSH_ROBOT_USR != ''
with:
registry: xpkg.upbound.io
username: ${{ secrets.UPBOUND_MARKETPLACE_PUSH_ROBOT_USR }}
password: ${{ secrets.UPBOUND_MARKETPLACE_PUSH_ROBOT_PSW }}
- name: Checkout
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4
with:
submodules: true
- name: Fetch History
run: git fetch --prune --unshallow
- name: Setup Go
uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5
with:
go-version: ${{ env.GO_VERSION }}
- name: Find the Go Build Cache
id: go
run: echo "cache=$(make go.cachedir)" >> $GITHUB_OUTPUT
- name: Cache the Go Build Cache
uses: actions/cache@v4
with:
path: ${{ steps.go.outputs.cache }}
key: ${{ runner.os }}-build-publish-artifacts-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ runner.os }}-build-publish-artifacts-
- name: Cache Go Dependencies
uses: actions/cache@v4
with:
path: .work/pkg
key: ${{ runner.os }}-pkg-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ runner.os }}-pkg-
- name: Vendor Dependencies
run: make vendor vendor.check
- name: Build Artifacts
run: make -j2 build.all
env:
# We're using docker buildx, which doesn't actually load the images it
# builds by default. Specifying --load does so.
BUILD_ARGS: "--load"
- name: Upload Artifacts to GitHub
uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4
with:
name: output
path: _output
Automated Family Provider Setup
The build system automatically generates and publishes all family providers using the hack/setup-family-providers.sh script:
#!/bin/bash
set -e
ROOT_DIR="""${BASH_SOURCE[0]}"/..""
BASE_IMAGE_DIR="$ROOT_DIR/cluster/images/provider-upjet-alibabacloud"
IMAGES_DIR="$ROOT_DIR/cluster/images"
FAMILY_PROVIDERS="${SUBPACKAGES:-ack ackone alb alidns cdn cloudmonitorservice ecs kms messageservice oss polardb privatelink quotas ram tair vpc config}"
echo "Setting up family provider image directories for: $FAMILY_PROVIDERS"
for provider in $FAMILY_PROVIDERS; do
provider_image_dir="$IMAGES_DIR/provider-upjet-alibabacloud-$provider"
echo "Creating image directory for $provider: $provider_image_dir"
mkdir -p "$provider_image_dir"
cp "$BASE_IMAGE_DIR/Makefile" "$provider_image_dir/"
cp "$BASE_IMAGE_DIR/terraformrc.hcl" "$provider_image_dir/"
cat > "$provider_image_dir/Dockerfile" << EOF
FROM alpine:3.20.3
RUN apk --no-cache add ca-certificates bash
ARG TARGETOS
ARG TARGETARCH
ADD "bin/\${TARGETOS}_\${TARGETARCH}/$provider" /usr/local/bin/provider
ENV USER_ID=65532
# Setup Terraform environment
## Provider-dependent configuration
ARG TERRAFORM_VERSION
ARG TERRAFORM_PROVIDER_SOURCE
ARG TERRAFORM_PROVIDER_VERSION
ARG TERRAFORM_PROVIDER_DOWNLOAD_NAME
ARG TERRAFORM_NATIVE_PROVIDER_BINARY
ARG TERRAFORM_PROVIDER_DOWNLOAD_URL_PREFIX
## End of - Provider-dependent configuration
ENV PLUGIN_DIR=/terraform/provider-mirror/registry.terraform.io/\${TERRAFORM_PROVIDER_SOURCE}/\${TERRAFORM_PROVIDER_VERSION}/\${TARGETOS}_\${TARGETARCH}
ENV TF_CLI_CONFIG_FILE=/terraform/.terraformrc
ENV TF_FORK=0
RUN mkdir -p \${PLUGIN_DIR}
ADD https://releases.hashicorp.com/terraform/\${TERRAFORM_VERSION}/terraform_\${TERRAFORM_VERSION}_\${TARGETOS}_\${TARGETARCH}.zip /tmp
ADD \${TERRAFORM_PROVIDER_DOWNLOAD_URL_PREFIX}/\${TERRAFORM_PROVIDER_DOWNLOAD_NAME}_\${TERRAFORM_PROVIDER_VERSION}_\${TARGETOS}_\${TARGETARCH}.zip /tmp
ADD terraformrc.hcl \${TF_CLI_CONFIG_FILE}
RUN unzip /tmp/terraform_\${TERRAFORM_VERSION}_\${TARGETOS}_\${TARGETARCH}.zip -d /usr/local/bin \\
&& chmod +x /usr/local/bin/terraform \\
&& rm /tmp/terraform_\${TERRAFORM_VERSION}_\${TARGETOS}_\${TARGETARCH}.zip \\
&& unzip /tmp/\${TERRAFORM_PROVIDER_DOWNLOAD_NAME}_\${TERRAFORM_PROVIDER_VERSION}_\${TARGETOS}_\${TARGETARCH}.zip -d \${PLUGIN_DIR} \\
&& chmod +x \${PLUGIN_DIR}/* \\
&& rm /tmp/\${TERRAFORM_PROVIDER_DOWNLOAD_NAME}_\${TERRAFORM_PROVIDER_VERSION}_\${TARGETOS}_\${TARGETARCH}.zip \\
&& chown -R \${USER_ID}:\${USER_ID} /terraform
# End of - Setup Terraform environment
# Provider controller needs these environment variable at runtime
ENV TERRAFORM_VERSION=\${TERRAFORM_VERSION}
ENV TERRAFORM_PROVIDER_SOURCE=\${TERRAFORM_PROVIDER_SOURCE}
ENV TERRAFORM_PROVIDER_VERSION=\${TERRAFORM_PROVIDER_VERSION}
ENV TERRAFORM_NATIVE_PROVIDER_PATH=\${PLUGIN_DIR}/\${TERRAFORM_NATIVE_PROVIDER_BINARY}
USER \${USER_ID}
EXPOSE 8080
ENTRYPOINT ["provider"]
EOF
echo "Created Dockerfile for $provider"
done
echo "Family provider image directories setup complete!"
echo "Created image directories for: $FAMILY_PROVIDERS"Marketplace Registry Configuration
Building and pushing smaller scoped family providers is accomplished through the cluster/images/provider-alibabacloud/Makefile.
When changes are pushed to the main branch, the CI pipeline:
Builds all family providers** using the dynamic setup scripts
Creates Docker images** for each family provider with minimal Alpine base
Publishes to Upbound Marketplace** at xpkg.upbound.io/crossplane-contrib/provider-upjet-alibabacloud-*
Validates deployments** through automated testing
This automated publishing ensures all 19 family providers are consistently available to the Crossplane community with each release.
Validation and Quality Assurance
End-to-end testing proved the architecture's robustness:
make e2e-provider.vpc
make e2e-provider.oss
make e2e-provider.ecs
make e2e-monolith
make build-family
Readiness Checklist:
✅ All 19 family providers build successfully
✅ Monolithic provider can still be used for full local tests
✅ End-to-end tests validate real infrastructure provisioning
✅ Memory usage reduces from 166+ CRDs to service-specific subsets
✅ Template system ensures consistent package generation
Key Implementation Patterns
1. Resource Group Mapping Strategy
// Pattern: Service prefix determines group
"alicloud_vpc": ReplaceGroupWords("vpc", 0), // vpc group
"alicloud_oss_bucket": ReplaceGroupWords("oss", 1), // oss group
"alicloud_cs_kubernetes": ReplaceGroupWords("ack", 1
2. Controller Isolation Pattern
func Setup_vpc(mgr ctrl.Manager, o controller.Options) error
func Setup_oss(mgr ctrl.Manager, o controller.Options) error
func Setup_ack(mgr ctrl.Manager, o controller.Options) error
3. Template-Driven Generation
{{ if ne .Service "monolith" }}
{{ else }}
4. Build System Flexibility
make build-provider.vpc
make build-provider.vpc,oss,ecs
make build-family
make build
AI-Assisted Development Tips
Human + AI Partnership:
Claude: Pattern recognition, code generation, systematic implementation
Human: Architecture decisions, validation testing, business requirements
Approach
Proof of Concept Approach: Validate core concepts before full implementation
Systematic Implementation: Let AI handle repetitive tasks while humans focus on strategy
Continuous Validation: Test early and often to catch issues immediately
The future of infrastructure management is modular, efficient, and AI-accelerated. Family-scoped providers are just the beginning.
Ready to modernize your Crossplane providers? Start with a proof of concept using Claude or your favorite LLM as your development partner. The velocity gains will transform how you think about infrastructure code.