Progressive Infrastructure Delivery with Kargo and Argo CD

CNCF [Cloud Native Computing Foundation]

Summary:

This lightning talk by Engin Diri from Pulumi introduces progressive infrastructure delivery using Kargo and Argo CD, addressing the pain points of promoting infrastructure changes across multiple environments in GitOps.

• GitOps requires Kubernetes operators like Crossplane or Pulumi for infrastructure deployment.
• Deploying infrastructure via GitOps with Argo CD typically only syncs to a single environment (e.g., dev).
• Promoting infrastructure changes beyond dev often leads to complex, obscure, and difficult-to-maintain CI/CD pipelines.
• Using Git branches for environment promotion is problematic due to configuration drift, merge conflicts, and scalability issues.
• A better approach is to model environments using separate folders in a Git repository, leveraging tools like Helm and Kustomize.
• Kargo, an open-source GitOps-native project, provides multi-stage promotion capabilities for various artifacts (container images, Kubernetes manifests, Helm charts).
• Kargo enforces a strict separation of CI (building artifacts) and CD (everything else), offering flexible, manual, and automatic promotion pipelines with native Argo CD integration.
• It also enables progressive delivery by listening to monitoring systems and can roll back unsuccessful deployments.

Introduction to Progressive Infrastructure Delivery and Pain Points [0:00:00]

The speaker, Engin Diri from Pulumi, introduces the topic of progressive infrastructure delivery using Kargo and Argo CD. He highlights the common pain points experienced in large-scale GitOps environments, particularly when promoting artifacts and infrastructure changes between different environments.

• The "Pain Scale" of GitOps
  • Promoting artifacts across different environments in a large enterprise GitOps setup is identified as a source of significant operational pain, similar to a high number on a pain scale.

GitOps and Operators for Infrastructure [0:01:04]

The presentation delves into the foundational concepts of GitOps and the role of Kubernetes operators in deploying infrastructure.

• The Necessity of Operators for GitOps [0:01:04]
  • GitOps, especially in a Kubernetes context, relies heavily on operators to manage and deploy infrastructure.
  • What are Kubernetes Operators? [0:01:19]
    • Operators extend the Kubernetes API through custom resource definitions (CRDs) and custom controllers.
    • A user defines a custom resource, which the operator watches.
    • Upon detecting changes, the operator takes predefined actions to achieve the desired state in the Kubernetes cluster.
• Tools for Infrastructure as Code (IaC) with GitOps [0:01:44]
  • Crossplane and Pulumi are used as examples of tools that provide operators to define and deploy infrastructure as code on Kubernetes clusters.
• Basic GitOps Deployment with Argo CD [0:02:00]
  • A typical workflow involves defining infrastructure in a Git repository.
  • Argo CD picks up these definitions and deploys them to a target Kubernetes cluster.
  • This initial deployment usually targets a single environment, such as a "dev" environment.

Challenges of Environment Promotion in GitOps [0:02:30]

A core problem discussed is how to effectively promote infrastructure changes beyond the initial development environment to staging, UAT, and production environments.

• Limitations of Single-Environment Syncs [0:02:30]
  • Simply syncing to a "dev" environment is insufficient for real-world scenarios, as most organizations have multiple environments.
• Complex and Obscure Promotion Pipelines [0:02:59]
  • Organizations often resort to highly complex, "mystical, obscure" pipelines involving various CI/CD tools (e.g., Azure DevOps, GitLab) to manage promotions.
  • These custom scripts become difficult to understand and maintain over time.
• The Need for Better GitOps Promotion Patterns [0:03:30]
  • The central question is identifying a better, GitOps-native way to promote infrastructure changes across environments.

Promotion Patterns: Why Not Branches? [0:03:39]

The talk addresses a common but problematic approach to environment promotion: using Git branches.

• Definition of Promotion Patterns [0:03:41]
  • Promotion patterns describe different ways to implement and manage environments (stages like Test, Staging, Production) with GitOps.
• Disadvantages of Using Branches for Environments [0:04:08]
  • Outdated and Problematic: Using Git branches for different environments leads to configuration drift and significant maintenance challenges.
  • Merge Conflicts: Promotion via Git merges can cause unwanted changes and frequent merge conflicts, making the process painful.
  • Scalability Issues: Branch-per-environment models are difficult to scale when managing many environments.
  • Tool Recommendations: Popular Kubernetes tools like Helm and Kustomize advocate for using file folders (overlay mechanisms) rather than branches for environment modeling.

Promotion Patterns: Using Folders for Environments [0:05:08]

The recommended best practice for modeling GitOps environments is to use a folder-based approach.

• Benefits of Folder-Based Environments [0:05:28]
  • Cleaner Management: Separating configuration and environment-specific files into distinct folders simplifies management.
  • Simple Promotion: Environment promotions can be handled with straightforward file-copy operations.
  • Gradual Transitions: Ensures smooth transitions between environments without causing disruptions.
  • Trunk-Based Development: Encourages trunk-based development and using Git tags for versioning, improving consistency.
• Leveraging Helm and Kustomize [0:05:45]
  • This approach integrates well with tools like Kustomize (using base folders and overlays) and Helm (using different value files to override properties).
  • It's also possible to mix and match both approaches.

Introducing Kargo for Smoother Promotions [0:06:01]

Kargo is presented as a solution designed to streamline multi-stage environment promotions in a GitOps-native way.

• What is Kargo? [0:06:14]
  • An open-source project from Akuity, launched approximately a year prior to the talk, focused on progressive delivery.
• Key Features of Kargo [0:06:23]
  • Multi-stage Promotion: Enables promotion of artifacts through various stages.
  • GitOps Native: Fully adheres to GitOps principles, eliminating the need for complex CI push pipelines for CD logic.
  • Flexible Promotion Pipelines: Allows creation of flexible pipelines for artifacts.
  • Supported Artifacts: Handles container images, Kubernetes manifests, and Helm charts.
  • Manual and Automatic Promotion: Supports both manual and automated promotion workflows.
  • Native Integration: Seamlessly integrates with existing ecosystems, with Argo CD as the primary first-class citizen implementation, and future support for Flux.
  • Progressive Delivery Capabilities: Can listen to monitoring system data or deployment statuses within clusters to determine promotion success or trigger automatic rollbacks to previous versions if issues arise.
• How Kargo Works [0:07:23]
  • CI builds an image and pushes it to an image repository.
  • Platform/DevOps engineers define infrastructure as code (IaC) in a Git repository.
  • Kargo observes both the image repository and the Git repository.
  • It detects changes and promotes artifacts to the defined environment.
  • For progressive delivery, Kargo integrates with monitoring systems to validate deployments and initiate rollbacks if necessary.

Demo of Kargo with Pulumi and Crossplane [0:08:23]

The speaker demonstrates Kargo's capabilities by deploying infrastructure defined using Pulumi and Crossplane.

• Demo Setup [0:08:23]
  • An umbrella Helm chart is used to deploy infrastructure, containing both Crossplane and Pulumi definitions for a basic static application on Digital Ocean.
  • Different stages (Test, UAT, Production) are defined, each referencing a specific version of the infrastructure chart.
  • The Helm umbrella chart defines dependencies on other charts, including the infrastructure chart.

    

  • Specific configurations like the deployment region are defined in the Helm values.

    

• Kargo UI Overview [0:09:08]
  • The Kargo UI visualizes the deployed infrastructure versions across different stages.
  • It shows, for example, that Test and UAT might be on version 1.8, while Production (New York City) is on 1.6.

    

• Promoting a New Infrastructure Version [0:10:08]
  • The speaker demonstrates promoting a newer version [1] of the infrastructure definition from a pre-deployed stage to the Production (New York City) stage using the Kargo UI.
  • The UI enforces dependencies between stages, allowing promotion only when previous stages are aligned.

    

  • (During the live demo, there were minor delays in the update reflecting in the Argo CD UI for the Digital Ocean application, but Kargo confirmed the promotion.)

Key Takeaways [0:13:07]

The presentation concludes with three main takeaways regarding Kargo and progressive infrastructure delivery.

• Stricter CI/CD Separation: Kargo enforces a clear separation where CI builds artifacts, and CD (handled by Kargo and Argo CD) manages everything else, including promotion and deployment.
• GitOps Native: Kargo is fundamentally GitOps native, aligning with best practices.
• Elegant Promotion Solution: Kargo provides an elegant solution to the previously complex and painful process of promoting infrastructure changes across multiple environments.