GitOps with Fully Rendered Configurations: Achieving Faster, Safer Operations by Separating Data from Code

CNCF [Cloud Native Computing Foundation]

Introduction and Presenters' Background [00:00:00]

• Brian Grant [00:00:00]
  • Original architect of Kubernetes.
  • Creator of Kustomize.
  • Former member of the inaugural CNCF Technical Oversight Committee (TOC).
• Alexis Richardson [00:00:34]
  • First chair of the CNCF TOC.
  • Founder and former CEO of Weave Works, where GitOps and Flux were created.
  • Previous roles at VMware and Pivotal (Spring Boot, Spring Cloud).

Challenges with Traditional GitOps and Templated Configurations [00:01:13]

• Configuration Sprawl and Heterogeneity [00:01:41]
  • Applications run across multiple environments (dev, prod) and clusters.
  • Configuration files (Helm charts, YAML) are spread across many repositories, organized differently, and use varied conventions.
  • Different teams (operations, security) struggle to understand and manage this diverse configuration landscape.
  • Example: Three sample applications (AI chatbot, voting app, online store) demonstrate diverse YAML file structures and organization.

    

• Urgent Updates and "Breaking Glass" [00:03:58]
  • Urgent issues like container Out-of-Memory (OOMs), new feature bugs, or security vulnerabilities require rapid fixes.
  • Platform teams often "break glass" by directly changing live production state using kubectl, which creates configuration drift.

    

  • Operations staff find GitOps hard to use due to the need to learn development tools, manage complex YAML, and navigate various files to find the correct configuration.
• Complexity of Finding and Changing Values (Blast Radius) [00:05:05]
  • Finding Configuration: Locating the correct configuration and input values in a large, heterogeneous Git repository can be challenging for non-application teams.
    • Helm annotations provide release information but don't directly point to value files.
    • GitOps tools like Flux add labels and traces to help link deployed resources back to their source and value files.

      

  • Changing Values: Modifying the right values without unintended side effects (blast radius) is difficult.
    • Values can be layered (overlays, overrides) or transformed by multiple chained tools (e.g., Helm + Kustomize).
    • Sometimes values are hard-coded or unspecified, requiring template changes that can affect all environments and regions, increasing blast radius risk.

      

    • The parameterization approach, taken to an extreme (e.g., "one chart" projects), makes Helm charts overly complex to understand, with thousands of lines of templates and default values.

      

      

• Mass Changes [00:14:04]
  • Making widespread changes (e.g., due to a security policy update) across multiple applications with entirely different templates is error-prone and painful, especially with dozens or hundreds of configurations.

DRY vs. WET Configuration Approaches [00:14:57]

• DRY (Don't Repeat Yourself) [00:14:57]
  • Configuration generators like Helm, Kustomize, CDK8s, Jsonnet, and CUElang factor out common parts of configuration into code.
  • Intended to facilitate the generation of multiple similar configurations (e.g., for different environments or regions).
  • The complexity of these formats makes understanding and changing individual configurations difficult.
• WET (Write Every Time) [00:15:32]
  • Refers to plain YAML or fully rendered manifests.
  • Advocates for explicitly writing out the full configuration for each specific scenario.
  • Rendered Manifest Pattern: Rendering configurations in advance before deployment helps in reviewing and applying linting/policy tools (like kube-conform or Kyverno).
    • This increases confidence in changes but doesn't speed up the modification process.
    • Can even slow down the process due to CI worker bottlenecks if rendering for many environments.

      

  • The speakers advocate for always using WET, fully rendered, explicit configuration, viewing configuration as data, not code.

    

ConfigHub Demo: GitOps with Fully Rendered Configurations [00:17:04]

• Overview of ConfigHub Interface [00:17:38]
  • ConfigHub stores configuration as data objects in a database, not as generation code in Git.
  • It displays configuration units for three applications across dev and prod clusters.
  • Each unit is standard Kubernetes YAML, without variables, conditionals, or loops.

    

  • There's a one-to-one correspondence between ConfigHub configuration and resources in the clusters, ensuring a constrained blast radius.
• Making Targeted Changes [00:19:48]
  • Users can trace a cluster issue back to its ConfigHub source using annotations, similar to Flux's trace command.
  • Changes can be made programmatically using "functions" (like simple tools) via the UI or CLI.
  • Example: Fixing a container OOM by increasing memory resources. The function validates inputs and makes precise changes.

    

  • Example: Disabling a feature using an environment variable.

    

  • Example: Setting security context values (e.g., runAsNonRoot: true).

    

• Performing Mass Changes and Queries [00:24:51]
  • ConfigHub allows querying for specific configuration states, e.g., all deployments without runAsNonRoot set.
  • Users can select multiple units (e.g., 42 units) and apply a function to all of them simultaneously.
  • This enables mass transformations across diverse configurations, which is challenging with DRY formats.

    

• Validation and Policy Enforcement [00:26:24]
  • ConfigHub integrates validation similar to dynamic admission control in Kubernetes.
  • kube-conform and policy engines (e.g., Cell policies currently, with potential for OPA/Kyverno) can be run on every change to ensure validity before deployment.

Key Takeaways: Configuration as Data [00:27:13]

• Configuration is Data: The core principle is to treat configuration as explicit data rather than programmatic code.
  • This involves separating the data part from the programmatic part, which sits as a layer above, like traditional database management.
  • This separation allows for proper management of configuration data and enables advanced operations.

    

• Operational Benefits:
  • Speed up critical changes.
  • Reduce configuration sprawl across systems, teams, and formats.
  • Enable "Operational Happiness" by making changes easier and safer.
• Compatibility: Existing tools like Kubernetes, Flux, Argo, and Helm can still be used. Helm, for instance, should be viewed as a packaging tool, not an operational delivery tool, with explicit values being the focus.