Architecting Large Software Projects: A Modular and Resilient Approach

Eskil Steenberg

Introduction & Core Philosophy [00:00:02]

This video details Eskil Steenberg's approach to structuring large software projects, aiming to break them into manageable, individual modules. It serves as a follow-up to a previous talk about enabling single-person software development. The goal is to apply this philosophy to complex projects like a video editor, a digital healthcare system, and a jet fighter, focusing on process over specific domain expertise.

• Optimization Goals for Big Projects: [00:02:28]

  • Dependability: Aim for software that is long-lasting and rarely breaks. [00:02:35]
  • Extendability: Design the system to easily incorporate new features, capabilities, and adaptations to change. [00:02:41]
  • Team Scalability: Structure the project so that individual sections can be worked on by very few people, ideally one, minimizing team meetings and interdependencies. [00:02:48]
  • Development Velocity: Maintain consistent development speed, preventing projects from slowing down as they grow. [00:03:14]

• Reducing Risk: [00:03:29]

  • Anticipate and mitigate potential failures, understanding the project's risk profile. [00:03:38]
  • Factors that can change and introduce risk: [00:03:48]
    • Changing platforms (APIs, terms of service, user trends).
    • Language implementations (future compiler support, backward compatibility breaks, e.g., Python 2 to 3).
    • Changing hardware.
    • Changing priorities or requirements.
    • Losing key personnel ("star programmers").
  • Impact of small failures: Many small, trivial fixes can accumulate, leading to context switching, slowdowns, and increased fatigue over time. [00:05:02]
  • Code Longevity: Write code that is "finished" and never needs fixing in the future, even if it means writing more explicit code now. [00:07:06]

• Modularizing Software (Black Boxes): [00:07:48]

  • Modules should be black boxes, exposing their functionality only through clearly defined and documented APIs or protocols. [00:08:00]
  • Users of a module should not need to understand its internal implementation. [00:08:19]
  • One module, one person: Each module should ideally be small enough for a single person to write and maintain, promoting independence. [00:09:00]
  • Applications are built by merging and using existing modules developed by different individuals. [00:09:22]
  • Design of APIs can be collaborative, but implementation should be single-owner. [00:09:57]
  • Modules can be exchanged or rewritten from scratch without affecting the rest of the system, provided their interfaces remain consistent. [00:10:27]
  • Senior developers should handle complex "senior" modules, while junior developers work on easier "junior" modules. [00:10:10]

Video Editor Example: Base Layers [00:11:05]

The talk begins by designing a basic video editor, demonstrating a layered software stack suitable for native applications with a UI.

• The Software Stack: [00:11:19]

  • UI Toolkits: Provides user interface elements.
  • Text: Handles text rendering.
  • Drawing Layer: Manages basic graphical primitives.
  • Platform Layer: Interfaces with the operating system and hardware.

    

• Platform Layer: [00:11:45]

  • Wrapping External Systems: Crucially, wrap any external or third-party platforms (e.g., SDL, Win32 API) with your own wrapper. [00:11:48]
    • This insulates your application from external changes (API updates, terms of service, development direction) and gives you full control.
    • Example: betray.h as a wrapper for windowing and input, initially built over SDL but later expanded to Win32 and other backends. [00:12:40]
  • Test Application/Demo: Write a minimal test application for the platform layer. [00:13:22]
    • This small app (e.g., opens window, reads mouse/keyboard, draws primitives) is invaluable for porting to new platforms, allowing quick verification without porting the entire codebase.
    • The test app helps confirm that all API functionalities are correctly implemented on a new platform by progressively uncommenting and testing features. [00:14:53]
  • Platform Considerations: [00:15:16]
    • Define API for inputs (mouse, keyboard, touch, multiple users/devices).
    • Manage screen resolutions, aspect ratios, and scaling.
    • Implement cut and paste functionality.
    • Handle file requesters and safe screen areas (e.g., avoiding notches on phones).
    • Support multi-user input (e.g., two mice/keyboards for distinct inputs, even if the OS doesn't natively support it yet). [00:16:00]
  • Consolidate Functionality: Abstract disparate inputs into generic types (e.g., all buttons are the same, all pointers are the same, all axes are the same). [00:17:00]

• Drawing Layer: [00:17:49]

  • Provides basic primitives for drawing UI: lines, surfaces, images, potentially shaders and geometry.
  • Example: Relinquish, built over OpenGL but designed for any backend like Vulkan. [00:18:01]

• Text Layer: [00:18:15]

  • All applications require text.
  • Iterative Implementation with Forward-Thinking API:
    • Start with the simplest possible text rendering (e.g., a basic bitmap font). [00:18:20]
    • Design the API (text_draw) from the outset to support future advanced features (e.g., fonts, letter spacing, UTF8, color, length limits, return values for text length). [00:19:55]
    • Even if features aren't implemented initially, having placeholders in the API allows other developers to write code as if those features exist. [00:20:40]
    • When advanced features (TrueType, anti-aliasing, kerning) are later implemented, existing code using the API won't need to change. [00:19:10]
  • Avoid "Good-Enough-for-Now" APIs: Never compromise on the API design, only on the initial implementation. The API should anticipate future needs (e.g., Arabic text, Korean text). [01:09:18]

• UI Toolkit: [00:21:58]

  • Building a custom UI toolkit for portability is often easier than figuring out how multiple platforms implement standard UI elements (e.g., sliders). [00:22:36]
  • Complex elements like text input are tricky, but most buttons and sliders are straightforward. [00:22:14]

• Helper Libraries: [00:23:44]

  • The platform, drawing, text, and UI layers are general-purpose helper libraries.
  • Maximize putting functionality into helper libraries for reuse across projects.
  • Examples: File parsing, physics engines, scripting, data storage, networking. [00:23:51]
  • Example: Testify networking library abstracts socket APIs, allowing internal improvements without affecting applications that use it. [00:24:05]

Video Editor Example: Core & Plugins [00:24:50]

• Defining the Core: [00:25:01]
  • Identify the fundamental "primitive" or data type the application works with.
  • For a video editor, the core primitive is "clips on a timeline with animated parameters." [00:25:11]
    • A video is essentially a rearrangement and transformation of input data (clips) over time.
    • Parameters allow for zooming, color correction, titling, etc.
  • "Everything is X" Principle: [00:25:52]
    • Unix: Everything is a file (allowing tools to interoperate by processing files). [00:26:02]
    • Houdini / Notch: Everything is a node (visual effects software built on node graphs with parameters). [00:26:17]
    • For the video editor, "everything is a clip" on a timeline. This generalization is key to robust software.
    • Contrast with less generalized systems like Unreal, which combine C++, blueprints, and scene graphs, leading to complexities in data generalization. [00:26:51]

• Primitives vs. Structure: [00:31:09]

  • Structure: How data is organized or flows (e.g., Unix command line with pipes).
  • Primitive: The fundamental type of data itself (e.g., Unix's text files).
  • Changing the primitive allows for new types of applications (e.g., video pipes instead of text pipes).
  • Example: Photoshop (layers, bitmaps) vs. Nuke (tree structure, bitmaps). Both work with bitmaps but organize them differently. [00:31:59]

• Building the Core Black Box: [00:33:01]

  • Create a core module that stores the generalized data (timeline of clips) and provides a clear API to access it. [00:33:04]
  • Guarantees: The core enforces data integrity and consistency (e.g., clips cannot have negative time, clips not longer than source assets). [00:33:31]
  • Undo/Redo: The core can manage changes and provide undo/redo functionality automatically for all operations performed via its API. [00:34:09]

• Plugin Architecture: [00:34:30]

  • To avoid a monolithic core that defines every possible clip type or effect, implement a plugin architecture. [00:34:52]
  • Plugins register their capabilities (e.g., "I can show MP4 video," "I have parameters X, Y, Z"). [00:35:02]
  • Modular Structure: The core interacts with plugins, and the UI toolkit builds the user interface based on the information provided by these plugins. [00:35:27]

    

  • Launcher UI: A separate application component that initializes the platform, core, and loads all available plugins. [00:35:48]
  • Decoupling Core and UI: The core can exist independently of the UI. This allows for diverse applications (e.g., a command-line tool for server-side video processing that uses the same core logic without the UI). [00:39:39]
  • Plugin Implementation: Plugins are often implemented as separate dynamic link libraries (DLLs), allowing developers to work in isolation and iterate quickly. [00:40:19]
    • Plugins expose their parameters and functionality through a descriptor system, enabling the UI to dynamically generate controls. [00:40:40]
    • This means most of the application's functionality resides in these independent plugins.

Healthcare System Example [00:44:58]

• Primitive: Instead of medical journals, the primitive should be "healthcare events" (e.g., birth, doctor visits, appointments). [00:45:21]

  • This allows for time-based access (e.g., "all events at this clinic today") and controlled information access (e.g., clerks only see appointment times, not full journals).

• Access API and Storage Black Box: [00:46:16]

  • Build a black box for storing healthcare events with an API that hides the underlying storage technology (SQL, NoSQL, cloud provider). [00:46:18]
  • This provides flexibility to change the storage backend at any time without impacting users of the API, avoiding vendor lock-in. [00:46:57]

• Gradual Migration with Glue Code: [00:47:39]

  • To transition from an old system, create "glue code" that translates data between your new black box API and the old system's API. [00:47:58]
  • This allows both systems to run concurrently, avoiding painful "big bang" switches and enabling a smooth, gradual migration. [00:48:18]

    

• Multi-Access API and Layered Apps: [00:48:39]

  • Develop multiple glue layers (e.g., C++, Python bindings) on top of the core API to serve different types of developers and applications. [00:48:48]
  • These APIs can then support various end-user applications:
    • Specific UIs for different healthcare roles.
    • Internet-connected features.
    • Websites for patient access.
  • All these applications interact through the robust, central black box, regardless of their specific front-end or external connections. [00:49:32]

    

Jet Fighter Example [00:50:50]

• Primitive: "State of the World": [00:51:37]

  • Unlike a healthcare system that focuses on history, a jet fighter needs real-time, current information about its environment and internal status. [00:51:39]
  • This "state of the world" primitive includes data like:
    • Confidence (how much to trust the data).
    • Accuracy (precision of measurements).
    • Source (where the data came from, e.g., radar, other planes).
    • Format (different representations of the same data, e.g., position relative to plane vs. global coordinates).
    • Capabilities (e.g., remaining missiles, sensors).
    • Damage status affecting accuracy.

• Authoritative Core & Subscriber Model: [00:53:34]

  • Establish an "authoritative core" that aggregates and maintains the single, most reliable "state of the world." [00:53:41]
  • Other components (radar, engine, displays) act as "subscribers" that request and receive only the specific information they need from the core. [00:54:07]
  • This allows for efficient data distribution to diverse hardware, from powerful computers to microcontrollers.
  • Supports multiple "wire types" or communication protocols to handle various hardware connectors and bandwidth requirements. [00:55:09]

    

• Crucial Role of Tooling: [00:56:17]

  • For complex projects like jet fighters, extensive tooling is essential for development and testing. These tools typically don't ship with the final product. [00:57:55]
  • Key Tools:
    • Recorder: Captures all data flowing through the system. [00:56:51]
    • Playback: Replays recorded data, allowing for simulated testing of components (e.g., testing a missile with real flight data). [00:57:05]
    • Python API: Enables scripting for automated testing and simulation of various scenarios (e.g., sensor failure). [00:57:20]
    • Logger: Provides visibility into data flow and component behavior for debugging. [00:57:34]
    • Visualizer: Graphically displays system state. [00:57:51]
    • Simulator: Emulates entire parts of the system for comprehensive testing. [00:57:53]
  • Tooling allows separate teams (e.g., missile developers, cockpit designers) to test their components against a consistent system, even if other parts are incomplete. [00:58:03]

• Open Source Core (Conceptual): [00:59:19]

  • The fundamental core that defines the "state of the world" and its API could theoretically be open-sourced.
  • This would allow various contractors to develop and integrate components (sensors, weapons, UIs) without revealing proprietary secrets, as only the communication format is public.

• Redundant Cores for Reliability: [01:00:37]

  • For mission-critical systems like jet fighters, a single central core is a single point of failure.
  • The system can be designed with multiple, redundant cores that communicate and verify each other's state. [01:00:47]
  • The key is that all external systems (subscribers) still interact via the same consistent API, regardless of the internal redundancy. [01:01:06]

    

  • This allows initial simple core implementation for rapid development, followed by complex, robust re-implementation of the internal core without affecting users. [01:01:09]

• Generalizability: [01:02:18]

  • The "state of the world" primitive and subscriber/publisher model can be applied to other military vehicles (tanks, transport jets) or even broader interconnected systems. [01:02:26]
  • This promotes knowledge transfer and reusability across different projects.

Key Architectural Principles (Wrap Up) [01:03:43]

• Format Design is Core: [01:03:58]

  • All fundamental elements in software development are formats: APIs, files, network protocols, and even programming languages.
  • Designing these formats is crucial but often overlooked in education.

• Semantic vs. Structure: [01:04:59]

  • Semantics: The meaning of the data (e.g., 3 meters – the actual length).
  • Structure: How the data is organized or transmitted (e.g., JSON – a way to store data, but not what the data means).
  • Often, a simple structure with rich semantics is desirable, allowing generic loaders/parsers (like a JSON loader for any data) while relying on higher-level code to interpret the meaning.

• Formats Need to be Implementable: [01:06:36]

  • If an API or format is too complex, implementations will be buggy or incomplete, leading to incompatibilities.
  • Smaller, simpler formats are easier to implement correctly, leading to higher quality and better interoperability.
  • Strive to pack maximum power into the simplest possible interface. [01:07:33]

• Formats on Both Ends: [01:07:41]

  • When designing a format, remember that it must be implemented by both the sender and the receiver.
  • Supporting multiple alternatives within a format (e.g., both polygons and NURBS for 3D geometry) means both ends must implement all alternatives, doubling work even for those who prefer only one.
  • It's better to make clear choices and enforce a single, minimal format, even if it makes some parties "less happy," as it reduces overall implementation burden. [01:09:18]

• Manage Constraints: [01:09:38]

  • Clearly define the requirements and guarantees of your format (e.g., all data is in metric, or the core guarantees conversion). This helps implementers rely on predictable behavior.

• Think About Indirections: [01:10:17]

  • Consider the number of abstraction layers. Too few limits flexibility; too many introduces unnecessary complexity and makes traversing the system difficult.
  • Example: Should a video editor support one timeline or many? One video stream or multiple? Single display or fifty?

• Implementation Freedom: [01:11:25]

  • Design APIs that explicitly separate the "what" (what the API does) from the "how" (how it's implemented internally).
  • Avoid exposing internal implementation details (e.g., allowing SQL queries directly to your storage backend) as this locks you into specific technologies and hinders future changes.
  • The API should protect the internal implementation from external dependencies.

• "Plug In or Get Plugged In": [01:12:45]

  • When designing a module, aim for it to be a standalone piece that can be plugged into a larger system, rather than a module that needs to plug itself into other systems.
  • This maintains control over your core environment and allows you to define the rules of integration.
  • Standalone modules are highly reusable. If a plug-in architecture is necessary, ensure your system dictates the interface, not the other way around.