Real-World Java Concurrency & Parallelism Challenges: Solutions with Virtual Threads & Rate Limiting

Summary:

This video addresses complex concurrency and parallelism challenges encountered in real-world Java applications, using a high-scale order processing system as a case study.

The speaker introduces fundamental concepts: sequential, parallel, and concurrent processing, synchronous vs. asynchronous operations, and CPU-bound vs. IO-bound tasks.

Diagram illustrating sequential, parallel, and concurrent processing. [ 00:12:40 ]
The core problem involves processing millions of orders and products, generating 270,000 requests per second, which leads to performance issues and Out-Of-Memory (OOM) errors.

Table summarizing the large-scale order processing problem. [ 00:19:20 ]
Initial attempts with sequential processing were too slow, while parallel streams led to issues like nested streams and lost thread context.

Diagram illustrating the issue of nested parallel streams overloading shared CPU cores. [ 00:38:39 ]
Moving to IO-bound solutions with Executors and CompletableFuture exposed OOM errors due to shared thread pools, necessitating pool separation and rate limiting for dependencies.

Graph illustrating a memory spike leading to an Out-of-Memory (OOM) error. [ 00:59:49 ]
Virtual Threads, introduced as a lighter alternative, also presented challenges like "pinning" and further OOMs when replacing traditional threads without careful resource management.

Diagram illustrating how virtual threads are mapped to platform threads. [ 01:26:58 ]
The most effective solution combined virtual threads with semaphores to control backpressure, achieving optimal performance.

Performance metrics for the simplified Virtual Threads solution, showing reduced peak RPS. [ 01:37:38 ]
Key takeaways emphasize the importance of experimentation, measurement, protecting resources (especially external dependencies), and prioritizing simplicity in concurrency solutions.

Key takeaways for implementing concurrency solutions. [ 01:38:18 ]

Introduction and Speaker's Background [0:00:00]

The live session begins with welcoming remarks and attendee shout-outs from various cities in Brazil, and even Texas, highlighting a diverse audience. [0:00:00]
The hosts, Alberto and Rafael Ponte, introduce themselves and their work, which focuses on helping software developers evolve by delivering better software. [0:02:50]
Hugo Marques, the main speaker, introduces himself as a software engineer with 20 years of experience, having worked at Amazon, Twitter, and currently Netflix. [0:05:55]
- He emphasizes his role as a product development-focused engineer, not a JVM or performance specialist, ensuring a practical application-level perspective on concurrency. [0:09:08]
This talk is designed for developers writing concurrent code in real applications, engineers facing scaling issues (especially with IO), and anyone curious about concurrency challenges, regardless of their preferred language. [0:09:53]
The talk is explicitly not a benchmark, a deep dive into JVM/low-level tuning, or a full reference of the Java concurrency API. [0:10:56]

Concurrency and Parallelism Fundamentals [0:11:42]

The speaker clarifies the differences between sequential, parallel, and concurrent execution: [0:11:55]
- Sequential: Tasks execute one after another on a single CPU. [0:12:01]
- Parallel: Multiple tasks execute simultaneously on multiple CPUs. [0:12:20]
- Concurrent: Tasks appear to run simultaneously on a single CPU by rapidly switching between them, giving the impression of parallel execution. [0:12:36]
  
  Diagram illustrating sequential, parallel, and concurrent processing. [ 00:12:40 ]
Synchronous vs. Asynchronous: [0:13:26]
- Synchronous: Process A calls Process B and waits for B to complete before continuing its own execution. [0:13:37]
- Asynchronous: Process A calls Process B and continues its own execution immediately, only waiting for B's result when absolutely necessary. This allows for better machine utilization, especially with multiple cores. [0:14:04]
  
  Diagram comparing synchronous and asynchronous process flows. [ 00:14:10 ]
CPU-bound vs. IO-bound tasks: [0:14:48]
- CPU-bound (CPU-intensive): Tasks that primarily utilize the CPU for calculations, compression, or tight loops (e.g., in-memory object transformations). [0:14:56]
- IO-bound: Tasks that involve waiting for external operations like network calls, file I/O, database interactions, or gRPC calls. These are generally much slower than CPU-bound tasks. [0:15:32]
  
  List differentiating CPU-bound and IO-bound tasks. [ 00:14:50 ]
- Most modern web applications are IO-bound due to frequent external calls. [0:16:10]
Java Concurrency Toolbox: [0:16:34]
- Thread: Old-school, rarely used directly but forms the foundation. [0:16:41]
- ExecutorService: Used for thread pool management, reusing threads to avoid expensive creation/destruction. [0:17:01]
- CompletableFuture: Java's way of handling non-blocking composition, similar to promises in JavaScript, enabling asynchronous programming. [0:17:28]
- Parallel Streams: Introduced in Java 8, useful for easy concurrency in CPU-bound work. [0:17:54]
- Virtual Threads: New, flexible, and efficient for IO-heavy flows, as they are very cheap to create and destroy. [0:18:21]
  
  Table outlining Java's concurrency tools and their applications. [ 00:16:53 ]

The Real-World Problem: Orders Processing at Scale [0:19:15]

The problem involves processing a large volume of orders every 30 minutes. [0:19:15]
- Scheduler: 1 run every 30 minutes. [0:19:22]
- Regions: ~10 regions per run. [0:19:31]
- Orders: 10,000 orders per region, totaling 100,000 orders. [0:19:39]
- Products: 5,000 products per order, totaling 500 million products. [0:19:50]
- gRPC Calls: 1 call per 100 products, resulting in ~5 million calls. [0:20:06]
- Total: ~270,000 requests per second. [0:20:22]
- The system must process all data in memory and write to a file only after all processing is complete, with no event-driven processing. [0:21:44]
  
  Table summarizing the large-scale order processing problem. [ 00:19:20 ]
  
  High-level architecture diagram of the order processing flow. [ 00:21:10 ]

Initial Solutions and Their Pitfalls [0:21:55]

Sequential Solution [0:21:55]

The first and simplest approach was sequential processing: region by region, order by order, product by product, making calls and writing results. [0:22:01]

Code snippet demonstrating sequential order processing. [ 00:22:20 ]
- Pros: Works, easy to understand, memory-safe (less object creation), and safe for dependencies (no parallel calls stressing them). [0:22:36]
- Cons: Extremely slow and underutilizes resources (e.g., only one CPU core used out of many available). [0:23:14]

CPU-Bound Solution: Parallel Streams [0:24:01]

To address slowness, parallel streams (introduced in Java 8) were considered for CPU-bound tasks, specifically for reading and transforming raw data into region objects. [0:24:01]
- Introduction to Parallel Streams: Achieved by parallelStream() or parallel() on a stream. They use the ForkJoinPool.commonPool() by default (size = #processors - 1). Custom pools can be created using ForkJoinPool for specific tasks. [0:24:21]
- Problem #1: Nested Parallel Stream: An attempt to use parallel streams in multiple nested loops led to performance degradation. [0:37:59]
  - Nesting parallel streams overloads CPU cores with excessive context switching and overhead, as the single commonPool is shared globally. This results in no actual speedup for nested tasks. [0:38:38]

Diagram illustrating the issue of nested parallel streams overloading shared CPU cores. [ 00:38:39 ]

* Solution #1: Parallelize where it matters most: Apply parallel streams only at the highest level where there are large, independent data sets (e.g., raw data processing), avoiding nesting. [0:39:14]

Code snippet showing the optimized parallel stream application at the highest data level. [ 00:40:20 ]

* Problem #2: A Flatmap Surprise: Using parallelStream() within a flatMap operation surprisingly results in sequential execution due to flatMap's internal implementation transforming the stream to sequential. [0:40:29]

Code snippet demonstrating a parallel stream unexpectedly running sequentially within a `flatMap`. [ 00:40:29 ]

* Solution #2: Reverse the order: Move the parallelStream() call to an outer level of iteration to ensure parallelism is applied where intended. [0:41:43] * Problem #3: Parallel Stream & Thread Context: Parallel streams execute tasks in different threads from the commonPool, meaning they lose access to the original thread's context (e.g., user transaction context). This makes them unsuitable for many server-side applications that rely on thread-local contexts. [0:42:42]

Code snippet highlighting how parallel streams lose the original thread's context. [ 00:42:49 ]

IO-Bound Solution: Executors & CompletableFuture [0:53:11]

For IO-bound operations (like numerous gRPC calls), the focus shifted to Executors and CompletableFuture for asynchronous processing. [0:53:11]
- Executor Types: FixedThreadPool, CachedThreadPool (caution: can lead to excessive threads and performance degradation due to context switching), SingleThreadExecutor (useful for testing). [0:54:05]
- CompletableFuture allows non-blocking composition, enabling other tasks to run while waiting for an IO operation. [0:55:51]
- Spring Integration: Spring offers @Async annotation, either with a specified executor name (Option 2) or using Spring's default executor (Option 3), simplifying asynchronous calls. [0:56:15]
- The proposed solution for processing orders and products involved chaining CompletableFutures for each step, with each call executing in a dedicated executor. [0:58:22]
  
  Code snippet showing the initial setup where a single executor handles multiple tasks. [ 01:00:24 ]
- Problem #4: Our first OOM (Out Of Memory): The initial implementation led to Out-Of-Memory errors despite increased machine memory. [0:59:45]
  
  Graph illustrating a memory spike leading to an Out-of-Memory (OOM) error. [ 00:59:49 ]
  - The root cause was sharing a single executor for both request processing and large response processing (from gRPC calls). Large response objects would accumulate in the shared queue faster than they could be processed, leading to memory exhaustion. [1:00:17]

Diagram explaining how a shared thread pool leads to OOM due to accumulating large response objects. [ 01:00:49 ]

* Solution #4: Separate the pools: Use different ExecutorService instances for different responsibilities (e.g., one for initiating requests, another for processing responses), ensuring that large, incoming data doesn't block the main processing queue. thenApply (faster, no context switch) and thenApplyAsync (allows specifying executor) were discussed for processing CompletableFuture results. [1:02:01]

Diagram showing how separating thread pools mitigates Out-of-Memory issues. [ 01:02:09 ]

Table comparing `thenApply` and `thenApplyAsync` performance, flexibility, and context preservation. [ 01:05:19 ]

Managing Load: Self-Throttling [1:12:11]

After fixing OOM, the system started "DDoS-ing" downstream dependencies with high request per second (RPS) rates (e.g., 30,000 RPS). [1:12:11]
- This required self-throttling to control the outbound request rate. [1:12:40]
Semaphores vs. Rate Limiter: [1:14:06]
- Semaphore: Controls the number of concurrent calls (in-flight tasks) by blocking threads once a fixed limit is reached. Useful when tasks take too long and build up. Protects CPU, memory, and threads. [1:13:27]
- Rate Limiter: Controls the frequency of calls (tasks started per second). Useful for protecting external dependencies that have a defined throughput limit. [1:14:13]
  
  Table comparing the characteristics and use cases of Semaphores and Rate Limiters. [ 01:14:58 ]
Problem #6: OOM Strikes Back (Backpressure): Even with a rate limiter, OOM recurred. The rate limiter only controlled outbound requests, but the internal production of requests was still faster than the processing capacity, leading to an accumulation of incoming data in memory. [1:17:49]

Diagram showing tasks accumulating in memory despite a rate limiter, causing another OOM. [ 01:18:28 ]
Solution #6: Add a Semaphore to control backpressure: A semaphore was added before the request generation to limit the number of simultaneous orders being processed, effectively controlling the internal build-up and preventing OOM. This created two "levers" to control: the semaphore for internal concurrency and the rate limiter for external throughput. [1:18:40]

Diagram illustrating the addition of a semaphore to control backpressure and prevent OOM. [ 01:18:38 ]
The architecture was becoming increasingly complex, with multiple layers of concurrency control. [1:20:19]

The Latest Innovation: Virtual Threads [1:25:55]

Understanding Virtual Threads [1:25:55]

Main Idea: Virtual threads are lightweight, cheap to create, and allow mapping many virtual threads onto a few platform threads (OS threads). When a virtual thread is blocked by an IO operation, the JVM can unmount it from its platform thread and allow another virtual thread to run on that same platform thread, improving resource utilization for IO-bound tasks. [1:26:56]

Diagram illustrating how virtual threads are mapped to platform threads. [ 01:26:58 ]
Enabling virtual threads in Spring is straightforward via configuration (spring.threads.virtual.enabled=true) or by creating a VirtualThreadExecutor bean. [1:28:02]

Problem #7: The Pinning Problem [1:29:05]

The Problem: In earlier Java versions (before JDK 20/21), a "pinning" problem occurred where virtual threads inside synchronized blocks or native calls would "pin" their underlying platform thread, preventing it from being released for other virtual threads. This negated the benefits of virtual threads. [1:29:05]

Diagram illustrating the 'pinning' problem in virtual threads due to synchronized blocks. [ 01:29:08 ]
Solution #7: Update to JDK 20/21 (or newer): This issue has been largely resolved in modern JDK versions (JDK 24+ for LTS). No code changes are needed; simply upgrading the JDK fixes it, except for specific native calls. [1:30:36]

Problem #8: Yet Another Out Of Memory [1:31:18]

Even with virtual threads and the pinning problem solved, OOM still occurred. [1:31:18]
The issue arose from creating nested asynchronous blocks (@Async) at both the order and gRPC call levels. While virtual threads are lightweight, simultaneously creating many thousands of intermediate objects (orders, products, gRPC calls, responses) consumed excessive memory, leading to OOM. [1:31:59]

Diagram showing nested `@Async` tasks leading to another Out-of-Memory error with virtual threads. [ 01:31:28 ]
Lesson: Simply replacing platform threads with virtual threads is not a magical solution. Resource management is still critical. [1:32:11]

Solution #8: Back to the Semaphore [1:32:23]

The solution reverted to using a semaphore to limit the number of concurrent virtual threads processing orders. This re-introduced backpressure and prevented memory exhaustion. [1:32:23]
By setting the semaphore to limit ~50 concurrent order tasks, the system maintained a consistent rate of 3,000 RPC requests per second without OOM. [1:35:39]

Diagram showing a semaphore limiting concurrent virtual threads for backpressure. [ 01:32:28 ]

Performance Results and Key Takeaways [1:33:28]

Bounded-Executor vs. Virtual Threads: [1:33:28]
- Both solutions achieved similar job duration times (around 12 minutes). [1:33:39]
- However, the Bounded-Executor solution showed higher RPS spikes (up to 34,000 RPS), while Virtual Threads achieved the same job duration with lower RPS peaks (around 16,000 RPS), indicating more efficient resource utilization and less stress on dependencies. [1:34:00]
  
  Comparison of job duration and Requests Per Second (RPS) for Bounded-Executor vs. Virtual Threads. [ 01:34:28 ]
A Simpler Virtual Threads Solution (Final Optimization): [1:35:01]
- By further simplifying the virtual thread implementation to only use one layer of @Async (at the order level) and relying on the single semaphore for backpressure, the system became even more efficient and simpler to understand. The internal RPC calls were made sequentially within each virtual order thread. [1:35:39]
- This reduced execution time slightly (from ~12 min to ~10-11 min) and further lowered peak RPS to dependencies (13,000 RPS peak, stabilizing at 49 concurrent calls). [1:37:20]
  
  Performance metrics for the simplified Virtual Threads solution, showing reduced peak RPS. [ 01:37:38 ]
The speaker declared this "perfection" because it achieved optimal performance with the simplest possible code. [1:37:51]
Key Takeaways: [1:38:09]
1. Experiment and Measure Your Solutions: Don't assume a solution will work; always test and validate with data. [1:38:09]
2. Protect Your Resources and Work Backwards from There: Identify bottlenecks (often external dependencies like databases or payment platforms) and implement protective measures (e.g., rate limiters) first, then design your system to respect those limits. [1:38:41]
3. Concurrency is Hard, So Keeping it Simple is Better: Even if a simpler solution isn't the most "optimized" on paper, its ease of understanding, maintenance, and debugging often outweighs marginal performance gains from overly complex concurrency models. [1:39:26]
  
  Key takeaways for implementing concurrency solutions. [ 01:38:18 ]