IndianTechnoEra: Chapter 9: Background Work & Async Processing in .NET

Introduction

In real-world systems, many API operations should NOT be done inside the request–response cycle. Examples:

• Sending emails/SMS
• Generating PDFs/Reports
• Processing images
• Clearing cache
• Performing long business workflows
• Calling external APIs with retries
• Importing/exporting large data
• Queue-based microservice communication

ASP.NET Core provides:

• IHostedService
• BackgroundService
• Queued background workers
• Channels for producer-consumer patterns

We will build production-grade background processing step-by-step.

9.1 Why Background Processing Is Necessary

Imagine an API endpoint:

POST /orders/create

Inside the controller you:

• Insert order in DB
• Send payment request
• Send email
• Update analytics
• Notify warehouse
• Log to external system

If you do all of this inside the API call, the user waits for:

• DB latency
• Network latency
• External services
• File I/O

❌ Bad API experience

❌ Slow response times

❌ Timeouts

❌ Users retry → duplicate requests

❌ Scaling issues

9.2 The Solution: Background Workers

You offload heavy/non-urgent work to a background queue, allowing the API to return fast:

API (fast)

// Put job in background queue
await _backgroundQueue.EnqueueAsync(job);
// Return response immediately
return Accepted();

Background Worker (async)

while (!token.IsCancellationRequested)
{
    var job = await _backgroundQueue.DequeueAsync(token);
    await Process(job, token);
}

This architecture is:

• Fast
• Scalable
• Resilient
• Production-friendly

9.3 Understanding IHostedService & BackgroundService

Option 1: IHostedService (Low-level interface)

public class MyWorker : IHostedService
{
    public Task StartAsync(CancellationToken token)
    {
        // start worker
    }
    
    public Task StopAsync(CancellationToken token)
    {
        // graceful shutdown
    }
}

Option 2: BackgroundService (Recommended)

.NET provides BackgroundService, a simpler base class:

public class MyWorker : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken token)
    {
        while (!token.IsCancellationRequested)
        {
            await DoWorkAsync(token);
            await Task.Delay(1000, token);
        }
    }
}

9.4 Example 1: Scheduled Background Cleanup Worker

public class CleanupWorker : BackgroundService
{
    private readonly ILogger<CleanupWorker> _logger;
    
    public CleanupWorker(ILogger<CleanupWorker> logger)
    {
        _logger = logger;
    }
    
    protected override async Task ExecuteAsync(CancellationToken token)
    {
        while (!token.IsCancellationRequested)
        {
            _logger.LogInformation("Running cleanup...");
            await CleanupOldRecordsAsync();
            await Task.Delay(TimeSpan.FromMinutes(10), token);
        }
    }
}

Register it:

builder.Services.AddHostedService<CleanupWorker>();

9.5 Example 2: Email Sender Background Worker

Step 1: Background Queue Interface

public interface IBackgroundTaskQueue
{
    ValueTask EnqueueAsync(Func<CancellationToken, Task> workItem);
    ValueTask<Func<CancellationToken, Task>> DequeueAsync(CancellationToken token);
}

Step 2: Queue Implementation (using Channels)

public class BackgroundTaskQueue : IBackgroundTaskQueue
{
    private readonly Channel<Func<CancellationToken, Task>> _queue;
    
    public BackgroundTaskQueue(int capacity = 100)
    {
        _queue = Channel.CreateBounded<Func<CancellationToken, Task>>(capacity);
    }
    
    public async ValueTask EnqueueAsync(Func<CancellationToken, Task> workItem)
    {
        await _queue.Writer.WriteAsync(workItem);
    }
    
    public async ValueTask<Func<CancellationToken, Task>> DequeueAsync(CancellationToken token)
    {
        return await _queue.Reader.ReadAsync(token);
    }
}

Step 3: Worker That Runs Queue Jobs

public class QueuedWorker : BackgroundService
{
    private readonly IBackgroundTaskQueue _queue;
    private readonly ILogger<QueuedWorker> _logger;
    
    public QueuedWorker(IBackgroundTaskQueue queue, ILogger<QueuedWorker> logger)
    {
        _queue = queue;
        _logger = logger;
    }
    
    protected override async Task ExecuteAsync(CancellationToken token)
    {
        while (!token.IsCancellationRequested)
        {
            var workItem = await _queue.DequeueAsync(token);
            try
            {
                await workItem(token);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Background task failed.");
            }
        }
    }
}

Step 4: Register Queue + Worker

builder.Services.AddSingleton<IBackgroundTaskQueue, BackgroundTaskQueue>();
builder.Services.AddHostedService<QueuedWorker>();

Step 5: API Controller Enqueues Email Sending

[HttpPost("register")]
public async Task<IActionResult> Register(UserDto dto)
{
    // Save user in DB...
    await _queue.EnqueueAsync(async token =>
    {
        await _emailService.SendWelcomeEmail(dto.Email, token);
    });
    
    return Accepted("User created. Email will be sent shortly.");
}

Benefits:

• API returns instantly
• Email is processed in background
• Highly scalable
• Fault-tolerant

9.6 Use Case: PDF/Excel Report Generation

Why Background?

Reports can take:

• 5–30 seconds to generate
• Large memory
• Multiple DB calls

Instead of blocking API:

API

[HttpPost("report/generate")]
public async Task<IActionResult> GenerateReport()
{
    var reportId = Guid.NewGuid().ToString();
    await _queue.EnqueueAsync(token => 
        _reportService.GenerateReportAsync(reportId, token));
    
    return Accepted(new { reportId });
}

Client calls a status endpoint:

[HttpGet("report/status/{id}")]
public IActionResult GetStatus(string id)
{
    return Ok(_reportService.GetStatus(id));
}

Background Worker generates the report asynchronously.

This is exactly how:

• Banking systems
• Insurance systems
• Government portals
• E-commerce admin reports

are built.

9.7 Use Case: Retrying Failed Jobs with Exponential Backoff

You can add retry logic inside the worker:

private async Task ProcessWithRetry(Func<Task> job)
{
    int retries = 0;
    while (true)
    {
        try
        {
            await job();
            return;
        }
        catch
        {
            retries++;
            if (retries > 5) throw;
            await Task.Delay(TimeSpan.FromSeconds(Math.Pow(2, retries)));
        }
    }
}

9.8 Communication Between APIs Using Queues or Events

Async workers can consume:

• RabbitMQ messages
• Kafka topics
• Azure Service Bus
• AWS SQS

Example worker:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    await foreach (var message in _messageBus.ReceiveAsync(stoppingToken))
    {
        await ProcessMessageAsync(message, stoppingToken);
    }
}

9.9 Graceful Shutdown (Very Important)

ASP.NET Core gives worker a stoppingToken.

Your worker must respect it:

protected override async Task ExecuteAsync(CancellationToken token)
{
    while (!token.IsCancellationRequested)
    {
        // do work
    }
}

⚠️ NEVER ignore cancellation.

Otherwise:

• App hangs on shutdown
• Jobs get corrupted
• Data inconsistency happens

9.10 Fire-and-Forget in API: Why It's Dangerous

❌ WRONG

Task.Run(() => ProcessAsync()); // fire-and-forget ❌

Problems:

• No exception handling
• No retry
• No cancellation
• No logging
• Can be killed during app recycle
• Hard to debug
• Causes data loss

✅ Right Way

Use background workers or message queues.

9.11 Real-Life Architecture: High-Load Enterprise Pattern

API Layer

• Inserts job into queue
• Returns 202 (Accepted) immediately

Queue

• Stores tasks (Channel, Redis, RabbitMQ, SQS, etc.)

Worker

• Pulls tasks
• Executes async logic
• Retries on failure
• Logs all attempts

Dashboard

• Shows job history, status, failures

This is how:

• UPI processors
• E-commerce order systems
• SMS/email providers
• Financial workflows

are structured.

9.12 Observability for Background Tasks

You MUST log:

• Job ID
• Enqueue time
• Dequeue time
• Duration
• Retries
• Failure reason

Example:

_logger.LogInformation(
    "Processing job {JobId} started at {Time}",
    jobId, DateTime.UtcNow);

9.13 Performance Tips

• ✔ Use Channel for in-memory queues
• ✔ Use background workers instead of Task.Run
• ✔ Use async all the way
• ✔ Do NOT block threads
• ✔ Use cancellation tokens correctly
• ✔ Scale workers horizontally
• ✔ Use persistent queues for critical jobs
• ✔ Use batching for DB inserts

9.14 Best-Practices Checklist

API

• Return immediately
• Use queues for slow or external processes

Worker

• Always async
• Use cancellation tokens
• Retry failures
• Log everything
• Gracefully shut down

System

• Monitor worker health
• Use centralized queue (RabbitMQ, SQS, Azure Service Bus)
• Isolate heavy processing from main API

9.15 Background Service Lifecycle Management

The Complete Lifecycle

public class OrderProcessingWorker : BackgroundService
{
    private readonly ILogger<OrderProcessingWorker> _logger;
    
    public OrderProcessingWorker(ILogger<OrderProcessingWorker> logger)
    {
        _logger = logger;
    }
    
    public override Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Order processing worker starting...");
        // Initialize resources here (DB connections, HTTP clients)
        return base.StartAsync(cancellationToken);
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Order processing worker running...");
        
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessOrdersAsync(stoppingToken);
                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
            }
            catch (OperationCanceledException)
            {
                // Graceful shutdown
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error in order processing worker");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
        }
    }
    
    public override Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Order processing worker stopping...");
        // Clean up resources here
        return base.StopAsync(cancellationToken);
    }
}

Worker Dependencies and Scoped Services

// Problem: BackgroundService is singleton, but needs scoped services
public class EmailWorker : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    
    public EmailWorker(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await using (var scope = _serviceProvider.CreateAsyncScope())
            {
                var emailService = scope.ServiceProvider
                    .GetRequiredService<IEmailService>();
                var dbContext = scope.ServiceProvider
                    .GetRequiredService<AppDbContext>();
                
                await ProcessEmailsAsync(emailService, dbContext, stoppingToken);
            }
            
            await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
        }
    }
}

9.16 Concurrent Background Services and Coordination

Running Multiple Services with Different Frequencies

public class SchedulerWorker : BackgroundService
{
    private readonly IEnumerable<IScheduledTask> _tasks;
    
    public SchedulerWorker(IEnumerable<IScheduledTask> tasks)
    {
        _tasks = tasks;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var taskExecutions = _tasks.Select(task => 
            RunTaskWithIntervalAsync(task, stoppingToken));
        
        await Task.WhenAll(taskExecutions);
    }
    
    private async Task RunTaskWithIntervalAsync(
        IScheduledTask task, 
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await task.ExecuteAsync(stoppingToken);
                await Task.Delay(task.Interval, stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Task {TaskName} failed", task.Name);
                await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
            }
        }
    }
}

Service Coordination with SemaphoreSlim

public class CoordinatedWorker : BackgroundService
{
    private readonly SemaphoreSlim _semaphore = new(1, 1);
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await _semaphore.WaitAsync(stoppingToken);
            
            try
            {
                // Ensure only one instance runs at a time
                await ProcessCriticalSectionAsync(stoppingToken);
            }
            finally
            {
                _semaphore.Release();
            }
            
            await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
        }
    }
    
    private async Task ProcessCriticalSectionAsync(CancellationToken token)
    {
        // Critical work that should not run concurrently
        await _dbContext.SaveChangesAsync(token);
    }
}

9.17 Persistent Queues for Reliable Background Processing

Redis Queue Implementation

public class RedisBackgroundQueue : IBackgroundTaskQueue
{
    private readonly IConnectionMultiplexer _redis;
    private readonly ILogger<RedisBackgroundQueue> _logger;
    private const string QueueName = "background-jobs";
    
    public async ValueTask EnqueueAsync(string jobType, object jobData)
    {
        var job = new BackgroundJob
        {
            Id = Guid.NewGuid().ToString(),
            Type = jobType,
            Data = JsonSerializer.Serialize(jobData),
            CreatedAt = DateTime.UtcNow,
            Status = JobStatus.Pending
        };
        
        var db = _redis.GetDatabase();
        await db.ListRightPushAsync(QueueName, JsonSerializer.Serialize(job));
        
        _logger.LogInformation("Enqueued job {JobId} of type {JobType}", 
            job.Id, jobType);
    }
    
    public async ValueTask<BackgroundJob> DequeueAsync(CancellationToken token)
    {
        var db = _redis.GetDatabase();
        
        while (!token.IsCancellationRequested)
        {
            var result = await db.ListLeftPopAsync(QueueName);
            if (!result.HasValue)
            {
                await Task.Delay(1000, token);
                continue;
            }
            
            var job = JsonSerializer.Deserialize<BackgroundJob>(result.ToString());
            job.Status = JobStatus.Processing;
            job.StartedAt = DateTime.UtcNow;
            
            return job;
        }
        
        throw new OperationCanceledException();
    }
}

Database-Backed Queue (SQL Server)

public class DatabaseBackgroundQueue : IBackgroundTaskQueue
{
    private readonly AppDbContext _dbContext;
    
    public async ValueTask EnqueueAsync(BackgroundJob job)
    {
        await _dbContext.BackgroundJobs.AddAsync(job);
        await _dbContext.SaveChangesAsync();
    }
    
    public async ValueTask<BackgroundJob> DequeueAsync(CancellationToken token)
    {
        // Use pessimistic locking to prevent concurrent processing
        await using var transaction = await _dbContext.Database
            .BeginTransactionAsync(token);
        
        try
        {
            var job = await _dbContext.BackgroundJobs
                .FromSqlRaw(@"
                    SELECT TOP 1 * FROM BackgroundJobs WITH (UPDLOCK, ROWLOCK, READPAST)
                    WHERE Status = 'Pending' AND Attempts < 3
                    ORDER BY Priority DESC, CreatedAt ASC")
                .FirstOrDefaultAsync(token);
            
            if (job != null)
            {
                job.Status = JobStatus.Processing;
                job.StartedAt = DateTime.UtcNow;
                job.Attempts++;
                await _dbContext.SaveChangesAsync(token);
                await transaction.CommitAsync(token);
                return job;
            }
            
            await transaction.RollbackAsync(token);
            return null;
        }
        catch
        {
            await transaction.RollbackAsync(token);
            throw;
        }
    }
}

9.18 Monitoring and Health Checks for Background Workers

Implementing Health Checks

public class BackgroundWorkerHealthCheck : IHealthCheck
{
    private readonly IBackgroundTaskQueue _queue;
    private readonly ILogger<BackgroundWorkerHealthCheck> _logger;
    private DateTime _lastProcessedTime = DateTime.UtcNow;
    
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            var timeSinceLastProcess = DateTime.UtcNow - _lastProcessedTime;
            
            if (timeSinceLastProcess > TimeSpan.FromMinutes(5))
            {
                return HealthCheckResult.Degraded(
                    $"No jobs processed in {timeSinceLastProcess.TotalMinutes} minutes");
            }
            
            // Check queue depth
            var queueSize = await _queue.GetSizeAsync();
            if (queueSize > 1000)
            {
                return HealthCheckResult.Degraded(
                    $"Queue backlog: {queueSize} jobs");
            }
            
            return HealthCheckResult.Healthy("Background worker is healthy");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("Background worker check failed", ex);
        }
    }
}

// Registration
builder.Services.AddHealthChecks()
    .AddCheck<BackgroundWorkerHealthCheck>("background-worker");

Metrics Collection for Background Workers

public class BackgroundWorkerMetrics
{
    private readonly Counter<int> _processedJobs;
    private readonly Histogram<double> _processingTime;
    private readonly Counter<int> _failedJobs;
    
    public BackgroundWorkerMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("BackgroundWorker");
        
        _processedJobs = meter.CreateCounter<int>(
            "background.jobs.processed",
            description: "Number of background jobs processed");
        
        _processingTime = meter.CreateHistogram<double>(
            "background.job.processing.time",
            unit: "ms",
            description: "Time to process a background job");
        
        _failedJobs = meter.CreateCounter<int>(
            "background.jobs.failed",
            description: "Number of failed background jobs");
    }
    
    public async Task<T> TrackJobAsync<T>(
        Func<Task<T>> job,
        string jobType)
    {
        var stopwatch = Stopwatch.StartNew();
        
        try
        {
            var result = await job();
            _processedJobs.Add(1, new("job.type", jobType));
            _processingTime.Record(stopwatch.ElapsedMilliseconds);
            return result;
        }
        catch (Exception)
        {
            _failedJobs.Add(1, new("job.type", jobType));
            throw;
        }
    }
}

9.19 Error Handling and Dead Letter Queues

Implementing Dead Letter Queue

public class ResilientBackgroundWorker : BackgroundService
{
    private readonly IBackgroundTaskQueue _queue;
    private readonly IDeadLetterQueue _deadLetterQueue;
    private readonly ILogger<ResilientBackgroundWorker> _logger;
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                var job = await _queue.DequeueAsync(stoppingToken);
                
                try
                {
                    await ProcessJobWithRetryAsync(job, stoppingToken);
                    await _queue.CompleteJobAsync(job.Id);
                }
                catch (Exception ex) when (IsTransientFailure(ex))
                {
                    await _queue.RetryJobAsync(job.Id);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Job {JobId} failed permanently", job.Id);
                    await _deadLetterQueue.MoveToDeadLetterAsync(job, ex);
                }
            }
            catch (OperationCanceledException)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error in worker loop");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
        }
    }
    
    private async Task ProcessJobWithRetryAsync(
        BackgroundJob job, 
        CancellationToken token,
        int maxRetries = 3)
    {
        for (int attempt = 1; attempt <= maxRetries; attempt++)
        {
            try
            {
                await ExecuteJobAsync(job, token);
                return;
            }
            catch (Exception ex) when (attempt < maxRetries)
            {
                _logger.LogWarning(ex, 
                    "Job {JobId} failed on attempt {Attempt}", job.Id, attempt);
                await Task.Delay(ExponentialBackoff(attempt), token);
            }
        }
    }
}

9.20 Configurable Background Workers

Configuration-Based Worker Settings

public class WorkerSettings
{
    public int BatchSize { get; set; } = 100;
    public TimeSpan Interval { get; set; } = TimeSpan.FromMinutes(5);
    public int MaxRetries { get; set; } = 3;
    public bool Enabled { get; set; } = true;
}

public class ConfigurableWorker : BackgroundService
{
    private readonly WorkerSettings _settings;
    private readonly ILogger<ConfigurableWorker> _logger;
    
    public ConfigurableWorker(IOptions<WorkerSettings> settings,
        ILogger<ConfigurableWorker> logger)
    {
        _settings = settings.Value;
        _logger = logger;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (!_settings.Enabled)
        {
            _logger.LogInformation("Worker is disabled");
            return;
        }
        
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessBatchAsync(_settings.BatchSize, stoppingToken);
                await Task.Delay(_settings.Interval, stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error in worker execution");
                await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
            }
        }
    }
}

// appsettings.json
{
  "WorkerSettings": {
    "BatchSize": 500,
    "Interval": "00:01:00",
    "MaxRetries": 5,
    "Enabled": true
  }
}

9.21 Testing Background Workers

Unit Testing Background Services

[Fact]
public async Task Worker_ProcessesJobs_WhenQueueHasItems()
{
    // Arrange
    var mockQueue = new Mock<IBackgroundTaskQueue>();
    var mockService = new Mock<IEmailService>();
    
    var job = (CancellationToken token) => 
        mockService.Object.SendEmailAsync("test@example.com", token);
    
    mockQueue.SetupSequence(q => q.DequeueAsync(It.IsAny<CancellationToken>()))
             .ReturnsAsync(job)
             .ThrowsAsync(new OperationCanceledException());
    
    var worker = new QueuedWorker(mockQueue.Object, Mock.Of<ILogger<QueuedWorker>>());
    
    // Act
    var cancellationToken = new CancellationTokenSource(TimeSpan.FromSeconds(1)).Token;
    await worker.StartAsync(cancellationToken);
    
    // Wait for worker to process
    await Task.Delay(100);
    
    // Assert
    mockService.Verify(s => s.SendEmailAsync("test@example.com", It.IsAny<CancellationToken>()), 
        Times.Once);
}

[Fact]
public async Task Worker_StopsGracefully_WhenCancelled()
{
    // Arrange
    var cts = new CancellationTokenSource();
    var worker = new CleanupWorker(Mock.Of<ILogger<CleanupWorker>>());
    
    // Act
    var startTask = worker.StartAsync(cts.Token);
    await Task.Delay(100);
    cts.Cancel();
    await worker.StopAsync(CancellationToken.None);
    
    // Assert
    await Assert.ThrowsAsync<TaskCanceledException>(() => startTask);
}

Integration Testing with Host

[Fact]
public async Task Worker_RunsPeriodically_InIntegrationTest()
{
    // Arrange
    var hostBuilder = new HostBuilder()
        .ConfigureServices(services =>
        {
            services.AddHostedService<TestWorker>();
            services.AddSingleton<ITestService, TestService>();
        });
    
    using var host = await hostBuilder.StartAsync();
    
    var testService = host.Services.GetRequiredService<ITestService>();
    var worker = host.Services.GetServices<IHostedService>()
        .OfType<TestWorker>()
        .First();
    
    // Act
    await Task.Delay(TimeSpan.FromSeconds(2));
    
    // Assert
    Assert.True(testService.WasCalled, "Worker should have called service");
    
    // Cleanup
    await host.StopAsync(TimeSpan.FromSeconds(5));
}

9.22 Scaling Background Workers

Multiple Worker Instances with Leader Election

public class ScalableWorker : BackgroundService
{
    private readonly IDistributedLock _distributedLock;
    private readonly string _workerId;
    private readonly TimeSpan _lockTimeout = TimeSpan.FromMinutes(5);
    
    public ScalableWorker(IDistributedLock distributedLock)
    {
        _distributedLock = distributedLock;
        _workerId = Guid.NewGuid().ToString();
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            // Try to acquire lock (only one instance runs at a time)
            var lockAcquired = await _distributedLock.AcquireAsync(
                "worker-lock", 
                _workerId, 
                _lockTimeout);
            
            if (lockAcquired)
            {
                try
                {
                    _logger.LogInformation("Worker {WorkerId} acquired lock", _workerId);
                    await ProcessWorkAsync(stoppingToken);
                }
                finally
                {
                    await _distributedLock.ReleaseAsync("worker-lock", _workerId);
                }
            }
            else
            {
                _logger.LogDebug("Worker {WorkerId} waiting for lock", _workerId);
                await Task.Delay(TimeSpan.FromSeconds(10), stoppingToken);
            }
        }
    }
}

Horizontal Scaling with Work Partitioning

public class PartitionedWorker : BackgroundService
{
    private readonly int _workerIndex;
    private readonly int _totalWorkers;
    
    public PartitionedWorker(IConfiguration configuration)
    {
        _workerIndex = configuration.GetValue<int>("WorkerIndex");
        _totalWorkers = configuration.GetValue<int>("TotalWorkers");
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            // Process only items assigned to this worker
            var items = await GetAssignedItemsAsync();
            
            foreach (var item in items)
            {
                if (ShouldProcessItem(item))
                {
                    await ProcessItemAsync(item, stoppingToken);
                }
            }
            
            await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken);
        }
    }
    
    private bool ShouldProcessItem(WorkItem item)
    {
        // Simple hash-based partitioning
        var partition = item.Id.GetHashCode() % _totalWorkers;
        return partition == _workerIndex;
    }
}

Chapter 9 Summary

You learned how to build production-level async background processing using:

• BackgroundService
• IHostedService
• Async queues (Channels)
• Retry logic
• Graceful cancellation
• Report/email/background workflows
• Task offloading for high performance
• Real enterprise architecture patterns

Background workers turn your API into a high-performance distributed system capable of handling massive load without slowing down users.

✅ Chapter 9 is now complete.

Next up is an extremely important chapter for microservices & async architecture:

👉 Chapter 10: Common Async Pitfalls & Anti-Patterns

This includes:

• Sync-over-async issues
• Deadlocks
• Thread starvation
• Using Task.Run incorrectly
• Blocking I/O in async apps
• Async void disasters
• Misusing ValueTask
• Forgetting ConfigureAwait

Would you like me to proceed with Chapter 10 (Deep Dive) now?