Chapter 9: Background Work & Async Processing in .NET

Introduction

In real-world systems, many API operations should NOT be done inside the request–response cycle. Examples:

Sending emails/SMS
Generating PDFs/Reports
Processing images
Clearing cache
Performing long business workflows
Calling external APIs with retries
Importing/exporting large data
Queue-based microservice communication

ASP.NET Core provides:

IHostedService
BackgroundService
Queued background workers
Channels for producer-consumer patterns

We will build production-grade background processing step-by-step.

9.1 Why Background Processing Is Necessary

Imagine an API endpoint:

POST /orders/create

Inside the controller you:

Insert order in DB
Send payment request
Send email
Update analytics
Notify warehouse
Log to external system

If you do all of this inside the API call, the user waits for:

DB latency
Network latency
External services
File I/O

❌ Bad API experience

❌ Slow response times

❌ Timeouts

❌ Users retry → duplicate requests

❌ Scaling issues

9.2 The Solution: Background Workers

You offload heavy/non-urgent work to a background queue, allowing the API to return fast:

API (fast)

// Put job in background queue
await _backgroundQueue.EnqueueAsync(job);
// Return response immediately
return Accepted();

Background Worker (async)

while (!token.IsCancellationRequested)
{
    var job = await _backgroundQueue.DequeueAsync(token);
    await Process(job, token);
}

This architecture is:

Fast
Scalable
Resilient
Production-friendly

9.3 Understanding IHostedService & BackgroundService

Option 1: IHostedService (Low-level interface)

public class MyWorker : IHostedService
{
    public Task StartAsync(CancellationToken token)
    {
        // start worker
    }
    
    public Task StopAsync(CancellationToken token)
    {
        // graceful shutdown
    }
}

Option 2: BackgroundService (Recommended)

.NET provides BackgroundService, a simpler base class:

public class MyWorker : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken token)
    {
        while (!token.IsCancellationRequested)
        {
            await DoWorkAsync(token);
            await Task.Delay(1000, token);
        }
    }
}

9.4 Example 1: Scheduled Background Cleanup Worker

public class CleanupWorker : BackgroundService
{
    private readonly ILogger<CleanupWorker> _logger;
    
    public CleanupWorker(ILogger<CleanupWorker> logger)
    {
        _logger = logger;
    }
    
    protected override async Task ExecuteAsync(CancellationToken token)
    {
        while (!token.IsCancellationRequested)
        {
            _logger.LogInformation("Running cleanup...");
            await CleanupOldRecordsAsync();
            await Task.Delay(TimeSpan.FromMinutes(10), token);
        }
    }
}

Register it:

builder.Services.AddHostedService<CleanupWorker>();

9.5 Example 2: Email Sender Background Worker

Step 1: Background Queue Interface

public interface IBackgroundTaskQueue
{
    ValueTask EnqueueAsync(Func<CancellationToken, Task> workItem);
    ValueTask<Func<CancellationToken, Task>> DequeueAsync(CancellationToken token);
}

Step 2: Queue Implementation (using Channels)

public class BackgroundTaskQueue : IBackgroundTaskQueue
{
    private readonly Channel<Func<CancellationToken, Task>> _queue;
    
    public BackgroundTaskQueue(int capacity = 100)
    {
        _queue = Channel.CreateBounded<Func<CancellationToken, Task>>(capacity);
    }
    
    public async ValueTask EnqueueAsync(Func<CancellationToken, Task> workItem)
    {
        await _queue.Writer.WriteAsync(workItem);
    }
    
    public async ValueTask<Func<CancellationToken, Task>> DequeueAsync(CancellationToken token)
    {
        return await _queue.Reader.ReadAsync(token);
    }
}

Step 3: Worker That Runs Queue Jobs

public class QueuedWorker : BackgroundService
{
    private readonly IBackgroundTaskQueue _queue;
    private readonly ILogger<QueuedWorker> _logger;
    
    public QueuedWorker(IBackgroundTaskQueue queue, ILogger<QueuedWorker> logger)
    {
        _queue = queue;
        _logger = logger;
    }
    
    protected override async Task ExecuteAsync(CancellationToken token)
    {
        while (!token.IsCancellationRequested)
        {
            var workItem = await _queue.DequeueAsync(token);
            try
            {
                await workItem(token);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Background task failed.");
            }
        }
    }
}

Step 4: Register Queue + Worker

builder.Services.AddSingleton<IBackgroundTaskQueue, BackgroundTaskQueue>();
builder.Services.AddHostedService<QueuedWorker>();

Step 5: API Controller Enqueues Email Sending

[HttpPost("register")]
public async Task<IActionResult> Register(UserDto dto)
{
    // Save user in DB...
    await _queue.EnqueueAsync(async token =>
    {
        await _emailService.SendWelcomeEmail(dto.Email, token);
    });
    
    return Accepted("User created. Email will be sent shortly.");
}

Benefits:

API returns instantly
Email is processed in background
Highly scalable
Fault-tolerant

9.6 Use Case: PDF/Excel Report Generation

Why Background?

Reports can take:

5–30 seconds to generate
Large memory
Multiple DB calls

Instead of blocking API:

API

[HttpPost("report/generate")]
public async Task<IActionResult> GenerateReport()
{
    var reportId = Guid.NewGuid().ToString();
    await _queue.EnqueueAsync(token => 
        _reportService.GenerateReportAsync(reportId, token));
    
    return Accepted(new { reportId });
}

Client calls a status endpoint:

[HttpGet("report/status/{id}")]
public IActionResult GetStatus(string id)
{
    return Ok(_reportService.GetStatus(id));
}

Background Worker generates the report asynchronously.

This is exactly how:

Banking systems
Insurance systems
Government portals
E-commerce admin reports

are built.

9.7 Use Case: Retrying Failed Jobs with Exponential Backoff

You can add retry logic inside the worker:

private async Task ProcessWithRetry(Func<Task> job)
{
    int retries = 0;
    while (true)
    {
        try
        {
            await job();
            return;
        }
        catch
        {
            retries++;
            if (retries > 5) throw;
            await Task.Delay(TimeSpan.FromSeconds(Math.Pow(2, retries)));
        }
    }
}

9.8 Communication Between APIs Using Queues or Events

Async workers can consume:

RabbitMQ messages
Kafka topics
Azure Service Bus
AWS SQS

Example worker:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    await foreach (var message in _messageBus.ReceiveAsync(stoppingToken))
    {
        await ProcessMessageAsync(message, stoppingToken);
    }
}

9.9 Graceful Shutdown (Very Important)

ASP.NET Core gives worker a stoppingToken.

Your worker must respect it:

protected override async Task ExecuteAsync(CancellationToken token)
{
    while (!token.IsCancellationRequested)
    {
        // do work
    }
}

⚠️ NEVER ignore cancellation.

Otherwise:

App hangs on shutdown
Jobs get corrupted
Data inconsistency happens

9.10 Fire-and-Forget in API: Why It's Dangerous

❌ WRONG

Task.Run(() => ProcessAsync()); // fire-and-forget ❌

Problems:

No exception handling
No retry
No cancellation
No logging
Can be killed during app recycle
Hard to debug
Causes data loss

✅ Right Way

Use background workers or message queues.

9.11 Real-Life Architecture: High-Load Enterprise Pattern

API Layer

Inserts job into queue
Returns 202 (Accepted) immediately

Queue

Stores tasks (Channel, Redis, RabbitMQ, SQS, etc.)

Worker

Pulls tasks
Executes async logic
Retries on failure
Logs all attempts

Dashboard

Shows job history, status, failures

This is how:

UPI processors
E-commerce order systems
SMS/email providers
Financial workflows

are structured.

9.12 Observability for Background Tasks

You MUST log:

Job ID
Enqueue time
Dequeue time
Duration
Retries
Failure reason

Example:

_logger.LogInformation(
    "Processing job {JobId} started at {Time}",
    jobId, DateTime.UtcNow);

9.13 Performance Tips

✔ Use Channel for in-memory queues
✔ Use background workers instead of Task.Run
✔ Use async all the way
✔ Do NOT block threads
✔ Use cancellation tokens correctly
✔ Scale workers horizontally
✔ Use persistent queues for critical jobs
✔ Use batching for DB inserts

9.14 Best-Practices Checklist

API

Return immediately
Use queues for slow or external processes

Worker

Always async
Use cancellation tokens
Retry failures
Log everything
Gracefully shut down

System

Monitor worker health
Use centralized queue (RabbitMQ, SQS, Azure Service Bus)
Isolate heavy processing from main API

9.15 Background Service Lifecycle Management

The Complete Lifecycle

public class OrderProcessingWorker : BackgroundService
{
    private readonly ILogger<OrderProcessingWorker> _logger;
    
    public OrderProcessingWorker(ILogger<OrderProcessingWorker> logger)
    {
        _logger = logger;
    }
    
    public override Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Order processing worker starting...");
        // Initialize resources here (DB connections, HTTP clients)
        return base.StartAsync(cancellationToken);
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Order processing worker running...");
        
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessOrdersAsync(stoppingToken);
                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
            }
            catch (OperationCanceledException)
            {
                // Graceful shutdown
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error in order processing worker");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
        }
    }
    
    public override Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Order processing worker stopping...");
        // Clean up resources here
        return base.StopAsync(cancellationToken);
    }
}

Worker Dependencies and Scoped Services

// Problem: BackgroundService is singleton, but needs scoped services
public class EmailWorker : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    
    public EmailWorker(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await using (var scope = _serviceProvider.CreateAsyncScope())
            {
                var emailService = scope.ServiceProvider
                    .GetRequiredService<IEmailService>();
                var dbContext = scope.ServiceProvider
                    .GetRequiredService<AppDbContext>();
                
                await ProcessEmailsAsync(emailService, dbContext, stoppingToken);
            }
            
            await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
        }
    }
}

9.16 Concurrent Background Services and Coordination

Running Multiple Services with Different Frequencies

public class SchedulerWorker : BackgroundService
{
    private readonly IEnumerable<IScheduledTask> _tasks;
    
    public SchedulerWorker(IEnumerable<IScheduledTask> tasks)
    {
        _tasks = tasks;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var taskExecutions = _tasks.Select(task => 
            RunTaskWithIntervalAsync(task, stoppingToken));
        
        await Task.WhenAll(taskExecutions);
    }
    
    private async Task RunTaskWithIntervalAsync(
        IScheduledTask task, 
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await task.ExecuteAsync(stoppingToken);
                await Task.Delay(task.Interval, stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Task {TaskName} failed", task.Name);
                await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
            }
        }
    }
}

Service Coordination with SemaphoreSlim

public class CoordinatedWorker : BackgroundService
{
    private readonly SemaphoreSlim _semaphore = new(1, 1);
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await _semaphore.WaitAsync(stoppingToken);
            
            try
            {
                // Ensure only one instance runs at a time
                await ProcessCriticalSectionAsync(stoppingToken);
            }
            finally
            {
                _semaphore.Release();
            }
            
            await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
        }
    }
    
    private async Task ProcessCriticalSectionAsync(CancellationToken token)
    {
        // Critical work that should not run concurrently
        await _dbContext.SaveChangesAsync(token);
    }
}

9.17 Persistent Queues for Reliable Background Processing

Redis Queue Implementation

public class RedisBackgroundQueue : IBackgroundTaskQueue
{
    private readonly IConnectionMultiplexer _redis;
    private readonly ILogger<RedisBackgroundQueue> _logger;
    private const string QueueName = "background-jobs";
    
    public async ValueTask EnqueueAsync(string jobType, object jobData)
    {
        var job = new BackgroundJob
        {
            Id = Guid.NewGuid().ToString(),
            Type = jobType,
            Data = JsonSerializer.Serialize(jobData),
            CreatedAt = DateTime.UtcNow,
            Status = JobStatus.Pending
        };
        
        var db = _redis.GetDatabase();
        await db.ListRightPushAsync(QueueName, JsonSerializer.Serialize(job));
        
        _logger.LogInformation("Enqueued job {JobId} of type {JobType}", 
            job.Id, jobType);
    }
    
    public async ValueTask<BackgroundJob> DequeueAsync(CancellationToken token)
    {
        var db = _redis.GetDatabase();
        
        while (!token.IsCancellationRequested)
        {
            var result = await db.ListLeftPopAsync(QueueName);
            if (!result.HasValue)
            {
                await Task.Delay(1000, token);
                continue;
            }
            
            var job = JsonSerializer.Deserialize<BackgroundJob>(result.ToString());
            job.Status = JobStatus.Processing;
            job.StartedAt = DateTime.UtcNow;
            
            return job;
        }
        
        throw new OperationCanceledException();
    }
}

Database-Backed Queue (SQL Server)

public class DatabaseBackgroundQueue : IBackgroundTaskQueue
{
    private readonly AppDbContext _dbContext;
    
    public async ValueTask EnqueueAsync(BackgroundJob job)
    {
        await _dbContext.BackgroundJobs.AddAsync(job);
        await _dbContext.SaveChangesAsync();
    }
    
    public async ValueTask<BackgroundJob> DequeueAsync(CancellationToken token)
    {
        // Use pessimistic locking to prevent concurrent processing
        await using var transaction = await _dbContext.Database
            .BeginTransactionAsync(token);
        
        try
        {
            var job = await _dbContext.BackgroundJobs
                .FromSqlRaw(@"
                    SELECT TOP 1 * FROM BackgroundJobs WITH (UPDLOCK, ROWLOCK, READPAST)
                    WHERE Status = 'Pending' AND Attempts < 3
                    ORDER BY Priority DESC, CreatedAt ASC")
                .FirstOrDefaultAsync(token);
            
            if (job != null)
            {
                job.Status = JobStatus.Processing;
                job.StartedAt = DateTime.UtcNow;
                job.Attempts++;
                await _dbContext.SaveChangesAsync(token);
                await transaction.CommitAsync(token);
                return job;
            }
            
            await transaction.RollbackAsync(token);
            return null;
        }
        catch
        {
            await transaction.RollbackAsync(token);
            throw;
        }
    }
}

9.18 Monitoring and Health Checks for Background Workers

Implementing Health Checks

public class BackgroundWorkerHealthCheck : IHealthCheck
{
    private readonly IBackgroundTaskQueue _queue;
    private readonly ILogger<BackgroundWorkerHealthCheck> _logger;
    private DateTime _lastProcessedTime = DateTime.UtcNow;
    
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            var timeSinceLastProcess = DateTime.UtcNow - _lastProcessedTime;
            
            if (timeSinceLastProcess > TimeSpan.FromMinutes(5))
            {
                return HealthCheckResult.Degraded(
                    $"No jobs processed in {timeSinceLastProcess.TotalMinutes} minutes");
            }
            
            // Check queue depth
            var queueSize = await _queue.GetSizeAsync();
            if (queueSize > 1000)
            {
                return HealthCheckResult.Degraded(
                    $"Queue backlog: {queueSize} jobs");
            }
            
            return HealthCheckResult.Healthy("Background worker is healthy");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("Background worker check failed", ex);
        }
    }
}

// Registration
builder.Services.AddHealthChecks()
    .AddCheck<BackgroundWorkerHealthCheck>("background-worker");

Metrics Collection for Background Workers

public class BackgroundWorkerMetrics
{
    private readonly Counter<int> _processedJobs;
    private readonly Histogram<double> _processingTime;
    private readonly Counter<int> _failedJobs;
    
    public BackgroundWorkerMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("BackgroundWorker");
        
        _processedJobs = meter.CreateCounter<int>(
            "background.jobs.processed",
            description: "Number of background jobs processed");
        
        _processingTime = meter.CreateHistogram<double>(
            "background.job.processing.time",
            unit: "ms",
            description: "Time to process a background job");
        
        _failedJobs = meter.CreateCounter<int>(
            "background.jobs.failed",
            description: "Number of failed background jobs");
    }
    
    public async Task<T> TrackJobAsync<T>(
        Func<Task<T>> job,
        string jobType)
    {
        var stopwatch = Stopwatch.StartNew();
        
        try
        {
            var result = await job();
            _processedJobs.Add(1, new("job.type", jobType));
            _processingTime.Record(stopwatch.ElapsedMilliseconds);
            return result;
        }
        catch (Exception)
        {
            _failedJobs.Add(1, new("job.type", jobType));
            throw;
        }
    }
}

9.19 Error Handling and Dead Letter Queues

Implementing Dead Letter Queue

public class ResilientBackgroundWorker : BackgroundService
{
    private readonly IBackgroundTaskQueue _queue;
    private readonly IDeadLetterQueue _deadLetterQueue;
    private readonly ILogger<ResilientBackgroundWorker> _logger;
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                var job = await _queue.DequeueAsync(stoppingToken);
                
                try
                {
                    await ProcessJobWithRetryAsync(job, stoppingToken);
                    await _queue.CompleteJobAsync(job.Id);
                }
                catch (Exception ex) when (IsTransientFailure(ex))
                {
                    await _queue.RetryJobAsync(job.Id);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Job {JobId} failed permanently", job.Id);
                    await _deadLetterQueue.MoveToDeadLetterAsync(job, ex);
                }
            }
            catch (OperationCanceledException)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error in worker loop");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
        }
    }
    
    private async Task ProcessJobWithRetryAsync(
        BackgroundJob job, 
        CancellationToken token,
        int maxRetries = 3)
    {
        for (int attempt = 1; attempt <= maxRetries; attempt++)
        {
            try
            {
                await ExecuteJobAsync(job, token);
                return;
            }
            catch (Exception ex) when (attempt < maxRetries)
            {
                _logger.LogWarning(ex, 
                    "Job {JobId} failed on attempt {Attempt}", job.Id, attempt);
                await Task.Delay(ExponentialBackoff(attempt), token);
            }
        }
    }
}

9.20 Configurable Background Workers

Configuration-Based Worker Settings

public class WorkerSettings
{
    public int BatchSize { get; set; } = 100;
    public TimeSpan Interval { get; set; } = TimeSpan.FromMinutes(5);
    public int MaxRetries { get; set; } = 3;
    public bool Enabled { get; set; } = true;
}

public class ConfigurableWorker : BackgroundService
{
    private readonly WorkerSettings _settings;
    private readonly ILogger<ConfigurableWorker> _logger;
    
    public ConfigurableWorker(IOptions<WorkerSettings> settings,
        ILogger<ConfigurableWorker> logger)
    {
        _settings = settings.Value;
        _logger = logger;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (!_settings.Enabled)
        {
            _logger.LogInformation("Worker is disabled");
            return;
        }
        
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessBatchAsync(_settings.BatchSize, stoppingToken);
                await Task.Delay(_settings.Interval, stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error in worker execution");
                await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
            }
        }
    }
}

// appsettings.json
{
  "WorkerSettings": {
    "BatchSize": 500,
    "Interval": "00:01:00",
    "MaxRetries": 5,
    "Enabled": true
  }
}

9.21 Testing Background Workers

Unit Testing Background Services

[Fact]
public async Task Worker_ProcessesJobs_WhenQueueHasItems()
{
    // Arrange
    var mockQueue = new Mock<IBackgroundTaskQueue>();
    var mockService = new Mock<IEmailService>();
    
    var job = (CancellationToken token) => 
        mockService.Object.SendEmailAsync("test@example.com", token);
    
    mockQueue.SetupSequence(q => q.DequeueAsync(It.IsAny<CancellationToken>()))
             .ReturnsAsync(job)
             .ThrowsAsync(new OperationCanceledException());
    
    var worker = new QueuedWorker(mockQueue.Object, Mock.Of<ILogger<QueuedWorker>>());
    
    // Act
    var cancellationToken = new CancellationTokenSource(TimeSpan.FromSeconds(1)).Token;
    await worker.StartAsync(cancellationToken);
    
    // Wait for worker to process
    await Task.Delay(100);
    
    // Assert
    mockService.Verify(s => s.SendEmailAsync("test@example.com", It.IsAny<CancellationToken>()), 
        Times.Once);
}

[Fact]
public async Task Worker_StopsGracefully_WhenCancelled()
{
    // Arrange
    var cts = new CancellationTokenSource();
    var worker = new CleanupWorker(Mock.Of<ILogger<CleanupWorker>>());
    
    // Act
    var startTask = worker.StartAsync(cts.Token);
    await Task.Delay(100);
    cts.Cancel();
    await worker.StopAsync(CancellationToken.None);
    
    // Assert
    await Assert.ThrowsAsync<TaskCanceledException>(() => startTask);
}

Integration Testing with Host

[Fact]
public async Task Worker_RunsPeriodically_InIntegrationTest()
{
    // Arrange
    var hostBuilder = new HostBuilder()
        .ConfigureServices(services =>
        {
            services.AddHostedService<TestWorker>();
            services.AddSingleton<ITestService, TestService>();
        });
    
    using var host = await hostBuilder.StartAsync();
    
    var testService = host.Services.GetRequiredService<ITestService>();
    var worker = host.Services.GetServices<IHostedService>()
        .OfType<TestWorker>()
        .First();
    
    // Act
    await Task.Delay(TimeSpan.FromSeconds(2));
    
    // Assert
    Assert.True(testService.WasCalled, "Worker should have called service");
    
    // Cleanup
    await host.StopAsync(TimeSpan.FromSeconds(5));
}

9.22 Scaling Background Workers

Multiple Worker Instances with Leader Election

public class ScalableWorker : BackgroundService
{
    private readonly IDistributedLock _distributedLock;
    private readonly string _workerId;
    private readonly TimeSpan _lockTimeout = TimeSpan.FromMinutes(5);
    
    public ScalableWorker(IDistributedLock distributedLock)
    {
        _distributedLock = distributedLock;
        _workerId = Guid.NewGuid().ToString();
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            // Try to acquire lock (only one instance runs at a time)
            var lockAcquired = await _distributedLock.AcquireAsync(
                "worker-lock", 
                _workerId, 
                _lockTimeout);
            
            if (lockAcquired)
            {
                try
                {
                    _logger.LogInformation("Worker {WorkerId} acquired lock", _workerId);
                    await ProcessWorkAsync(stoppingToken);
                }
                finally
                {
                    await _distributedLock.ReleaseAsync("worker-lock", _workerId);
                }
            }
            else
            {
                _logger.LogDebug("Worker {WorkerId} waiting for lock", _workerId);
                await Task.Delay(TimeSpan.FromSeconds(10), stoppingToken);
            }
        }
    }
}

Horizontal Scaling with Work Partitioning

public class PartitionedWorker : BackgroundService
{
    private readonly int _workerIndex;
    private readonly int _totalWorkers;
    
    public PartitionedWorker(IConfiguration configuration)
    {
        _workerIndex = configuration.GetValue<int>("WorkerIndex");
        _totalWorkers = configuration.GetValue<int>("TotalWorkers");
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            // Process only items assigned to this worker
            var items = await GetAssignedItemsAsync();
            
            foreach (var item in items)
            {
                if (ShouldProcessItem(item))
                {
                    await ProcessItemAsync(item, stoppingToken);
                }
            }
            
            await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken);
        }
    }
    
    private bool ShouldProcessItem(WorkItem item)
    {
        // Simple hash-based partitioning
        var partition = item.Id.GetHashCode() % _totalWorkers;
        return partition == _workerIndex;
    }
}

Chapter 9 Summary

You learned how to build production-level async background processing using:

BackgroundService
IHostedService
Async queues (Channels)
Retry logic
Graceful cancellation
Report/email/background workflows
Task offloading for high performance
Real enterprise architecture patterns

Background workers turn your API into a high-performance distributed system capable of handling massive load without slowing down users.

✅ Chapter 9 is now complete.

Next up is an extremely important chapter for microservices & async architecture:

👉 Chapter 10: Common Async Pitfalls & Anti-Patterns

This includes:

Sync-over-async issues
Deadlocks
Thread starvation
Using Task.Run incorrectly
Blocking I/O in async apps
Async void disasters
Misusing ValueTask
Forgetting ConfigureAwait

Would you like me to proceed with Chapter 10 (Deep Dive) now?

Chapter 9: Background Work & Async Processing in .NET

Introduction

9.1 Why Background Processing Is Necessary

9.2 The Solution: Background Workers

API (fast)

Background Worker (async)

9.3 Understanding IHostedService & BackgroundService

Option 1: IHostedService (Low-level interface)

Option 2: BackgroundService (Recommended)

9.4 Example 1: Scheduled Background Cleanup Worker

Register it:

9.5 Example 2: Email Sender Background Worker

Step 1: Background Queue Interface

Step 2: Queue Implementation (using Channels)

Step 3: Worker That Runs Queue Jobs

Step 4: Register Queue + Worker

Step 5: API Controller Enqueues Email Sending

9.6 Use Case: PDF/Excel Report Generation

Why Background?

API

Client calls a status endpoint:

9.7 Use Case: Retrying Failed Jobs with Exponential Backoff

9.8 Communication Between APIs Using Queues or Events

Example worker:

9.9 Graceful Shutdown (Very Important)

9.10 Fire-and-Forget in API: Why It's Dangerous

❌ WRONG

✅ Right Way

9.11 Real-Life Architecture: High-Load Enterprise Pattern

API Layer

Queue

Worker

Dashboard

9.12 Observability for Background Tasks

Example:

9.13 Performance Tips

9.14 Best-Practices Checklist

API

Worker

System

9.15 Background Service Lifecycle Management

The Complete Lifecycle

Worker Dependencies and Scoped Services

9.16 Concurrent Background Services and Coordination

Running Multiple Services with Different Frequencies

Service Coordination with SemaphoreSlim

9.17 Persistent Queues for Reliable Background Processing

Redis Queue Implementation

Database-Backed Queue (SQL Server)

9.18 Monitoring and Health Checks for Background Workers

Implementing Health Checks

Metrics Collection for Background Workers

9.19 Error Handling and Dead Letter Queues

Implementing Dead Letter Queue

9.20 Configurable Background Workers

Configuration-Based Worker Settings

9.21 Testing Background Workers

Unit Testing Background Services

Integration Testing with Host

9.22 Scaling Background Workers

Multiple Worker Instances with Leader Election

Horizontal Scaling with Work Partitioning

Chapter 9 Summary

Post a Comment