SOMS/.kiro/specs/upload-inspection-task-result-performance/design.md

Design Document: UploadInspectionTaskResult Performance Optimization

Overview

本设计文档描述了对 UploadInspectionTaskResult 方法的性能优化方案。该方法目前存在超时问题，主要原因包括同步数据库查询、低效的文件处理、缺少批量操作和缓存机制。

优化策略包括：

• 异步数据库查询和并行加载
• 实现 Redis 缓存层
• 批量数据库操作
• 异步文件 I/O 和并行处理
• 内存优化和资源管理
• 性能监控和日志记录

Architecture

当前架构问题

Request → Validate → Load All Presets (Sync) → Load All Videos (Sync) → 
  Loop Items → Process Files (Sync) → Insert MongoDB (One by One) → Response

性能瓶颈：

1. 同步加载所有预置点和视频设备到内存
2. 循环中逐个处理文件
3. 循环中逐个插入 MongoDB
4. 每次请求都重新加载相同数据

优化后架构

Request → Validate → 
  [Parallel: Check Cache → Load Presets (Async) + Load Videos (Async)] →
  [Parallel: Process Files Batch] → 
  Batch Insert MongoDB → 
  Update Cache → Response

优化点：

1. 并行异步加载数据
2. Redis 缓存层减少数据库查询
3. 并行处理文件
4. 批量插入 MongoDB
5. 性能监控和指标收集

Components and Interfaces

1. CacheService (新增组件)

负责管理 Redis 缓存操作。

public interface ICacheService
{
    Task<List<PresetPoint>> GetPresetsAsync(bool forceRefresh = false);
    Task<List<VideoDev>> GetVideosAsync(bool forceRefresh = false);
    Task InvalidatePresetsAsync();
    Task InvalidateVideosAsync();
}

public class CacheService : ICacheService
{
    private readonly IRedisRepository<List<PresetPoint>, string> _presetCache;
    private readonly IRedisRepository<List<VideoDev>, string> _videoCache;
    private readonly IRepository<PresetPoint, Guid> _presetRepository;
    private readonly IRepository<VideoDev, Guid> _videoRepository;
    private readonly TimeSpan _cacheExpiration = TimeSpan.FromHours(1);
    
    // Implementation details...
}

2. FileProcessingService (新增组件)

负责并行处理文件上传。

public interface IFileProcessingService
{
    Task<List<string>> ProcessFilesAsync(
        IFormCollection form, 
        int itemIndex, 
        int fileCount, 
        string relativePath,
        CancellationToken cancellationToken = default);
}

public class FileProcessingService : IFileProcessingService
{
    private const int MaxDegreeOfParallelism = 4;
    private const int MaxFileSize = 50 * 1024 * 1024;
    private static readonly string[] AllowedExtensions = { ".jpg", ".jpeg", ".png", ".bmp", ".gif" };
    
    // Implementation details...
}

3. BatchOperationService (新增组件)

负责批量数据库操作。

public interface IBatchOperationService
{
    Task<bool> BatchInsertInspectionResultsAsync(
        List<OriginalInspectionStoreResult> items,
        CancellationToken cancellationToken = default);
}

public class BatchOperationService : IBatchOperationService
{
    private readonly IMongoDbRepository<BsonDocument, string> _mongo;
    private const int BatchSize = 100;
    
    // Implementation details...
}

4. PerformanceMonitoringService (新增组件)

负责性能监控和指标收集。

public interface IPerformanceMonitoringService
{
    IDisposable BeginOperation(string operationName);
    void RecordMetric(string metricName, long value);
    void RecordMetric(string metricName, TimeSpan duration);
}

public class PerformanceMonitoringService : IPerformanceMonitoringService
{
    private readonly ILogger _logger;
    
    public IDisposable BeginOperation(string operationName)
    {
        return new OperationTimer(operationName, this);
    }
    
    private class OperationTimer : IDisposable
    {
        private readonly Stopwatch _stopwatch;
        private readonly string _operationName;
        private readonly PerformanceMonitoringService _service;
        
        public OperationTimer(string operationName, PerformanceMonitoringService service)
        {
            _operationName = operationName;
            _service = service;
            _stopwatch = Stopwatch.StartNew();
        }
        
        public void Dispose()
        {
            _stopwatch.Stop();
            _service.RecordMetric(_operationName, _stopwatch.Elapsed);
        }
    }
}

5. 优化后的 HandleUploadInspectionItemsAppService

主要修改：

• 注入新的服务组件
• 使用缓存服务获取数据
• 使用文件处理服务并行处理文件
• 使用批量操作服务插入数据
• 添加性能监控

public class HandleUploadInspectionItemsAppService : ApplicationService
{
    private readonly ICacheService _cacheService;
    private readonly IFileProcessingService _fileProcessingService;
    private readonly IBatchOperationService _batchOperationService;
    private readonly IPerformanceMonitoringService _performanceMonitoring;
    
    // 其他现有依赖...
    
    public async Task<ExcternalResquestSimpleResult> UploadInspectionTaskResult()
    {
        using var _ = _performanceMonitoring.BeginOperation("UploadInspectionTaskResult");
        
        // 优化后的实现...
    }
}

Data Models

CacheKey 常量

public static class CacheKeys
{
    public const string PresetPoints = "cache:preset_points";
    public const string VideoDevices = "cache:video_devices";
}

PerformanceMetrics 模型

public class PerformanceMetrics
{
    public string OperationName { get; set; }
    public DateTime Timestamp { get; set; }
    public TimeSpan Duration { get; set; }
    public int ItemsProcessed { get; set; }
    public long MemoryUsed { get; set; }
}

现有模型保持不变

• OriginalInspectionStoreResult
• PresetPoint
• VideoDev
• ExcternalResquestSimpleResult

Correctness Properties

A property is a characteristic or behavior that should hold true across all valid executions of a system—essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.

Property 1: Parallel Query Execution Performance

For any data loading operation requiring multiple data sources, executing queries in parallel should complete faster than executing them sequentially.

Validates: Requirements 1.3

Property 2: Cache Hit Reduces Database Calls

For any data request within the cache validity period, subsequent requests for the same data should retrieve from cache without querying the database.

Validates: Requirements 1.4, 4.1, 4.2

Property 3: Cache Miss Triggers Database Load and Update

For any data request when cache is empty or expired, the system should load from database and update the cache.

Validates: Requirements 4.3

Property 4: Parallel File Processing Performance

For any file collection exceeding the threshold, parallel processing should complete faster than sequential processing.

Validates: Requirements 2.3

Property 5: File Processing Resilience

For any collection of files where one or more fail to save, the system should continue processing remaining files and return results for successful saves.

Validates: Requirements 2.4

Property 6: Batch Insert Reduces Database Calls

For any collection of inspection results, batch insertion should result in fewer database calls than individual insertions.

Validates: Requirements 3.1

Property 7: Batch Size Partitioning

For any data collection exceeding the batch size threshold, the system should partition into multiple batches to avoid memory overflow.

Validates: Requirements 3.3

Property 8: Performance Logging Completeness

For any method execution, the system should log both start timestamp and total execution time.

Validates: Requirements 6.1, 6.2, 6.3

Property 9: Exception Context Logging

For any exception that occurs during processing, the system should log detailed error context including operation name, timestamp, and error details.

Validates: Requirements 6.5

Property 10: Early Validation Rejection

For any request with invalid or missing required fields, the system should return an error immediately without performing subsequent processing operations.

Validates: Requirements 7.1, 7.2

Property 11: Invalid File Skipping

For any file collection containing files that violate size or type constraints, the system should skip invalid files while successfully processing valid ones.

Validates: Requirements 7.3

Property 12: Validation Error Clarity

For any validation failure, the error message should clearly indicate which validation rule failed and what the expected format is.

Validates: Requirements 7.4

Property 13: Concurrent Request Independence

For any set of concurrent requests, processing one request should not block or interfere with the processing of other requests.

Validates: Requirements 8.1

Property 14: Thread-Safe Resource Access

For any shared resource accessed by concurrent requests, the system should use appropriate synchronization to prevent race conditions and data corruption.

Validates: Requirements 8.2

Error Handling

1. Request Validation Errors

Strategy: Fail fast with clear error messages

if (request == null || request.ContentType == null)
{
    return new ExcternalResquestSimpleResult
    {
        Status = 1,
        Message = "Invalid request: missing content type"
    };
}

2. Cache Failures

Strategy: Fallback to database with logging

try
{
    return await _cacheService.GetPresetsAsync();
}
catch (RedisException ex)
{
    Log4Helper.Warn(this.GetType(), "Cache unavailable, falling back to database", ex);
    return await _presetRepository.GetAllAsync();
}

3. File Processing Errors

Strategy: Continue processing with error collection

var errors = new List<string>();
foreach (var file in files)
{
    try
    {
        await ProcessFileAsync(file);
    }
    catch (Exception ex)
    {
        errors.Add($"Failed to process {file.FileName}: {ex.Message}");
        Log4Helper.Error(this.GetType(), $"File processing error", ex);
    }
}

4. Database Operation Errors

Strategy: Retry with exponential backoff for transient errors

var retryPolicy = Policy
    .Handle<MongoException>()
    .WaitAndRetryAsync(3, retryAttempt => 
        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

await retryPolicy.ExecuteAsync(async () =>
{
    await _batchOperationService.BatchInsertInspectionResultsAsync(items);
});

5. Timeout Handling

Strategy: Use CancellationToken with configurable timeout

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
try
{
    await ProcessRequestAsync(request, cts.Token);
}
catch (OperationCanceledException)
{
    Log4Helper.Warn(this.GetType(), "Operation timed out");
    return new ExcternalResquestSimpleResult
    {
        Status = 1,
        Message = "Request processing timed out"
    };
}

Testing Strategy

Dual Testing Approach

本项目将采用单元测试和性能测试相结合的方式：

单元测试：

• 验证特定的实现细节（如异步方法签名）
• 测试边界条件和错误处理
• 验证配置选项的行为
• 使用 xUnit 测试框架

性能测试：

• 验证并行处理比串行处理更快
• 验证缓存减少数据库调用
• 验证批量操作减少往返次数
• 测试并发处理能力
• 使用 BenchmarkDotNet 进行性能基准测试

单元测试示例

[Fact]
public async Task GetPresetsAsync_ShouldReturnFromCache_WhenCacheHit()
{
    // Arrange
    var cachedData = new List<PresetPoint> { /* test data */ };
    _mockRedisRepository.Setup(x => x.GetOneAsync(It.IsAny<string>()))
        .ReturnsAsync(cachedData);
    
    // Act
    var result = await _cacheService.GetPresetsAsync();
    
    // Assert
    Assert.Equal(cachedData, result);
    _mockPresetRepository.Verify(x => x.GetAllAsync(), Times.Never);
}

[Fact]
public async Task ProcessFilesAsync_ShouldSkipInvalidFiles_AndProcessValidOnes()
{
    // Arrange
    var form = CreateFormWithMixedFiles(); // Some valid, some invalid
    
    // Act
    var result = await _fileProcessingService.ProcessFilesAsync(form, 0, 5, "test/");
    
    // Assert
    Assert.True(result.Count > 0); // Valid files processed
    Assert.True(result.Count < 5); // Invalid files skipped
}

性能测试示例

[Benchmark]
public async Task ParallelQueryPerformance()
{
    var tasks = new[]
    {
        _cacheService.GetPresetsAsync(),
        _cacheService.GetVideosAsync()
    };
    await Task.WhenAll(tasks);
}

[Benchmark]
public async Task SequentialQueryPerformance()
{
    await _cacheService.GetPresetsAsync();
    await _cacheService.GetVideosAsync();
}

[Fact]
public async Task BatchInsert_ShouldBeFasterThan_IndividualInserts()
{
    // Arrange
    var items = GenerateTestItems(100);
    
    // Act
    var batchTime = await MeasureExecutionTime(() => 
        _batchOperationService.BatchInsertInspectionResultsAsync(items));
    
    var individualTime = await MeasureExecutionTime(() => 
        InsertIndividually(items));
    
    // Assert
    Assert.True(batchTime < individualTime);
}

集成测试

[Fact]
public async Task UploadInspectionTaskResult_EndToEnd_Performance()
{
    // Arrange
    var request = CreateTestRequest(itemCount: 50, filesPerItem: 3);
    
    // Act
    var stopwatch = Stopwatch.StartNew();
    var result = await _appService.UploadInspectionTaskResult();
    stopwatch.Stop();
    
    // Assert
    Assert.Equal(0, result.Status);
    Assert.True(stopwatch.ElapsedMilliseconds < 5000); // Should complete in < 5 seconds
}

测试配置

• 最小迭代次数：100 次（对于性能基准测试）
• 每个正确性属性必须有对应的测试
• 测试标签格式：Feature: upload-inspection-task-result-performance, Property {number}: {property_text}

测试覆盖率目标

• 代码覆盖率：> 80%
• 分支覆盖率：> 70%
• 性能回归检测：所有关键路径必须有性能基准

Implementation Notes

优化优先级

1. 高优先级（立即实施）：

   • 实现 Redis 缓存层
   • 异步数据库查询
   • 批量 MongoDB 插入

2. 中优先级（第二阶段）：

   • 并行文件处理
   • 性能监控和日志
   • 错误处理优化

3. 低优先级（可选）：

   • 流量控制和队列机制
   • 高级性能指标收集

配置参数

public class PerformanceOptimizationConfig
{
    public bool EnableCache { get; set; } = true;
    public TimeSpan CacheExpiration { get; set; } = TimeSpan.FromHours(1);
    public int MaxDegreeOfParallelism { get; set; } = 4;
    public int BatchSize { get; set; } = 100;
    public int MaxConcurrentRequests { get; set; } = 50;
    public TimeSpan RequestTimeout { get; set; } = TimeSpan.FromSeconds(30);
}

性能指标

需要监控的关键指标：

• 请求处理总时间
• 数据库查询时间
• 缓存命中率
• 文件处理时间
• 批量插入时间
• 内存使用量
• 并发请求数

向后兼容性

• 所有优化应该保持 API 接口不变
• 缓存功能可通过配置禁用以保持原有行为
• 性能监控不应影响功能正确性