SOMS/.kiro/specs/upload-inspection-task-result-performance/design.md

# Design Document: UploadInspectionTaskResult Performance Optimization

## Overview

本设计文档描述了对 `UploadInspectionTaskResult` 方法的性能优化方案。该方法目前存在超时问题，主要原因包括同步数据库查询、低效的文件处理、缺少批量操作和缓存机制。

优化策略包括：
- 异步数据库查询和并行加载
- 实现 Redis 缓存层
- 批量数据库操作
- 异步文件 I/O 和并行处理
- 内存优化和资源管理
- 性能监控和日志记录

## Architecture

### 当前架构问题

```
Request → Validate → Load All Presets (Sync) → Load All Videos (Sync) → 
  Loop Items → Process Files (Sync) → Insert MongoDB (One by One) → Response
```

**性能瓶颈：**
1. 同步加载所有预置点和视频设备到内存
2. 循环中逐个处理文件
3. 循环中逐个插入 MongoDB
4. 每次请求都重新加载相同数据

### 优化后架构

```
Request → Validate → 
  [Parallel: Check Cache → Load Presets (Async) + Load Videos (Async)] →
  [Parallel: Process Files Batch] → 
  Batch Insert MongoDB → 
  Update Cache → Response
```

**优化点：**
1. 并行异步加载数据
2. Redis 缓存层减少数据库查询
3. 并行处理文件
4. 批量插入 MongoDB
5. 性能监控和指标收集

## Components and Interfaces

### 1. CacheService (新增组件)

负责管理 Redis 缓存操作。

```csharp
public interface ICacheService
{
    Task<List<PresetPoint>> GetPresetsAsync(bool forceRefresh = false);
    Task<List<VideoDev>> GetVideosAsync(bool forceRefresh = false);
    Task InvalidatePresetsAsync();
    Task InvalidateVideosAsync();
}

public class CacheService : ICacheService
{
    private readonly IRedisRepository<List<PresetPoint>, string> _presetCache;
    private readonly IRedisRepository<List<VideoDev>, string> _videoCache;
    private readonly IRepository<PresetPoint, Guid> _presetRepository;
    private readonly IRepository<VideoDev, Guid> _videoRepository;
    private readonly TimeSpan _cacheExpiration = TimeSpan.FromHours(1);
    
    // Implementation details...
}
```

### 2. FileProcessingService (新增组件)

负责并行处理文件上传。

```csharp
public interface IFileProcessingService
{
    Task<List<string>> ProcessFilesAsync(
        IFormCollection form, 
        int itemIndex, 
        int fileCount, 
        string relativePath,
        CancellationToken cancellationToken = default);
}

public class FileProcessingService : IFileProcessingService
{
    private const int MaxDegreeOfParallelism = 4;
    private const int MaxFileSize = 50 * 1024 * 1024;
    private static readonly string[] AllowedExtensions = { ".jpg", ".jpeg", ".png", ".bmp", ".gif" };
    
    // Implementation details...
}
```

### 3. BatchOperationService (新增组件)

负责批量数据库操作。

```csharp
public interface IBatchOperationService
{
    Task<bool> BatchInsertInspectionResultsAsync(
        List<OriginalInspectionStoreResult> items,
        CancellationToken cancellationToken = default);
}

public class BatchOperationService : IBatchOperationService
{
    private readonly IMongoDbRepository<BsonDocument, string> _mongo;
    private const int BatchSize = 100;
    
    // Implementation details...
}
```

### 4. PerformanceMonitoringService (新增组件)

负责性能监控和指标收集。

```csharp
public interface IPerformanceMonitoringService
{
    IDisposable BeginOperation(string operationName);
    void RecordMetric(string metricName, long value);
    void RecordMetric(string metricName, TimeSpan duration);
}

public class PerformanceMonitoringService : IPerformanceMonitoringService
{
    private readonly ILogger _logger;
    
    public IDisposable BeginOperation(string operationName)
    {
        return new OperationTimer(operationName, this);
    }
    
    private class OperationTimer : IDisposable
    {
        private readonly Stopwatch _stopwatch;
        private readonly string _operationName;
        private readonly PerformanceMonitoringService _service;
        
        public OperationTimer(string operationName, PerformanceMonitoringService service)
        {
            _operationName = operationName;
            _service = service;
            _stopwatch = Stopwatch.StartNew();
        }
        
        public void Dispose()
        {
            _stopwatch.Stop();
            _service.RecordMetric(_operationName, _stopwatch.Elapsed);
        }
    }
}
```

### 5. 优化后的 HandleUploadInspectionItemsAppService

主要修改：
- 注入新的服务组件
- 使用缓存服务获取数据
- 使用文件处理服务并行处理文件
- 使用批量操作服务插入数据
- 添加性能监控

```csharp
public class HandleUploadInspectionItemsAppService : ApplicationService
{
    private readonly ICacheService _cacheService;
    private readonly IFileProcessingService _fileProcessingService;
    private readonly IBatchOperationService _batchOperationService;
    private readonly IPerformanceMonitoringService _performanceMonitoring;
    
    // 其他现有依赖...
    
    public async Task<ExcternalResquestSimpleResult> UploadInspectionTaskResult()
    {
        using var _ = _performanceMonitoring.BeginOperation("UploadInspectionTaskResult");
        
        // 优化后的实现...
    }
}
```

## Data Models

### CacheKey 常量

```csharp
public static class CacheKeys
{
    public const string PresetPoints = "cache:preset_points";
    public const string VideoDevices = "cache:video_devices";
}
```

### PerformanceMetrics 模型

```csharp
public class PerformanceMetrics
{
    public string OperationName { get; set; }
    public DateTime Timestamp { get; set; }
    public TimeSpan Duration { get; set; }
    public int ItemsProcessed { get; set; }
    public long MemoryUsed { get; set; }
}
```

### 现有模型保持不变

- `OriginalInspectionStoreResult`
- `PresetPoint`
- `VideoDev`
- `ExcternalResquestSimpleResult`


## Correctness Properties

*A property is a characteristic or behavior that should hold true across all valid executions of a system—essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*

### Property 1: Parallel Query Execution Performance

*For any* data loading operation requiring multiple data sources, executing queries in parallel should complete faster than executing them sequentially.

**Validates: Requirements 1.3**

### Property 2: Cache Hit Reduces Database Calls

*For any* data request within the cache validity period, subsequent requests for the same data should retrieve from cache without querying the database.

**Validates: Requirements 1.4, 4.1, 4.2**

### Property 3: Cache Miss Triggers Database Load and Update

*For any* data request when cache is empty or expired, the system should load from database and update the cache.

**Validates: Requirements 4.3**

### Property 4: Parallel File Processing Performance

*For any* file collection exceeding the threshold, parallel processing should complete faster than sequential processing.

**Validates: Requirements 2.3**

### Property 5: File Processing Resilience

*For any* collection of files where one or more fail to save, the system should continue processing remaining files and return results for successful saves.

**Validates: Requirements 2.4**

### Property 6: Batch Insert Reduces Database Calls

*For any* collection of inspection results, batch insertion should result in fewer database calls than individual insertions.

**Validates: Requirements 3.1**

### Property 7: Batch Size Partitioning

*For any* data collection exceeding the batch size threshold, the system should partition into multiple batches to avoid memory overflow.

**Validates: Requirements 3.3**

### Property 8: Performance Logging Completeness

*For any* method execution, the system should log both start timestamp and total execution time.

**Validates: Requirements 6.1, 6.2, 6.3**

### Property 9: Exception Context Logging

*For any* exception that occurs during processing, the system should log detailed error context including operation name, timestamp, and error details.

**Validates: Requirements 6.5**

### Property 10: Early Validation Rejection

*For any* request with invalid or missing required fields, the system should return an error immediately without performing subsequent processing operations.

**Validates: Requirements 7.1, 7.2**

### Property 11: Invalid File Skipping

*For any* file collection containing files that violate size or type constraints, the system should skip invalid files while successfully processing valid ones.

**Validates: Requirements 7.3**

### Property 12: Validation Error Clarity

*For any* validation failure, the error message should clearly indicate which validation rule failed and what the expected format is.

**Validates: Requirements 7.4**

### Property 13: Concurrent Request Independence

*For any* set of concurrent requests, processing one request should not block or interfere with the processing of other requests.

**Validates: Requirements 8.1**

### Property 14: Thread-Safe Resource Access

*For any* shared resource accessed by concurrent requests, the system should use appropriate synchronization to prevent race conditions and data corruption.

**Validates: Requirements 8.2**

## Error Handling

### 1. Request Validation Errors

**Strategy:** Fail fast with clear error messages

```csharp
if (request == null || request.ContentType == null)
{
    return new ExcternalResquestSimpleResult
    {
        Status = 1,
        Message = "Invalid request: missing content type"
    };
}
```

### 2. Cache Failures

**Strategy:** Fallback to database with logging

```csharp
try
{
    return await _cacheService.GetPresetsAsync();
}
catch (RedisException ex)
{
    Log4Helper.Warn(this.GetType(), "Cache unavailable, falling back to database", ex);
    return await _presetRepository.GetAllAsync();
}
```

### 3. File Processing Errors

**Strategy:** Continue processing with error collection

```csharp
var errors = new List<string>();
foreach (var file in files)
{
    try
    {
        await ProcessFileAsync(file);
    }
    catch (Exception ex)
    {
        errors.Add($"Failed to process {file.FileName}: {ex.Message}");
        Log4Helper.Error(this.GetType(), $"File processing error", ex);
    }
}
```

### 4. Database Operation Errors

**Strategy:** Retry with exponential backoff for transient errors

```csharp
var retryPolicy = Policy
    .Handle<MongoException>()
    .WaitAndRetryAsync(3, retryAttempt => 
        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

await retryPolicy.ExecuteAsync(async () =>
{
    await _batchOperationService.BatchInsertInspectionResultsAsync(items);
});
```

### 5. Timeout Handling

**Strategy:** Use CancellationToken with configurable timeout

```csharp
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
try
{
    await ProcessRequestAsync(request, cts.Token);
}
catch (OperationCanceledException)
{
    Log4Helper.Warn(this.GetType(), "Operation timed out");
    return new ExcternalResquestSimpleResult
    {
        Status = 1,
        Message = "Request processing timed out"
    };
}
```

## Testing Strategy

### Dual Testing Approach

本项目将采用单元测试和性能测试相结合的方式：

**单元测试：**
- 验证特定的实现细节（如异步方法签名）
- 测试边界条件和错误处理
- 验证配置选项的行为
- 使用 xUnit 测试框架

**性能测试：**
- 验证并行处理比串行处理更快
- 验证缓存减少数据库调用
- 验证批量操作减少往返次数
- 测试并发处理能力
- 使用 BenchmarkDotNet 进行性能基准测试

### 单元测试示例

```csharp
[Fact]
public async Task GetPresetsAsync_ShouldReturnFromCache_WhenCacheHit()
{
    // Arrange
    var cachedData = new List<PresetPoint> { /* test data */ };
    _mockRedisRepository.Setup(x => x.GetOneAsync(It.IsAny<string>()))
        .ReturnsAsync(cachedData);
    
    // Act
    var result = await _cacheService.GetPresetsAsync();
    
    // Assert
    Assert.Equal(cachedData, result);
    _mockPresetRepository.Verify(x => x.GetAllAsync(), Times.Never);
}

[Fact]
public async Task ProcessFilesAsync_ShouldSkipInvalidFiles_AndProcessValidOnes()
{
    // Arrange
    var form = CreateFormWithMixedFiles(); // Some valid, some invalid
    
    // Act
    var result = await _fileProcessingService.ProcessFilesAsync(form, 0, 5, "test/");
    
    // Assert
    Assert.True(result.Count > 0); // Valid files processed
    Assert.True(result.Count < 5); // Invalid files skipped
}
```

### 性能测试示例

```csharp
[Benchmark]
public async Task ParallelQueryPerformance()
{
    var tasks = new[]
    {
        _cacheService.GetPresetsAsync(),
        _cacheService.GetVideosAsync()
    };
    await Task.WhenAll(tasks);
}

[Benchmark]
public async Task SequentialQueryPerformance()
{
    await _cacheService.GetPresetsAsync();
    await _cacheService.GetVideosAsync();
}

[Fact]
public async Task BatchInsert_ShouldBeFasterThan_IndividualInserts()
{
    // Arrange
    var items = GenerateTestItems(100);
    
    // Act
    var batchTime = await MeasureExecutionTime(() => 
        _batchOperationService.BatchInsertInspectionResultsAsync(items));
    
    var individualTime = await MeasureExecutionTime(() => 
        InsertIndividually(items));
    
    // Assert
    Assert.True(batchTime < individualTime);
}
```

### 集成测试

```csharp
[Fact]
public async Task UploadInspectionTaskResult_EndToEnd_Performance()
{
    // Arrange
    var request = CreateTestRequest(itemCount: 50, filesPerItem: 3);
    
    // Act
    var stopwatch = Stopwatch.StartNew();
    var result = await _appService.UploadInspectionTaskResult();
    stopwatch.Stop();
    
    // Assert
    Assert.Equal(0, result.Status);
    Assert.True(stopwatch.ElapsedMilliseconds < 5000); // Should complete in < 5 seconds
}
```

### 测试配置

- 最小迭代次数：100 次（对于性能基准测试）
- 每个正确性属性必须有对应的测试
- 测试标签格式：**Feature: upload-inspection-task-result-performance, Property {number}: {property_text}**

### 测试覆盖率目标

- 代码覆盖率：> 80%
- 分支覆盖率：> 70%
- 性能回归检测：所有关键路径必须有性能基准

## Implementation Notes

### 优化优先级

1. **高优先级（立即实施）：**
   - 实现 Redis 缓存层
   - 异步数据库查询
   - 批量 MongoDB 插入

2. **中优先级（第二阶段）：**
   - 并行文件处理
   - 性能监控和日志
   - 错误处理优化

3. **低优先级（可选）：**
   - 流量控制和队列机制
   - 高级性能指标收集

### 配置参数

```csharp
public class PerformanceOptimizationConfig
{
    public bool EnableCache { get; set; } = true;
    public TimeSpan CacheExpiration { get; set; } = TimeSpan.FromHours(1);
    public int MaxDegreeOfParallelism { get; set; } = 4;
    public int BatchSize { get; set; } = 100;
    public int MaxConcurrentRequests { get; set; } = 50;
    public TimeSpan RequestTimeout { get; set; } = TimeSpan.FromSeconds(30);
}
```

### 性能指标

需要监控的关键指标：
- 请求处理总时间
- 数据库查询时间
- 缓存命中率
- 文件处理时间
- 批量插入时间
- 内存使用量
- 并发请求数

### 向后兼容性

- 所有优化应该保持 API 接口不变
- 缓存功能可通过配置禁用以保持原有行为
- 性能监控不应影响功能正确性