SOMS/.kiro/specs/inspection-result-daily-deduplication/tasks.md

Implementation Plan: Inspection Result Daily Deduplication Fix

Overview

This implementation plan addresses the bug where daily deduplication of inspection results occurs after pagination instead of before. The fix involves refactoring the MongoDB aggregation pipeline in SecondaryCircuitInspectionResultAppService.FindDatas to perform deduplication before pagination.

Tasks

• 1. Create new aggregation pipeline method with deduplication

  • Create QueryPagedDeduplicatedResultsAcrossCollectionsAsync method
  • Implement MongoDB aggregation pipeline with $group stage for deduplication
  • Use $facet to get both count and paginated results in single query
  • Return tuple of (Results, TotalCount)
  • Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3, 4.1, 4.2, 4.3

• * 1.1 Write property test for deduplication completeness

  • Property 1: Deduplication Completeness
  • Validates: Requirements 1.2

• 2. Refactor FindDatas method to use new pipeline

  • Replace QueryPagedResultsAcrossCollectionsAsync call with new method
  • Remove in-memory deduplication logic (lines 712-720)
  • Use TotalCount from pipeline result instead of separate count query
  • Update logging to reflect new approach
  • Requirements: 1.1, 1.4, 2.1, 2.2, 2.3

• * 2.1 Write property test for pagination consistency

  • Property 2: Pagination Consistency
  • Validates: Requirements 2.2

• * 2.2 Write property test for count accuracy

  • Property 4: Count Accuracy
  • Validates: Requirements 2.1

• 3. Implement grouping key construction

  • Build $group stage with composite key (Year, Month, Day, ItemId, Status)
  • Use $first operator to preserve first record after sorting
  • Handle null Status values in grouping
  • Requirements: 1.2, 1.3

• * 3.1 Write unit test for null status handling

  • Test that null Status values are handled correctly in grouping
  • Requirements: 1.2

• 4. Implement $facet stage for count and pagination

  • Create facet with two pipelines: totalCount and data
  • Extract count from totalCount array
  • Extract paginated results from data array
  • Handle empty result sets correctly
  • Requirements: 2.1, 2.2, 3.4

• * 4.1 Write unit test for empty result sets

  • Test that empty results return TotalCount=0 and empty list
  • Requirements: 2.1

• 5. Update cross-collection query logic

  • Ensure $unionWith stages are added before $group
  • Verify consistent field projection across all collections
  • Test deduplication across collection boundaries
  • Requirements: 4.1, 4.2, 4.3

• * 5.1 Write property test for cross-collection consistency

  • Property 5: Cross-Collection Consistency
  • Validates: Requirements 4.2

• 6. Checkpoint - Ensure all tests pass

  • Run all unit tests and property tests
  • Verify query performance is acceptable
  • Check that API response format is unchanged
  • Ask the user if questions arise

• 7. Add performance monitoring

  • Add logging for deduplication operation timing
  • Log before/after record counts
  • Add warning logs if query time exceeds thresholds
  • Requirements: 5.2, 5.3, 5.4

• * 7.1 Write unit test for performance logging

  • Verify that performance metrics are logged correctly
  • Requirements: 5.3

• 8. Create database index recommendation

  • Document compound index on (Year, Month, Day, ItemId, Status, ExecutionTime)
  • Add index creation script to migration folder
  • Document index purpose and benefits
  • Requirements: 5.1

• 9. Update method documentation

  • Update XML comments for FindDatas method
  • Document the deduplication behavior
  • Add remarks about performance characteristics
  • Requirements: 6.1, 6.2, 6.3, 6.4

• 10. Final checkpoint - Integration testing

  • Test with real data from multiple collections
  • Verify backward compatibility with existing clients
  • Confirm TotalCount accuracy across different scenarios
  • Ensure all tests pass, ask the user if questions arise

Notes

• Tasks marked with * are optional and can be skipped for faster MVP
• Each task references specific requirements for traceability
• Checkpoints ensure incremental validation
• Property tests validate universal correctness properties
• Unit tests validate specific examples and edge cases
• The fix is query-only, no data migration required
• Existing API contract is maintained for backward compatibility