SOMS/.kiro/specs/mongodb-compatibility-fix/tasks.md

Implementation Plan: MongoDB Compatibility Fix

Overview

This implementation replaces the MongoDB 4.4+ $unionWith operator with a MongoDB 3.x compatible approach by querying collections separately and performing deduplication in application memory using LINQ.

Tasks

• 1. Add helper method for dynamic property access

  • Create GetPropertyValue method to retrieve property values by name for sorting
  • Handle cases where property doesn't exist (log warning, return null)
  • Requirements: 1.5, 3.2

• * 1.1 Write property test for GetPropertyValue helper

  • Property: For any valid property name on SecondaryCircuitInspectionResult, GetPropertyValue should return the correct value
  • Validates: Requirements 1.5

• 2. Rewrite QueryPagedDeduplicatedResultsAcrossCollectionsAsync method

  • 2.1 Replace aggregation pipeline with separate collection queries

    • Remove all BsonDocument pipeline construction code
    • Query each collection separately using Find with filter
    • Apply projection to limit fields
    • Apply initial sort at MongoDB level
    • Use Task.WhenAll for parallel execution
    • Wrap each query in try-catch for resilience
    • Requirements: 1.1, 1.2, 1.3, 4.1

  • * 2.2 Write property test for parallel collection querying

    • Property 1: Deduplication Removes Duplicates by Key Fields
    • Validates: Requirements 1.4, 3.1

  • 2.3 Implement in-memory result merging

    • Use SelectMany to flatten collection results
    • Log number of results from each collection
    • Requirements: 1.3

  • 2.4 Implement in-memory sorting before deduplication

    • Use LINQ OrderBy/OrderByDescending with GetPropertyValue
    • Handle null sort field (skip sorting)
    • Requirements: 1.5

  • * 2.5 Write property test for sorting

    • Property 2: Sorting Produces Correct Order
    • Validates: Requirements 1.5

  • 2.6 Implement deduplication logic

    • Use LINQ GroupBy with anonymous type (Year, Month, Day, ItemId, Status)
    • Handle null Status by converting to empty string
    • Select first from each group
    • Requirements: 1.4, 3.1, 3.5

  • * 2.7 Write property test for first record selection

    • Property 3: First Record Selection After Sorting
    • Validates: Requirements 3.2

  • 2.8 Implement final sorting after deduplication

    • Apply same sort logic as before deduplication
    • Maintain user-requested sort order
    • Requirements: 1.5

  • 2.9 Implement counting and pagination

    • Count deduplicated results before pagination
    • Apply Skip and Take for pagination
    • Return tuple of (results, totalCount)
    • Requirements: 3.3, 3.4

  • * 2.10 Write property test for pagination

    • Property 4: Pagination Returns Correct Slice
    • Validates: Requirements 3.3

  • * 2.11 Write property test for total count

    • Property 5: Total Count Matches Deduplicated Count
    • Validates: Requirements 3.4

• 3. Update logging and performance monitoring

  • Log collection query start and completion
  • Log number of results from each collection
  • Log deduplication metrics (before/after counts)
  • Maintain existing performance warning logic (2000ms threshold)
  • Requirements: 2.2, 4.3, 4.4, 4.5

• * 3.1 Write unit tests for logging behavior

  • Test that performance warnings are logged when threshold exceeded
  • Test that collection query failures are logged
  • Requirements: 4.1, 4.2, 4.5

• 4. Handle edge cases and error conditions

  • 4.1 Add early return for empty collection list

    • Check if collectionNames is null or empty
    • Return (empty list, 0) immediately
    • Requirements: 5.5

  • 4.2 Ensure CancellationToken is passed through all async operations

    • Pass to all ToListAsync calls
    • Pass to Task.WhenAll
    • Requirements: 5.4

  • * 4.3 Write unit test for cancellation support

    • Test that cancellation is properly handled
    • Requirements: 5.4

  • * 4.4 Write unit test for empty collection list

    • Test that empty list returns empty results
    • Requirements: 5.5

  • * 4.5 Write unit test for all collections failing

    • Test that appropriate error handling occurs
    • Requirements: 4.2

• 5. Checkpoint - Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

• 6. Integration testing and verification

  • 6.1 Test with actual MongoDB 3.x instance

    • Verify queries execute without $unionWith errors
    • Compare results with MongoDB 4.4+ version (if available)
    • Requirements: 1.1, 3.1, 3.2, 3.3, 3.4

  • 6.2 Performance testing with realistic data volumes

    • Test with 1, 6, and 12 collections
    • Verify performance is acceptable (< 5 seconds)
    • Verify performance warnings are logged appropriately
    • Requirements: 2.1, 2.2

• 7. Final checkpoint - Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

Notes

• Tasks marked with * are optional and can be skipped for faster MVP
• Each task references specific requirements for traceability
• Checkpoints ensure incremental validation
• Property tests validate universal correctness properties
• Unit tests validate specific examples and edge cases
• The implementation maintains the same method signature for backward compatibility
• Performance may be slightly different from $unionWith but should remain acceptable