137 lines
5.4 KiB
Markdown
137 lines
5.4 KiB
Markdown
# Implementation Plan: MongoDB Compatibility Fix
|
|
|
|
## Overview
|
|
|
|
This implementation replaces the MongoDB 4.4+ `$unionWith` operator with a MongoDB 3.x compatible approach by querying collections separately and performing deduplication in application memory using LINQ.
|
|
|
|
## Tasks
|
|
|
|
- [x] 1. Add helper method for dynamic property access
|
|
- Create `GetPropertyValue` method to retrieve property values by name for sorting
|
|
- Handle cases where property doesn't exist (log warning, return null)
|
|
- _Requirements: 1.5, 3.2_
|
|
|
|
- [ ]* 1.1 Write property test for GetPropertyValue helper
|
|
- **Property: For any valid property name on SecondaryCircuitInspectionResult, GetPropertyValue should return the correct value**
|
|
- **Validates: Requirements 1.5**
|
|
|
|
- [x] 2. Rewrite QueryPagedDeduplicatedResultsAcrossCollectionsAsync method
|
|
- [x] 2.1 Replace aggregation pipeline with separate collection queries
|
|
- Remove all BsonDocument pipeline construction code
|
|
- Query each collection separately using Find with filter
|
|
- Apply projection to limit fields
|
|
- Apply initial sort at MongoDB level
|
|
- Use Task.WhenAll for parallel execution
|
|
- Wrap each query in try-catch for resilience
|
|
- _Requirements: 1.1, 1.2, 1.3, 4.1_
|
|
|
|
- [ ]* 2.2 Write property test for parallel collection querying
|
|
- **Property 1: Deduplication Removes Duplicates by Key Fields**
|
|
- **Validates: Requirements 1.4, 3.1**
|
|
|
|
- [x] 2.3 Implement in-memory result merging
|
|
- Use SelectMany to flatten collection results
|
|
- Log number of results from each collection
|
|
- _Requirements: 1.3_
|
|
|
|
- [x] 2.4 Implement in-memory sorting before deduplication
|
|
- Use LINQ OrderBy/OrderByDescending with GetPropertyValue
|
|
- Handle null sort field (skip sorting)
|
|
- _Requirements: 1.5_
|
|
|
|
- [ ]* 2.5 Write property test for sorting
|
|
- **Property 2: Sorting Produces Correct Order**
|
|
- **Validates: Requirements 1.5**
|
|
|
|
- [x] 2.6 Implement deduplication logic
|
|
- Use LINQ GroupBy with anonymous type (Year, Month, Day, ItemId, Status)
|
|
- Handle null Status by converting to empty string
|
|
- Select first from each group
|
|
- _Requirements: 1.4, 3.1, 3.5_
|
|
|
|
- [ ]* 2.7 Write property test for first record selection
|
|
- **Property 3: First Record Selection After Sorting**
|
|
- **Validates: Requirements 3.2**
|
|
|
|
- [x] 2.8 Implement final sorting after deduplication
|
|
- Apply same sort logic as before deduplication
|
|
- Maintain user-requested sort order
|
|
- _Requirements: 1.5_
|
|
|
|
- [x] 2.9 Implement counting and pagination
|
|
- Count deduplicated results before pagination
|
|
- Apply Skip and Take for pagination
|
|
- Return tuple of (results, totalCount)
|
|
- _Requirements: 3.3, 3.4_
|
|
|
|
- [ ]* 2.10 Write property test for pagination
|
|
- **Property 4: Pagination Returns Correct Slice**
|
|
- **Validates: Requirements 3.3**
|
|
|
|
- [ ]* 2.11 Write property test for total count
|
|
- **Property 5: Total Count Matches Deduplicated Count**
|
|
- **Validates: Requirements 3.4**
|
|
|
|
- [x] 3. Update logging and performance monitoring
|
|
- Log collection query start and completion
|
|
- Log number of results from each collection
|
|
- Log deduplication metrics (before/after counts)
|
|
- Maintain existing performance warning logic (2000ms threshold)
|
|
- _Requirements: 2.2, 4.3, 4.4, 4.5_
|
|
|
|
- [ ]* 3.1 Write unit tests for logging behavior
|
|
- Test that performance warnings are logged when threshold exceeded
|
|
- Test that collection query failures are logged
|
|
- _Requirements: 4.1, 4.2, 4.5_
|
|
|
|
- [x] 4. Handle edge cases and error conditions
|
|
- [x] 4.1 Add early return for empty collection list
|
|
- Check if collectionNames is null or empty
|
|
- Return (empty list, 0) immediately
|
|
- _Requirements: 5.5_
|
|
|
|
- [x] 4.2 Ensure CancellationToken is passed through all async operations
|
|
- Pass to all ToListAsync calls
|
|
- Pass to Task.WhenAll
|
|
- _Requirements: 5.4_
|
|
|
|
- [ ]* 4.3 Write unit test for cancellation support
|
|
- Test that cancellation is properly handled
|
|
- _Requirements: 5.4_
|
|
|
|
- [ ]* 4.4 Write unit test for empty collection list
|
|
- Test that empty list returns empty results
|
|
- _Requirements: 5.5_
|
|
|
|
- [ ]* 4.5 Write unit test for all collections failing
|
|
- Test that appropriate error handling occurs
|
|
- _Requirements: 4.2_
|
|
|
|
- [ ] 5. Checkpoint - Ensure all tests pass
|
|
- Ensure all tests pass, ask the user if questions arise.
|
|
|
|
- [ ] 6. Integration testing and verification
|
|
- [ ] 6.1 Test with actual MongoDB 3.x instance
|
|
- Verify queries execute without $unionWith errors
|
|
- Compare results with MongoDB 4.4+ version (if available)
|
|
- _Requirements: 1.1, 3.1, 3.2, 3.3, 3.4_
|
|
|
|
- [ ] 6.2 Performance testing with realistic data volumes
|
|
- Test with 1, 6, and 12 collections
|
|
- Verify performance is acceptable (< 5 seconds)
|
|
- Verify performance warnings are logged appropriately
|
|
- _Requirements: 2.1, 2.2_
|
|
|
|
- [ ] 7. Final checkpoint - Ensure all tests pass
|
|
- Ensure all tests pass, ask the user if questions arise.
|
|
|
|
## Notes
|
|
|
|
- Tasks marked with `*` are optional and can be skipped for faster MVP
|
|
- Each task references specific requirements for traceability
|
|
- Checkpoints ensure incremental validation
|
|
- Property tests validate universal correctness properties
|
|
- Unit tests validate specific examples and edge cases
|
|
- The implementation maintains the same method signature for backward compatibility
|
|
- Performance may be slightly different from $unionWith but should remain acceptable
|