ResourcePool.Docs/FIXES_SUMMARY.md

Summary of Fixes - ResourcePool Manager

📅 Date: 2026-01-22

🐛 Issues Fixed

1. ClickHouse "TOO_MANY_SIMULTANEOUS_QUERIES" Error

Error Code: 202
Impact: Service crashes when ClickHouse is overloaded

2. gRPC "Client Reset Request Stream" Errors

Errors:

• System.IO.IOException: The client reset the request stream
• Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted

Impact: Frequent connection drops, poor reliability

---

✅ Solutions Applied

Fix 1: ClickHouse Connection Management

Files Changed:

• src/Icomm.ResourcePool.Manager/Controllers/DTO/Context/ClickHouseContext.cs
• src/Icomm.ResourcePool.Manager/Controllers/Data/ProxyRepository.cs

Changes:

1. ClickHouseContext.cs

   • Changed from property proxy_logs to method CreateConnection()
   • Allows proper connection disposal with using statement
   • Cache connection string in constructor

2. ProxyRepository.cs

   • Use CreateConnection() with using statement
   • Add try-catch for ClickHouse errors
   • Catch error code 202 specifically
   • Add command timeout (5 seconds)
   • Log warning instead of throwing exception
   • Add using ClickHouse.Client.ADO

Results:

✅ Connections properly closed after each query
✅ Reduced simultaneous connections drastically
✅ Service continues working when ClickHouse is overloaded
✅ Logs warnings for monitoring

---

Fix 2: gRPC Connection Stability

Files Changed:

• src/Icomm.ResourcePool.Manager/Program.cs
• src/Icomm.ResourcePool.Manager/Startup.cs
• src/Icomm.ResourcePool.Manager/Services/ProxyService.cs

Changes:

1. Program.cs - Kestrel HTTP/2 Configuration

// HTTP/2 limits
options.Limits.Http2.MaxStreamsPerConnection = 100;
options.Limits.Http2.InitialConnectionWindowSize = 131072; // 128KB
options.Limits.Http2.InitialStreamWindowSize = 98304; // 96KB

// Timeouts
options.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(2);
options.Limits.RequestHeadersTimeout = TimeSpan.FromSeconds(30);

// Disable rate limits for gRPC
options.Limits.MinRequestBodyDataRate = null;
options.Limits.MinResponseDataRate = null;

Benefits:

• Prevents "HTTP/2 connection faulted" errors
• Allows more concurrent streams
• Increased window size for large messages
• No timeout on slow networks

2. Startup.cs - gRPC Configuration

services.AddGrpc(options =>
{
    options.MaxReceiveMessageSize = 10 * 1024 * 1024; // 10MB
    options.MaxSendMessageSize = 10 * 1024 * 1024; // 10MB
    options.EnableDetailedErrors = true;
});

Benefits:

• Prevents message size limit errors
• Detailed errors for easier debugging

3. ProxyService.cs - Logging & Error Handling

• Added Stopwatch to track request processing time
• Generate unique RequestId for each request
• Log when request starts (Debug level)
• Log when request completes with timing (Info level)
• Catch OperationCanceledException separately
• Log Warning (not Error) for client cancellations
• Applied to both GetProxy() and GetProxyV2()

Log Examples:

[abc12345] GetProxy started - AccessToken: token123
[abc12345] GetProxy completed in 150ms - ProxyId: 456

[xyz67890] GetProxy cancelled by client after 5000ms - AccessToken: token456

Results:

✅ More stable HTTP/2 connections
✅ Better handling of client disconnects
✅ Detailed logs for debugging
✅ Service doesn't crash on client cancellation

---

📊 Monitoring

Key Metrics to Watch:

ClickHouse:

# Count warnings
kubectl logs <pod> | grep -c "ClickHouse too many simultaneous queries"

# Should be < 1% of total requests

gRPC Cancellations:

# View cancelled requests
kubectl logs <pod> | grep "cancelled by client"

# View timing
kubectl logs <pod> | grep "completed in" | grep -oP '\d+ms' | sort -n

# Cancellation rate should be < 1%

Performance:

# P95 response time
kubectl logs <pod> | grep "completed in" | grep -oP '\d+ms' | sort -n | tail -n 50

# Should be < 1000ms for P95

---

🎯 Expected Improvements

Before:

• ❌ Service crashes when ClickHouse overloaded
• ❌ Frequent gRPC connection drops
• ❌ No visibility into request timing
• ❌ Errors logged without context

After:

• ✅ Service continues working during ClickHouse overload
• ✅ Stable gRPC connections with proper HTTP/2 config
• ✅ Detailed request timing logs
• ✅ Clear distinction between client cancellations and real errors
• ✅ Cancellation rate < 1%
• ✅ P95 response time < 1000ms

---

📝 Documentation Created

1. CLICKHOUSE_FIX.md - Detailed ClickHouse connection fix documentation
2. GRPC_CLIENT_RESET_DEBUG.md - gRPC debugging guide with solutions
3. FIXES_SUMMARY.md (this file) - Overall summary

---

🚀 Deployment Notes

Steps:

1. Deploy updated code
2. Monitor logs for first 30 minutes
3. Check metrics after 1 hour
4. Verify cancellation rate < 1%

Rollback Plan:

If issues occur, revert commits and:

• Check client timeout configuration
• Verify ClickHouse is not overloaded
• Review network connectivity

---

🔧 Future Optimizations (if needed)

If still experiencing issues:

ClickHouse:

• Implement batch insert (buffer multiple logs)
• Use background queue with Channel/BlockingCollection
• Implement circuit breaker pattern

gRPC:

• Increase client timeout
• Optimize RequestResource() performance
• Add caching for hot data
• Profile slow queries

---

📞 Support

If you see these patterns in logs:

Pattern 1: Many "cancelled by client after 5000ms" → Client timeout is 5 seconds, consider increasing

Pattern 2: "completed in 10000ms" (10+ seconds) → Server performance issue, need to optimize queries

Pattern 3: Continuous ClickHouse warnings → Consider batch insert or increase ClickHouse max_concurrent_queries

Pattern 4: "HTTP/2 connection faulted" → Check network stability and load balancer configuration