Sync README files from source repository [skip ci]
This commit is contained in:
Gitea Actions
@@ -0,0 +1,226 @@
|
||||
# Summary of Fixes - ResourcePool Manager
|
||||
|
||||
## 📅 Date: 2026-01-22
|
||||
|
||||
## 🐛 Issues Fixed
|
||||
|
||||
### 1. ClickHouse "TOO_MANY_SIMULTANEOUS_QUERIES" Error
|
||||
**Error Code:** 202
|
||||
**Impact:** Service crashes when ClickHouse is overloaded
|
||||
|
||||
### 2. gRPC "Client Reset Request Stream" Errors
|
||||
**Errors:**
|
||||
- `System.IO.IOException: The client reset the request stream`
|
||||
- `Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted`
|
||||
|
||||
**Impact:** Frequent connection drops, poor reliability
|
||||
|
||||
---
|
||||
|
||||
## ✅ Solutions Applied
|
||||
|
||||
### Fix 1: ClickHouse Connection Management
|
||||
|
||||
#### Files Changed:
|
||||
- `src/Icomm.ResourcePool.Manager/Controllers/DTO/Context/ClickHouseContext.cs`
|
||||
- `src/Icomm.ResourcePool.Manager/Controllers/Data/ProxyRepository.cs`
|
||||
|
||||
#### Changes:
|
||||
1. **ClickHouseContext.cs**
|
||||
- Changed from property `proxy_logs` to method `CreateConnection()`
|
||||
- Allows proper connection disposal with `using` statement
|
||||
- Cache connection string in constructor
|
||||
|
||||
2. **ProxyRepository.cs**
|
||||
- Use `CreateConnection()` with `using` statement
|
||||
- Add try-catch for ClickHouse errors
|
||||
- Catch error code 202 specifically
|
||||
- Add command timeout (5 seconds)
|
||||
- Log warning instead of throwing exception
|
||||
- Add `using ClickHouse.Client.ADO`
|
||||
|
||||
#### Results:
|
||||
✅ Connections properly closed after each query
|
||||
✅ Reduced simultaneous connections drastically
|
||||
✅ Service continues working when ClickHouse is overloaded
|
||||
✅ Logs warnings for monitoring
|
||||
|
||||
---
|
||||
|
||||
### Fix 2: gRPC Connection Stability
|
||||
|
||||
#### Files Changed:
|
||||
- `src/Icomm.ResourcePool.Manager/Program.cs`
|
||||
- `src/Icomm.ResourcePool.Manager/Startup.cs`
|
||||
- `src/Icomm.ResourcePool.Manager/Services/ProxyService.cs`
|
||||
|
||||
#### Changes:
|
||||
|
||||
**1. Program.cs - Kestrel HTTP/2 Configuration**
|
||||
```csharp
|
||||
// HTTP/2 limits
|
||||
options.Limits.Http2.MaxStreamsPerConnection = 100;
|
||||
options.Limits.Http2.InitialConnectionWindowSize = 131072; // 128KB
|
||||
options.Limits.Http2.InitialStreamWindowSize = 98304; // 96KB
|
||||
|
||||
// Timeouts
|
||||
options.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(2);
|
||||
options.Limits.RequestHeadersTimeout = TimeSpan.FromSeconds(30);
|
||||
|
||||
// Disable rate limits for gRPC
|
||||
options.Limits.MinRequestBodyDataRate = null;
|
||||
options.Limits.MinResponseDataRate = null;
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Prevents "HTTP/2 connection faulted" errors
|
||||
- Allows more concurrent streams
|
||||
- Increased window size for large messages
|
||||
- No timeout on slow networks
|
||||
|
||||
**2. Startup.cs - gRPC Configuration**
|
||||
```csharp
|
||||
services.AddGrpc(options =>
|
||||
{
|
||||
options.MaxReceiveMessageSize = 10 * 1024 * 1024; // 10MB
|
||||
options.MaxSendMessageSize = 10 * 1024 * 1024; // 10MB
|
||||
options.EnableDetailedErrors = true;
|
||||
});
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Prevents message size limit errors
|
||||
- Detailed errors for easier debugging
|
||||
|
||||
**3. ProxyService.cs - Logging & Error Handling**
|
||||
- Added `Stopwatch` to track request processing time
|
||||
- Generate unique `RequestId` for each request
|
||||
- Log when request starts (Debug level)
|
||||
- Log when request completes with timing (Info level)
|
||||
- Catch `OperationCanceledException` separately
|
||||
- Log Warning (not Error) for client cancellations
|
||||
- Applied to both `GetProxy()` and `GetProxyV2()`
|
||||
|
||||
**Log Examples:**
|
||||
```
|
||||
[abc12345] GetProxy started - AccessToken: token123
|
||||
[abc12345] GetProxy completed in 150ms - ProxyId: 456
|
||||
|
||||
[xyz67890] GetProxy cancelled by client after 5000ms - AccessToken: token456
|
||||
```
|
||||
|
||||
#### Results:
|
||||
✅ More stable HTTP/2 connections
|
||||
✅ Better handling of client disconnects
|
||||
✅ Detailed logs for debugging
|
||||
✅ Service doesn't crash on client cancellation
|
||||
|
||||
---
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
### Key Metrics to Watch:
|
||||
|
||||
**ClickHouse:**
|
||||
```bash
|
||||
# Count warnings
|
||||
kubectl logs <pod> | grep -c "ClickHouse too many simultaneous queries"
|
||||
|
||||
# Should be < 1% of total requests
|
||||
```
|
||||
|
||||
**gRPC Cancellations:**
|
||||
```bash
|
||||
# View cancelled requests
|
||||
kubectl logs <pod> | grep "cancelled by client"
|
||||
|
||||
# View timing
|
||||
kubectl logs <pod> | grep "completed in" | grep -oP '\d+ms' | sort -n
|
||||
|
||||
# Cancellation rate should be < 1%
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
```bash
|
||||
# P95 response time
|
||||
kubectl logs <pod> | grep "completed in" | grep -oP '\d+ms' | sort -n | tail -n 50
|
||||
|
||||
# Should be < 1000ms for P95
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Expected Improvements
|
||||
|
||||
### Before:
|
||||
- ❌ Service crashes when ClickHouse overloaded
|
||||
- ❌ Frequent gRPC connection drops
|
||||
- ❌ No visibility into request timing
|
||||
- ❌ Errors logged without context
|
||||
|
||||
### After:
|
||||
- ✅ Service continues working during ClickHouse overload
|
||||
- ✅ Stable gRPC connections with proper HTTP/2 config
|
||||
- ✅ Detailed request timing logs
|
||||
- ✅ Clear distinction between client cancellations and real errors
|
||||
- ✅ Cancellation rate < 1%
|
||||
- ✅ P95 response time < 1000ms
|
||||
|
||||
---
|
||||
|
||||
## 📝 Documentation Created
|
||||
|
||||
1. **CLICKHOUSE_FIX.md** - Detailed ClickHouse connection fix documentation
|
||||
2. **GRPC_CLIENT_RESET_DEBUG.md** - gRPC debugging guide with solutions
|
||||
3. **FIXES_SUMMARY.md** (this file) - Overall summary
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Deployment Notes
|
||||
|
||||
### Steps:
|
||||
1. Deploy updated code
|
||||
2. Monitor logs for first 30 minutes
|
||||
3. Check metrics after 1 hour
|
||||
4. Verify cancellation rate < 1%
|
||||
|
||||
### Rollback Plan:
|
||||
If issues occur, revert commits and:
|
||||
- Check client timeout configuration
|
||||
- Verify ClickHouse is not overloaded
|
||||
- Review network connectivity
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Future Optimizations (if needed)
|
||||
|
||||
If still experiencing issues:
|
||||
|
||||
**ClickHouse:**
|
||||
- Implement batch insert (buffer multiple logs)
|
||||
- Use background queue with Channel/BlockingCollection
|
||||
- Implement circuit breaker pattern
|
||||
|
||||
**gRPC:**
|
||||
- Increase client timeout
|
||||
- Optimize `RequestResource()` performance
|
||||
- Add caching for hot data
|
||||
- Profile slow queries
|
||||
|
||||
---
|
||||
|
||||
## 📞 Support
|
||||
|
||||
If you see these patterns in logs:
|
||||
|
||||
**Pattern 1:** Many "cancelled by client after 5000ms"
|
||||
→ Client timeout is 5 seconds, consider increasing
|
||||
|
||||
**Pattern 2:** "completed in 10000ms" (10+ seconds)
|
||||
→ Server performance issue, need to optimize queries
|
||||
|
||||
**Pattern 3:** Continuous ClickHouse warnings
|
||||
→ Consider batch insert or increase ClickHouse `max_concurrent_queries`
|
||||
|
||||
**Pattern 4:** "HTTP/2 connection faulted"
|
||||
→ Check network stability and load balancer configuration
|
||||
Reference in New Issue
Block a user