256 lines
4.9 KiB
Markdown
256 lines
4.9 KiB
Markdown
# Icomm.API.SmartPool
|
|
|
|
SmartPool API Backend - Intelligent proxy management system with multiple selection strategies.
|
|
|
|
## Features
|
|
|
|
- **Multiple Selection Strategies**:
|
|
- Random: Random selection from available proxies
|
|
- Round Robin: Balanced distribution across proxies
|
|
- Least Delay: Select fastest proxy based on response time
|
|
- Adaptive Ranking: Select proxy with highest success rate
|
|
- Alternative: Find similar proxy to replace failed one
|
|
|
|
- **Dual Protocol Support**:
|
|
- gRPC (MagicOnion) for high-performance RPC
|
|
- HTTP REST API for standard web clients
|
|
|
|
- **ClickHouse Integration**:
|
|
- High-performance logging
|
|
- Real-time analytics
|
|
- Automatic aggregation with materialized views
|
|
|
|
## Troubleshooting
|
|
|
|
### "No proxy available matching criteria"
|
|
|
|
If you get this error, check:
|
|
|
|
1. **Run initialization script:**
|
|
```bash
|
|
clickhouse-client < clickhouse_init.sql
|
|
```
|
|
|
|
2. **Add test data:**
|
|
```bash
|
|
clickhouse-client < clickhouse_test_data.sql
|
|
```
|
|
|
|
3. **Debug the issue:**
|
|
```bash
|
|
clickhouse-client < clickhouse_debug.sql
|
|
```
|
|
|
|
4. **Verify your access_token has mappings:**
|
|
```sql
|
|
SELECT * FROM smart_pool_meta.proxy_access_mapping FINAL
|
|
WHERE access_token = 'YOUR_TOKEN';
|
|
```
|
|
|
|
5. **Verify proxies exist in those clusters:**
|
|
```sql
|
|
SELECT m.*
|
|
FROM smart_pool_meta.proxy_metadata FINAL m
|
|
INNER JOIN smart_pool_meta.proxy_access_mapping FINAL a
|
|
ON m.cluster = a.cluster
|
|
WHERE a.access_token = 'YOUR_TOKEN'
|
|
AND m.status = 1;
|
|
```
|
|
|
|
### Common Issues
|
|
|
|
1. **Empty database**: Run `clickhouse_init.sql` and `clickhouse_test_data.sql`
|
|
2. **No access mapping**: Your access_token needs to be mapped to clusters
|
|
3. **All proxies inactive**: Check `status = 1` in proxy_metadata
|
|
4. **Wrong cluster names**: Ensure cluster names match between metadata and mappings
|
|
|
|
## Quick Start
|
|
|
|
### 1. Setup ClickHouse
|
|
|
|
```bash
|
|
# Run ClickHouse initialization script
|
|
clickhouse-client < clickhouse_init.sql
|
|
```
|
|
|
|
### 2. Configure Settings
|
|
|
|
Edit `appsettings.json`:
|
|
|
|
```json
|
|
{
|
|
"ClickHouse": {
|
|
"Host": "localhost",
|
|
"Port": 8123,
|
|
"Database": "smart_pool",
|
|
"Username": "default",
|
|
"Password": ""
|
|
},
|
|
"Redis": {
|
|
"Configuration": "localhost:6379"
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3. Run the API
|
|
|
|
```bash
|
|
dotnet run
|
|
```
|
|
|
|
The API will be available at:
|
|
- HTTP: `http://localhost:5000`
|
|
- gRPC: `http://localhost:5001`
|
|
- Swagger UI: `http://localhost:5000/swagger`
|
|
|
|
## API Endpoints
|
|
|
|
### Get Proxy
|
|
|
|
**POST** `/api/smart-pool/v1/SmartProxy/get-proxy`
|
|
|
|
Get a proxy server based on strategy and filters.
|
|
|
|
**Request Body:**
|
|
```json
|
|
{
|
|
"access_token": "your-token",
|
|
"strategy": "least_delay",
|
|
"target_domain": "facebook.com",
|
|
"ip_version": "v6",
|
|
"protocol": "http",
|
|
"country": "US"
|
|
}
|
|
```
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"error_code": 0,
|
|
"message": null,
|
|
"proxy": {
|
|
"id": 123,
|
|
"host": "192.168.1.100",
|
|
"port": 8080,
|
|
"protocol": "http",
|
|
"ip_version": "v6",
|
|
"country": "US",
|
|
"auth_username": "user",
|
|
"auth_password": "pass"
|
|
}
|
|
}
|
|
```
|
|
|
|
**Available Strategies:**
|
|
- `round_robin` (default): Balanced distribution
|
|
- `random`: Random selection
|
|
- `least_delay`: Fastest proxy (requires `target_domain`)
|
|
- `adaptive_ranking`: Highest success rate (requires `target_domain`)
|
|
- `alternative`: Similar proxy replacement (requires `referer_proxy.id`)
|
|
|
|
### Log Usage
|
|
|
|
**POST** `/api/smart-pool/v1/SmartProxy/log-usage`
|
|
|
|
Log proxy usage for analytics and strategy optimization.
|
|
|
|
**Request Body:**
|
|
```json
|
|
{
|
|
"access_token": "your-token",
|
|
"proxy_id": 123,
|
|
"target_domain": "facebook.com",
|
|
"status_code": 200,
|
|
"response_time_ms": 450,
|
|
"error_message": null,
|
|
"request_id": "req-12345-abc"
|
|
}
|
|
```
|
|
|
|
### Get Available Strategies
|
|
|
|
**GET** `/api/smart-pool/v1/SmartProxy/strategies`
|
|
|
|
Get list of available proxy selection strategies.
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"strategies": [
|
|
"round_robin",
|
|
"random",
|
|
"least_delay",
|
|
"adaptive_ranking",
|
|
"alternative"
|
|
]
|
|
}
|
|
```
|
|
|
|
### Health Check
|
|
|
|
**GET** `/api/smart-pool/v1/SmartProxy/health`
|
|
|
|
Check API health status.
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"status": "healthy",
|
|
"timestamp": "2024-01-15T10:30:00Z"
|
|
}
|
|
```
|
|
|
|
### Upsert Proxy
|
|
|
|
**POST** `/api/smart-pool/v1/SmartProxy/proxy/upsert`
|
|
|
|
Add or update proxy metadata.
|
|
|
|
### Add Access Mapping
|
|
|
|
**POST** `/api/smart-pool/v1/SmartProxy/access-mapping/add`
|
|
|
|
Add access token to cluster mapping.
|
|
|
|
## gRPC Usage
|
|
|
|
```csharp
|
|
var channel = GrpcChannel.ForAddress("http://localhost:5001");
|
|
var client = MagicOnionClient.Create<ISmartProxyService>(channel);
|
|
|
|
var response = await client.GetProxy(new GetProxyRequest
|
|
{
|
|
access_token = "your-token",
|
|
strategy = "adaptive_ranking"
|
|
});
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Client Request
|
|
↓
|
|
API Layer (gRPC/HTTP)
|
|
↓
|
|
Strategy Factory
|
|
↓
|
|
Strategy Implementation
|
|
↓
|
|
Repository Layer
|
|
↓
|
|
ClickHouse Database
|
|
```
|
|
|
|
## Dependencies
|
|
|
|
- .NET 8.0
|
|
- MagicOnion.Server 5.1.11
|
|
- ClickHouse.Client 7.7.0
|
|
- EasyCaching.Redis 1.9.2
|
|
- Serilog
|
|
|
|
## License
|
|
|
|
Internal use only.
|