74 lines
8.8 KiB
Markdown
74 lines
8.8 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Two systems in one repo
|
|
|
|
This repo hosts **two generations** of a resource-pool service. Know which one you are touching:
|
|
|
|
- **ResourcePool (legacy)** — manages a **token pool** and a **proxy pool**. Server is `Icomm.ResourcePool.Manager` (ASP.NET, .NET 7). Clients consume it via the `Icomm.TokenPool` SDK or call the proxy pool directly over gRPC/HTTP. Shared contracts live in `Icomm.ResourcePool.Abstractions`. External ports: **HTTP 31249**, **gRPC 31287**.
|
|
- **SmartPool (new)** — intelligent proxy management. Server is `Icomm.API.SmartPool` (ASP.NET, .NET 10). Clients use the `Icomm.SmartPool.Proxy` SDK. Shared contracts live in `Icomm.SmartPool.Abstractions`. Listens on **HTTP :5000**, **gRPC :5001** (in-cluster). The active branch (`smart-resourcepool`) is centered on this system.
|
|
|
|
The legacy and smart stacks do **not** share code — they share only the repo, the `ConfigManager` remote-config dependency, and MagicOnion as the gRPC framework. Do not wire one into the other.
|
|
|
|
> Many of the root `SMARTPOOL_*.md` / `*_FIX*.md` docs are point-in-time notes and have drifted (e.g. `SMARTPOOL_README.md` still says .NET 8 / MagicOnion 5 / ClickHouse.Client 7 — the code is now .NET 10 / MagicOnion 6.1.7 / `ClickHouse.Driver` 0.9.0). Trust the `.csproj` files over the markdown. `MIGRATION_GUIDE_PROXYPOOL_TO_SMARTPOOL.md` is the most useful cross-cutting doc.
|
|
|
|
## Build / test / run
|
|
|
|
Solution: `Icomm.ResourcePool.sln`. Requires the **.NET 10 SDK** (the smart projects and tests target `net10.0`; the legacy SDKs target `netstandard2.1` and the Manager host targets `net7.0` but all build under the .NET 10 SDK).
|
|
|
|
```bash
|
|
dotnet build Icomm.ResourcePool.sln -c Release # build everything
|
|
dotnet test # run all tests (Icomm.SmartPool.Tests, xUnit)
|
|
dotnet test --filter "FullyQualifiedName~RandomStrategyTests" # one test class
|
|
dotnet test --filter "DisplayName~picks the least delayed" # one test by name
|
|
|
|
dotnet run --project src/Icomm.API.SmartPool # run SmartPool API locally
|
|
dotnet run --project src/Icomm.ResourcePool.Manager # run legacy Manager
|
|
```
|
|
|
|
Only `Icomm.SmartPool.Tests` exists (xUnit + Moq + FluentAssertions); it covers strategies, the strategy factory, and the client SDK. There are no tests for the legacy Manager.
|
|
|
|
**Docker / CI:** Only `src/Icomm.ResourcePool.Manager/Dockerfile` exists (multi-stage, .NET 7 alpine→aspnet, sets Asia/Ho_Chi_Minh TZ). There is no Dockerfile for `Icomm.API.SmartPool` yet and no `docker-compose.yml`. The legacy service ships via **Jenkins** (root `Jenkinsfile` → `src/Icomm.ResourcePool.Manager/Jenkinsfile`: docker build → push to registry → `kubectl apply` of `deployment.yaml`). The only `.gitea` workflow just syncs `*.md` files to a public mirror repo — it does not build or deploy.
|
|
|
|
## SmartPool architecture (`Icomm.API.SmartPool`)
|
|
|
|
Request entry points both implement the same logic surface:
|
|
- **gRPC**: `Services/SmartProxyGrpcService.cs` — MagicOnion `ServiceBase<ISmartProxyService>`. This is the primary, high-throughput path.
|
|
- **HTTP REST**: `Controllers/v1/SmartProxyController.cs` — routes under `/api/smart-pool/v1/SmartProxy`.
|
|
|
|
Core flow for "get a proxy":
|
|
1. Validate `access_token`; reject if it has **no cluster mapping** (`IProxyMetadataRepository.HasAccessTokenMappingAsync`). Access control is **token → cluster**, never token → individual proxy.
|
|
2. Validate the requested `strategy` name against the registered set; `alternative` additionally requires a `referer_proxy`.
|
|
3. Load the candidate proxies the token may use, then delegate selection to a strategy.
|
|
|
|
**Strategy pattern** (`Strategies/`) is the heart of the system. Every strategy implements `IProxyPickStrategy` (`StrategyName`, `ValidateRequest`, `PickProxyAsync`). All implementations are registered as `IEnumerable<IProxyPickStrategy>` in DI; `ProxyStrategyFactory.GetStrategy(name)` resolves by name and **falls back to `round_robin`** when unknown. To add a strategy: implement the interface, register it in `Infrastructure/ServiceCollectionExtensions.cs`, and it is automatically discoverable. The five strategies:
|
|
- `random` — uniform pick from the permitted set.
|
|
- `round_robin` — rotates sequentially; index persisted in **Redis** for even distribution.
|
|
- `least_delay` — lowest recent response time for a `target_domain` (last 24h); domain required.
|
|
- `adaptive_ranking` — score-ranked (`success_rate` weighted across recent windows, last 7d); domain required.
|
|
- `alternative` — finds a substitute similar to `referer_proxy` (same cluster/ip_version/country/etc., weighted by `AlternativeStrategyOptions` in config).
|
|
|
|
**Data layer** — ClickHouse, split into two databases by lifecycle:
|
|
- `smart_pool_meta`: `proxy_metadata`, `proxy_access_mapping` (ReplacingMergeTree, no TTL, queried with FINAL) → `ProxyMetadataRepository`.
|
|
- `smart_pool_logs`: `proxy_usage_logs` (TTL ~90d) plus a **Refreshable Materialized View** that aggregates hourly stats every ~5 min → `ProxyLogRepository`. Adaptive scoring reads the MV; there is **no** background scoring service (it was removed in favor of the MV — see the note in `ServiceCollectionExtensions.cs`).
|
|
- ClickHouse access is via Dapper over `ClickHouse.Driver` (`Infrastructure/ClickHouseContext.cs`, `ClickHouseConnectionWrapper.cs`). SQL schema lives in `src/Icomm.API.SmartPool/clickhouse_init.sql` (`_test_data` / `_debug` variants alongside).
|
|
|
|
**Write path for logs** is asynchronous: usage logs are produced to **Kafka** (`KafkaProducerService`, a singleton `IHostedService` doing batch/fire-and-forget; topic and tuning in `KafkaLogsOptions`). A separate consumer (outside this repo) lands them in ClickHouse. `LogProxyUsageBatch` is the high-throughput variant.
|
|
|
|
**Performance-critical infrastructure** (`Infrastructure/`): `HotDataCache` (in-memory, singleton), Redis via **EasyCaching** (`src/EasyCaching.Redis`, a vendored provider), `CircuitBreaker`, gRPC + HTTP response compression (gzip/brotli, `CompressionLevel.Fastest`), and 1 MB gRPC message limits. This service is tuned for latency/throughput — preserve the caching tiers (hot in-memory → Redis → ClickHouse) and the compact-response/batch endpoints when changing behavior.
|
|
|
|
## Legacy ResourcePool architecture
|
|
|
|
- **Host**: `Icomm.ResourcePool.Manager` (`Startup.cs`). MagicOnion gRPC on `:5002` plus a MagicOnion **HTTP gateway** (`_gate`) and Swagger gateway; responses wrapped by **AutoWrapper** for paths under `/api`.
|
|
- **Controllers** (`Controllers/v1/`): `TokenPoolController`, `ProxyPoolController`. Business logic in `Services/` (`TokenService`, `ProxyService`, `ElasticProvider`); persistence in `Controllers/Data/` repositories (`TokenRepository`, `ProxyRepository`, `RateLimitRepository`) backed by ClickHouse (`DTO/Context/`) and **Elasticsearch** (`ElasticConnectionOptions`).
|
|
- **Client SDK**: `Icomm.TokenPool` (`netstandard2.1`, published as NuGet `Icomm.TokenPool`). Register with `services.AddConfigManager(...).AddTokenPool(...)`; consume `ITokenPoolService` (`RequestToken`, `UsedToken`, `ExpireBy*`, `UpdateStatus`). Proxy pool has **no SDK** — clients call gRPC (`IProxyService` via `MagicOnionClient`) or the HTTP REST endpoints (`/api/resource-pool/v1/ProxyPool/...`) directly.
|
|
|
|
## Cross-cutting conventions
|
|
|
|
- **gRPC contracts use MagicOnion**, not `.proto` files. The C# interface in the `*.Abstractions` project (e.g. `ISmartProxyService`, `IProxyService`) **is** the contract; client and server share that assembly. Changing a method signature is a breaking wire change for both sides.
|
|
- **Request/response field names are snake_case** in SmartPool DTOs (`access_token`, `ip_version`, `target_domain`, `referer_proxy`) — this is intentional for the public contract; match it.
|
|
- **Remote configuration** comes from `ConfigManager` (`Icomm.Configs.Providers.HttpProvider`). SmartPool: `builder.Configuration.AddConfigManagerHttpProvider()` + `AddConfigManager(...)`. The `ConfigManager:AccessToken` in `appsettings.json` authenticates to the config server; real settings (ClickHouse/Redis/Kafka connection details) are fetched remotely, which is why local `appsettings.json` often has those sections empty or commented out.
|
|
- **SmartPool error handling** uses numeric codes returned in the response (no exceptions across the wire): 1001 AccessTokenRequired, 1002 AccessTokenInvalid, 2001 NoAvailableProxies, 2002 NoMatchingProxies, 3001 StrategyNotFound, 3002 RefererProxyRequired, 4001 InvalidRequest, 5001 InternalError. Build them via `Helpers/ProxyErrorHelper`.
|
|
- **Shared/abstraction projects are published as NuGet packages** to a private feed (source `ic` / ProGet; see `push-proget.sh`, `nuget.config`). `Icomm.SmartPool.Abstractions` multi-targets `net10.0;net8.0` so older consumers can reference it — keep it consumer-compatible and bump the `<Version>` when changing public types.
|