From a judging engine to a full OJ platform, 145+ commits and 675 tests, built from zero to horizontally scalable in one night.

Why rewrite

The previous Leverage OJ judging approach was a long-lived connection architecture of a NestJS controller plus multiple judging-machine clients, and it had clear issues:

• Judge machines were stateful long-lived clients, so failures could not be auto-recovered.
• Judge tasks were not persisted, so they were dropped on restart.
• It did not support Botzone game AI matches (only standard OJ judging).
• The frontend lacked features like per-testpoint results and Botzone replay.

This time the goal was to build a truly production-ready judging system from scratch, including:

• Botzone Neo: an independent Judge Service
• Leverage backend: integrates Botzone Neo as a Judge Provider
• Leverage frontend: judging result display + Botzone game replay

Overall architecture

┌─────────────────────────── Server 1 ────────────────────────────┐
│                                                                   │
│  Leverage Frontend  →  Leverage Backend  →  Botzone Neo Worker   │
│                               ↑ callback           │             │
│                               └────────────────────┘             │
│                                           │                       │
│                               Redis (shared queue)                │
└───────────────────────────────────────────────────────────────────┘

┌─────────────────── Server 2..N ──────────────────────────────────┐
│  Botzone Neo Worker (stateless)  ────────────────→  Redis        │
└──────────────────────────────────────────────────────────────────┘

Core design principles:

• Botzone Neo is fully stateless: all task state lives in Redis, workers can scale horizontally without constraints.
• Leverage handles business: permissions, persistence, and display; it does not execute judging.
• Async + callback: Leverage returns immediately after enqueuing a task, and results are written back through callbacks.

Botzone Neo: judging engine design

DDD layering

src/
├── domain/           # Pure business logic, no framework imports
│   ├── verdict.ts    # AC/WA/TLE/RE/CE/SE enum
│   ├── oj/           # OJTask / TestcaseResult / IChecker
│   └── botzone/      # MatchTask / BotOutput / IJob
│
├── infrastructure/   # I/O adapter layer
│   ├── compile/      # CompileService: LRU cache, multi-language
│   ├── sandbox/      # ISandbox: DirectSandbox / NsjailSandbox
│   └── callback/     # CallbackService: retry + timeout + Request ID
│
├── strategies/       # Pluggable algorithms
│   ├── botzone/      # Restart / Standard / Checker / Longrun
│   └── oj/           # DiffChecker / CustomChecker
│
├── application/      # Use cases: compose domain + infrastructure
└── interface/        # HTTP controllers + Bull queues + DTOs

The Domain layer has zero NestJS imports, so tests are fast and logic stays clean.

ISandbox: the key abstraction

export interface ISandbox {
  execute(opts: SandboxExecuteOptions): Promise<SandboxResult>;
}

DirectSandbox (dev/Mac) runs through child_process.spawn, while NsjailSandbox (production Linux) wraps processes using nsjail with cgroups + seccomp + chroot.

Switching is only an environment variable, SANDBOX_BACKEND=nsjail|direct; no business code changes required.

Botzone protocol: four interaction modes

The core of Botzone is that the judge program communicates with bot processes through stdin/stdout:

Engine → Judger stdin: [current game log JSON]
Judger stdout → Engine: {"command":"request","content":{"0":"data-for-bot0"}}
Engine → Bot-0 stdin: "data-for-bot0"
Bot-0 stdout → Engine: {"response":"my-move"}
Engine → Judger stdin: [updated game log]
...
Judger stdout → Engine: {"command":"finish","content":{"0":1,"1":0}}

Four strategies:

Compile LRU cache

Compilation is the slowest step in judging, and in Botzone multiple rounds of the same bot against multiple opponents are common.

const key = `${language}:${sha256(source)}`;
if (this.cache.has(key)) {
  this.cacheHits.inc();    // Prometheus counter
  return this.cache.get(key)!;
}
const compiled = await this.doCompile(language, source);
this.cache.set(key, compiled);   // LRU auto evicts
return compiled;

The default capacity is 100, so the same bot source is compiled only once no matter how many games it runs.

SSRF protection

The judging engine needs to request user-provided callback URLs. Without protection, it effectively becomes an internal-network probing tool.

All common bypasses are blocked:

const PRIVATE_RANGES = [
  '127.0.0.0/8', '10.0.0.0/8',
  '172.16.0.0/12', '192.168.0.0/16',
  '::1', 'fc00::/7', 'fe80::/10',
  '169.254.0.0/16',
];
// Additional blocking:
// - IPv6 hex bypass:  http://0x7f000001/
// - URL-encoded:      http://%31%32%37.0.0.1/
// - Decimal encoding: http://2130706433/
// - Buffer overflow:  超长 hostname（>253 字节）

Each case has corresponding unit test coverage.

Bull async queue + result persistence

POST /v1/judge  →  Bull.add()  →  return { jobId }
                        │
               Worker Pool  (JUDGE_CONCURRENCY concurrency)
                        │ job.returnvalue = result
                        ▼
GET /v1/judge/:id/status  →  { state, result }

Results are written to job.returnvalue (Redis persistence), and clients can poll the status endpoint to fetch complete judging results without waiting on callbacks themselves.

Observability

# Prometheus metrics
botzone_judge_requests_total{type="oj",verdict="AC"} 42
botzone_judge_duration_ms_bucket{type="botzone",le="500"} 38
botzone_compile_cache_hits_total 156

Each HTTP request has X-Request-ID, propagated through pino structured logs and callback headers for end-to-end tracing.

Leverage backend integration

JudgeProvider abstraction

export interface IJudgeProvider {
  enqueue(params: EnqueueParams): Promise<EnqueueResult>;
  poll(submissionId: number, externalJobId: string): Promise<PollResult>;
  mapCallback(body: unknown): PollResult;
}

Integrating another judging platform later only requires implementing the same interface; Leverage business code stays untouched.

Database extension (append-only)

Add three nullable columns to the Submission table:

ALTER TABLE submission ADD COLUMN provider VARCHAR(32) NULL;
ALTER TABLE submission ADD COLUMN externalJobId VARCHAR(128) NULL;
ALTER TABLE submission ADD COLUMN providerMeta TEXT NULL;

providerMeta stores JSON. Botzone match results include gameLog (per-turn data) for frontend replay.

Callback security

botzone-neo  ──POST /botzone/callback──▶  leverage-backend
              Authorization: Bearer <BOTZONE_CALLBACK_TOKEN>

Three layers of protection:

1. Token validation: Bearer token check; requests without token return 401
2. Idempotency: process each jobId + state pair only once
3. Compensating polling: every 30s scan all pending submissions to avoid callback loss

Leverage frontend integration

Submission detail page

Updated /submissions/:id:

• Live polling: poll every 3s while pending/judging, stop on terminal status.
• OJ result panel: compile output + per-testpoint table (verdict badge, time, memory, expandable actual output).
• Botzone replay panel: auto-render when provider === 'botzone'.

Game Renderer plugin system

Each game can register its own Vue component as a renderer; if not registered, it falls back to a generic JSON view:

// Register renderer
registerRenderer('tictactoe', () => import('./TicTacToe.vue'));

// Root renderer loads dynamically by gameId
const loader = rendererRegistry.get(gameLog.gameId);
renderer.value = loader ? await loader() : GenericRenderer;

TicTacToe.vue is the first game-specific renderer. It renders a 3×3 board and shows the board state at the current turn.

ReplayViewer

Replay panel supports turn stepping (prev/next/first/last), autoplay, progress bar, and collapsible display for judgerDisplay plus botOutputs. The final turn highlights final scores.

Horizontal scaling

Workers are stateless, so scaling is just:

# New machine
git clone https://github.com/bkmashiro/botzone-neo
cp deploy/.env.worker.example .env
# Configure REDIS_HOST=<server1-ip>
docker compose -f deploy/docker-compose.worker.yml up -d

Concurrency tuning: JUDGE_CONCURRENCY = CPU cores × 2~4; set too high can lower throughput due to nsjail memory pressure.

Redis is recommended to be exposed to worker nodes through Tailscale private networking and not exposed to the public internet.