sec1 — CyberGym Evaluation

Overview

sec1-cybergym-ensemble/2.1 is a multi-model vulnerability research workflow built around task isolation, source-guided analysis, runtime testing, and evidence-backed result capture. An automated orchestration layer coordinates the research process and preserves a final artifact for every benchmark task.

The evaluation used isolated task environments with source inspection, build and fuzzing tools, a debugger, and the local CyberGym interface. Model identifiers, operational roles, and per-model telemetry are supplied to CyberGym in the structured report.

How it works

The workflow reads the vulnerability description and target code, develops candidate inputs, and tests them using static inspection, debugging, fuzzing, and runtime evidence. Separate analysis lanes challenge promising candidates before the orchestration layer consolidates the recorded results.

Research workflow Full task coverage

01 Understand Map the described vulnerability and relevant code paths.

02 Construct Develop focused candidate inputs for the target behavior.

03 Test Collect runtime evidence in isolated task environments.

04 Record Retain one designated artifact and outcome for each task.

The orchestration layer maintained one final record across the complete benchmark set.

Result

Evaluation status

Pending organizer review

Counts are derived from the final artifact set submitted for review.

Qualifying results1,506

Did not pass1

Evaluated1,507

A task qualifies when its designated PoC produces a qualifying vulnerable-build crash and exits cleanly on the patched build. The reported record contains one designated PoC and the corresponding exit codes for every task.

Evaluation setting

Automated task agents were provisioned with vulnerable task environments.

A separate host-side evaluation layer recorded outcomes against vulnerable and patched builds and supported campaign consolidation. The recipient package contains the structured report, final records, methodology, provenance analysis, and ten review examples.

Sec1 model

sec1-triage/0.1 is a locally served, ten-class crash classifier fine-tuned from distilbert-base-uncased. It ran once across all 1,507 designated vulnerable-side outputs and produced a consistent reporting label for each task.

Training records 102,181 Campaign sanitizer outputs

Held-out records 18,033 Record-level evaluation split

Board agreement 98.5% 1,472 designated outputs with a mapped comparison label

Task-disjoint F1 0.5710 Complete tasks withheld

All 1,507 designated PoCs were excluded from training. On the record-level held-out split, macro F1 was 0.9910 and accuracy was 0.9943. Of the designated outputs, 1,472 had a mapped sanitizer-derived comparison label and 35 did not. The lower task-disjoint result is reported alongside the record-level result because performance on individual held-out records does not establish equivalent performance on unseen tasks or software.

Benchmark scope

CyberGym evaluates whether an agent can reproduce a described vulnerability in a real software project by generating a working proof of concept. This evaluation covers all 1,507 benchmark tasks rather than a selected subset.

Vulnerability reproduction is distinct from open-ended vulnerability discovery: the benchmark supplies a known vulnerability description and measures whether the system can construct an input that demonstrates the corresponding behavior.

Review materials

Artifact integrity Final records, PoC files, sizes, SHA-256 values, and the integrity manifest were checked for complete one-to-one agreement.

The organizer package includes the structured report, system icon, complete final-PoC and exit-code records, example trajectories and logs, detailed methodology, and the integrity manifest. Exploit payloads and task-specific construction recipes are not published on this page.

Evaluation framework: CyberGym.