Key Takeaways
- Terminal-Bench 2.0 is a benchmark suite for evaluating AI agents on terminal-based tasks.
- Harbor is a new framework for testing AI agents in containerized environments.
- Terminal-Bench 2.0 addresses inconsistencies found in version 1.0.
- Harbor supports large-scale rollout infrastructure and integrates with Terminal-Bench 2.0.
- GPT-5 powered agents currently lead in task success on Terminal-Bench 2.0.
Terminal-Bench 2.0: Enhanced Benchmarking
The release of Terminal-Bench 2.0 introduces a more rigorous and verified task set for evaluating AI agents. This version replaces the initial release from May 2025, addressing inconsistencies and improving task clarity and reliability. The suite now includes 89 tasks, each validated to ensure they are solvable and realistic.
Improvements Over Version 1.0
- Task Validation: Tasks in Terminal-Bench 2.0 have undergone manual and LLM-assisted validation.
- Task Refinement: Tasks like “download-youtube” have been refactored due to unstable third-party APIs.
Harbor: Scalable Testing Framework
Harbor is a new framework designed to support the testing and optimization of AI agents in cloud-deployed containers. It enables scalable evaluations and integrates with both open-source and proprietary agents.
Features of Harbor
- Supports evaluation of any container-installable agent.
- Facilitates scalable supervised fine-tuning and reinforcement learning pipelines.
- Allows custom benchmark creation and deployment.
Early Results and Leaderboard
Initial results from the Terminal-Bench 2.0 leaderboard highlight the performance of various AI agents. OpenAI’s Codex CLI, powered by GPT-5, currently leads with a 49.6% success rate.
Top 5 Agent Results
- Codex CLI (GPT-5) — 49.6%
- Codex CLI (GPT-5-Codex) — 44.3%
- OpenHands (GPT-5) — 43.8%
- Terminus 2 (GPT-5-Codex) — 43.4%
- Terminus 2 (Claude Sonnet 4.5) — 42.8%
Submission and Integration
To test or submit an agent, users can install Harbor and run the benchmark using CLI commands. Submissions require five benchmark runs and can be emailed for validation.
FAQ
What is Terminal-Bench 2.0?
Terminal-Bench 2.0 is a benchmark suite for evaluating the performance of AI agents on terminal-based tasks, offering improved consistency and reliability.
How does Harbor enhance AI testing?
Harbor provides a scalable framework for testing AI agents in containerized environments, supporting large-scale evaluations and integration with Terminal-Bench 2.0.
Which AI agent currently leads on Terminal-Bench 2.0?
OpenAI’s Codex CLI, powered by GPT-5, currently leads with a 49.6% success rate.
Conclusion
The launch of Terminal-Bench 2.0 and Harbor represents a significant advancement in AI agent testing. For businesses running their own software or systems, these tools offer a robust foundation for consistent and scalable evaluation, crucial for optimizing AI performance in operational environments.
Source: Terminal-Bench 2.0 launches alongside Harbor, a new framework for testing agents in containers – venturebeat.com