Benchmarking Scientific Intelligence: Bioinformatics
Axon scores 92% on BixBench-Verified-50, a realistic benchmark for biological data analysis. We think this benchmark is saturated--what should come next?
Scientific research is inherently messy: questions are often under-specified, small experimental changes can have outsized impacts, and data analysis and interpretation can feel more subjective than quantitative. Consequently, benchmarking agents that perform scientific research is also inherently difficult, especially if you want to isolate specific areas to improve: how do you independently evaluate an agent’s knowledge base, reasoning skills, tool use, and resource efficiency? How do you weigh an agent’s process or progress against the accuracy of its final answer? How do you ensure both domain breadth and depth, or measure qualitative properties like creativity that are intangible yet critical to scientific progress? And how we do so reproducibly and objectively across agents? These are questions that the whole field of agent-driven research is grappling with and working hard to address.
Axon scores 92% on BixBench-Verified-50
Let’s start with a focused domain of biology, bioinformatics. The most widely adopted bioinformatics benchmark to-date is BixBench from FutureHouse, in which agents perform data analysis and interpretation to answer questions spanning genomics, transcriptomics, proteomics, imaging, and more. BixBench is generally well-regarded for its realism and is intended to probe agents’ ability to understand datasets, use Python- and R-based tools, perform multistep analyses, and interpret nuanced results.
However, BixBench is not without its faults, and since its release focus has shifted to BixBench-Verified-50, a refined subset of well-specified questions with unambiguous ground truth. BixBench-Verified-50 still covers a wide spread of bioinformatics domains but is significantly shorter (50 questions vs. 205), making it faster and cheaper to run.
We ran Axon on BixBench-Verified-50 and scored 92.0% ± 1.4% with multiple choice grading (no refusal option, using majority voting over 5 replicates; uncertainty estimate derived by bootstrapping across questions). Without majority voting, Axon scored 89.2% ± 2.7% on the same metric. These scores put Axon comfortably at the top of this benchmark, in line with other state-of-the-art agents including Biomni Lab, a biology agent by the creators of BixBench-Verified-50.
Importantly, Axon’s performance was achieved without any dedicated guidance or training on bioinformatics tasks. Rather, the quality of Axon’s reasoning harness and code execution environments enables the underlying intelligence (Anthropic’s Claude Opus 4.6 as of writing) to autonomously determine the right tools and analyses for the task at hand, and significantly outperform the raw, unharnessed model. To us, this out-of-the-box performance is also signal that Axon is ready to be tested in domains across biology and drug discovery, even those it has not been explicitly designed for.
Inspecting the errors
Across all 5 replicates we ran, Axon only ever answered unique 10 questions incorrectly, with 5 of these incorrectly selected 3 or more times. For three of these five questions, Axon’s replicates were unanimous in their “incorrect” responses. On inspection, we suspect that these are due to shortcomings in the benchmark questions. For bix-61-q5, calculating the answer from an intermediate VCF file provided for the sequencing pipeline in question yields the wrong answer—we calculated this manually to check. However, the provided VCF file incorrectly uses diploid ploidy rather than haploid, which is more appropriate for bacteria, so perhaps agents are expected to re-generate the intermediate file, in which case we find it misleading to provide an incorrect one. For bix-16-q1 and bix-16-q3, the questions use a semantic assumption for “gene essentiality” that Axon mis-interprets in the same way every time: more negative GeneEffect values indicate higher dependency in DepMap datasets, so Axon corresponds more negative essentiality with higher dependency while the benchmark takes the opposite convention.
Notably, K-Dense Web reported the same “wrong” answers as Axon for bix-61-q5 and bix-16-q1. Bio AI has also detailed the challenges with capsule bix-16 in a recent blog. These 3 cases are not methodological failures for Axon; the workflows are valid and the computations are correct based on the provided data and reasonable semantic assumptions. Revising the questions to ignore the existing file in bix-61-q5 and to clarify the “essentiality” ambiguity resulted in correct answers for all three problems.
For the two remaining questions Axon answered incorrectly 3 or more times, Axon consistently made reasonable but ultimately incorrect decisions around ambiguous parameters or definitions. Perhaps most importantly however, for all questions answered incorrectly, Axon clearly surfaces its reasoning to allow for frictionless user intervention and guidance in an interactive session.
Overall, BixBench-Verified-50 is a well-constructed benchmark; it is straightforward, realistic and fair. However, given the close scores of various agents over the past few months, we think BixBench-Verified-50 is saturated. BixBench was designed to probe nuanced data analytics in bioinformatics research, and state-of-the-art agents now meet its bar. While small additional gains are surely feasible with domain-specific training, without a range of more difficult questions, further maximizing scores on this benchmark will quickly dip into “benchmarking hacking” that may ultimately end up hurting the model.
The Need for New Benchmarks
And so, once again, the field of bioinformatics needs to develop new benchmarks if scientific AI is to continue advancing capabilities. Here’s our two cents on what we believe to be crucial to the next generation of benchmarks:
Commercial alignment: As agent capabilities grow, tasks should mature in tandem. Benchmarks that measure higher-order capabilities relevant to drug discovery, personalized medicines, rare disease biology or other commercial interests help ensure that model development aims to deliver practical value. Benchmarks should probe tasks that are closely aligned with the challenges researchers face in basic science and translational research, which can include nuanced interpretations, multi-objective optimizations, and simultaneous breadth and depth of domain expertise.
Clear execution instructions: Beyond the tasks that make up a benchmark, results are heavily affected by prompts, resources, and permissions given to an agent. In BixBench-Verified-50, for example, an agent limited to 5 minutes per question may perform differently than one without time limits. The intent of a benchmark is directed by the details of its instructions: the more clearly defined boundaries are, the more meaningful a comparison of entrants will be. We quite like the model MatBench Discovery uses for MLIPs, which maintains a leaderboard for both “compliant models” following stringent protocols and “non-compliant models” allowed free-reign in trying to optimize benchmark scores.
Clear evaluation protocols: Similarly, it should be obvious that the methods for evaluating agents’ responses on a benchmark must be well-defined. The original BixBench included three modes of evaluation (open answer, MCQ, MCQ with refusal option) and indicated use of majority voting in its scoring, but it is very unclear which metric users of the benchmark have been reporting. This ambiguity is exacerbated by a lack of portable, bug-free code, requiring entrants to interpret the intended protocols and roll their own implementations. Benchmark metrics and their method of calculation must be abundantly transparent to avoid confusion and muddling of the comparisons they aim to enable.
Process-aware evaluation: Binary scoring of final answers provides clear metrics and a meaningful measure of applied capabilities, but visibility into the process provides much more insightful guidance for improving models. Are agents performing efficiently, using the right tools, making sound analytical decisions, and/or applying appropriate methods? Trace-based evaluation, in addition to binary scoring, can examine the whole scientific process to better reflect the open-ended nature of real-world research.
Living benchmark: Agents and the domains they operate on are evolving faster than ever, and an ideal benchmark must similarly evolve to stay relevant. This may take different forms: regular and semi-automated updating, procedural generation based on recent literature, crowd-sourcing relevant problems like BioArena, etc. With any such implementation, clear versioning and release practices are critical to avoid the aforementioned issues of comparative ambiguity amongst evaluations on the benchmark.
We’re excited to contribute to the field of agentic science and to see it continue building and charting its progress. Let’s continue the conversation—please reach out with any ideas on what makes a meaningful benchmark!
We’re super excited about Axon and have more to say about it than fits in any blog post! If you have questions or comments or just want to talk with us, shoot me a message at sabrina@mirrorphysics.com.