Introducing Navigator

Evaluations

Online-Mind2Web

Online-Mind2Web⁶ is an open benchmark designed to evaluate the real-world performance of web agents on live websites, featuring 300 tasks across 136 popular sites in diverse domains.

We report both human evaluation results from Halluminate and auto-evaluation results from WebJudge, the official Online-Mind2Web evaluator.

Navigator sets a new state of the art on Online-Mind2Web with 78.7% human-evaluation accuracy, far exceeding prior results from Gemini 2.5⁵(69.0%)³, OpenAI Operator (61.3%)³, Claude 4.0 (61.0%)⁷, and Claude 4.5 (55.0%)⁷. It also achieves the highest auto-evaluation performance of 64.7%, outperforming Gemini 2.5 (57.3%)³, OpenAI Operator (58.3%)³, Claude 4.0 (47.7%)⁸, and Claude 4.5 (59.3%)⁸.

Navigator also establishes a new pareto-frontier over previous models — making strict improvements in both accuracy⁹ and per-step latency¹⁰.

Navi-Bench

While Online-Mind2Web provides a wide coverage of websites, it relies on LLM-as-a-judge. Such auto-evaluators generally require human verification, which limits rapid iterations for model development.

Thus, we built Navi-Bench in collaboration with Halluminate — a benchmark designed to reliably evaluate web agents on everyday tasks directly on real websites, without humans in the loop.

Fundamentally, Navi-Bench relies on two key observations:

1. There exists a generator-verifier gap, i.e. it is typically easier to verify that the agent has successfully reached certain website states than for the agent to generate the sequence of actions to reach them, and
2. Agents need to generalize and scale to the web, verifiers do not.

Consider the task “Check on OpenTable if the Four Kings restaurant is available for dinner next Wednesday”. This requires the agent to search for the restaurant, set the correct date/time, and then read the availability information from the page. A verifier for this task is a simple Javascript function that extracts relevant variables (selected date, selected time, etc.) from the web page DOM as the agent navigates and compares it to the desired state for this task. Note that while relying on DOM to build agents does not generalize² or scale, relying on DOM to build deterministic and accurate verifiers for specific tasks works because it doesn’t need to generalize; the skills learned by the agent do.

In Navi-Bench v1, we publish 100 tasks across 5 real websites: Apartments, Craigslist, OpenTable, Resy, and Google Flights. Please check out the dataset on Hugging Face and the Python SDK on GitHub.

Experiments on web tasks (e.g. checking restaurant availability) are difficult to compare because the web is dynamic (restaurant availability changes) and some tasks become infeasible (e.g. queries about past availability cannot be answered). Sometimes, realistic tasks are stated in a way where the task specification changes based on when the task is run (e.g., queries about availability tomorrow). Existing web benchmarks have settled on a particularly egregious configuration e.g., “Does this restaurant have availability on Aug 1, 2025?”. The answer will change based on when the query is run, you can’t check availability on a past date, and real users don’t often specify tasks in this way. This causes a steady degradation in benchmark quality and different experiments remain incomparable. In Navi-Bench, we introduce a dynamic task configuration that renders the agent query and instantiates the corresponding success criteria when the experiment is run. This ensures that the evaluation reflects a current, valid real-world scenario.

Navi-Bench shares the same dataset format with Westworld — a benchmark from Halluminate featuring five simulated environments for e-commerce and travel tasks. This allows both benchmarks to be combined for evaluation. Furthermore, we intentionally created the tasks for real Google Flights similar to those of the corresponding simulated environment in Westworld (Noodle Flights), which allows us to study the sim-to-real gap for this domain.

Table 1. Evaluation results across Navi-Bench v1 (real websites) and Halluminate Westworld (simulated websites). We report each model’s average success rate over 3 runs.

On average, Navigator outperformed the other models by 9.2% on Navi-Bench v1 and 18.3% on Westworld. It also achieved the best success rate on most websites, including a perfect score on Megamart. This suggests that our model training system can effectively improve model reliability, and we may want to keep evolving the benchmarks to be even more challenging.

Furthermore, we can see that the results on Google Flights and Noodle Flights are highly correlated across different models. In the Reinforcement Learning section below, we show that by training with Noodle Flights, the model achieved similar improvements on both the real Google Flights and the simulated Noodle Flights. These observations suggest that the Halluminate Noodle Flights website has minimal sim-to-real gaps.

Real-world Evaluation in Scouts

Scouts monitor the web for anything a user may care about. This is fundamentally an information-seeking problem complicated by the realities of the web — data may not exist yet, may appear unpredictably, and may change quickly once it does. Accessing that information often requires direct interaction with websites. Evaluating computer-use agents in Scouts reveals how they perform in the wild, in the hands of users, outside controlled benchmarks.

We conducted human-preference evaluations¹¹ to measure:

• Outcome quality - Whether the model retrieves the correct information, cites correct source pages, acknowledges when the results only satisfy partial requirements, and provides helpful complementary details that let the user plan their next steps.
• Process quality - Whether the model correctly interacts with the relevant UI elements, applies filters and controls appropriately, stays persistent when the task requires persistent exploration, and adapts to real-world surprises such as slow loads, pop-ups, or website malfunctions.

Navigator achieves outcome win rates of 79.4%, 61.6%, and 53.4%, and process win rates of 62.6%, 66.1%, and 52.8%, against Gemini 2.5, Claude 4.0, and Claude 4.5 respectively, evaluated using the same browser infrastructure. Notably, Claude 4.5 stands out as the second-strongest model in our real production setting, suggesting that it was likely underrated in prior academic benchmark evaluations⁷.

Table 2. Pairwise human-preference comparison between Navigator and other models when used in Scouts.

How n1 was Trained

Supervised Learning

Training for n1 begins with two supervised learning stages: mid-training and supervised fine-tuning (SFT). During mid-training, the model learns from a large set of navigation trajectories, including those with unsuccessful or noisy outcomes, broadening its knowledge and building a general understanding of the web. We observed a highly predictable improvement in test negative log-likelihood (NLL) during mid-training, where the test set consisted only of successful trajectories with any deemed-incorrect UI interactions removed. This is followed by SFT, where training solely on clean and successful trajectories induces a concentration effect, which substantially boosts performance over mid-training.

Interestingly, we find that mid-training checkpoints (without SFT) do not always yield improvements in online performance¹³, despite clear gains in test NLL. After SFT, however, we consistently observe stronger online performance than with mid-training alone¹⁴, and these improvements show no signs of plateauing yet. This suggests that mid-training continues to instill broader knowledge about the web, but this knowledge may not immediately translate into online performance until SFT concentrates the model's behavior towards producing successful trajectories.

We found that it was important to emphasize certain key behaviors in the model learned during supervised learning — reliable UI interaction, task planning, progress verification, and recovery and backtracking — so the model is not only performant in typical scenarios, but also resilient to the oddities of the web. These were critical to preparing the model well for reinforcement learning.

Reinforcement Learning

The final stage of training for n1 is reinforcement learning. Compared to the base model, RL improved absolute performance on Navi-Bench and Westworld by 23% and 34% respectively, instilling new behaviors for solving tasks, and improving efficiency by reducing the average number of steps taken by 30%.

Successfully training n1 with RL involved significant research and engineering effort, which we summarize below.

Fully asynchronous training. Navigator rollouts often take more than 5 minutes due to multi-turn browser interactions, and exhibit high variance of task completion times due to different task difficulties. This leads to extremely inefficient synchronous RL, as training is often blocked by straggler rollouts. Additionally, scaling up the number of GPUs for synchronous training leads to under-utilization during the rollout phase.

This system results in asynchronous training runs being 3x faster than standard synchronous training runs, with even larger gains observed when scaling to additional GPUs.

RL algorithm. We trained n1 using GRPO¹⁵ with a group size of 16 trajectories per prompt. To address the off-policyness between rollout and training, we adopt Truncated Importance Sampling¹⁶ to reweigh each token, leading to the following policy gradient:

where $A_t$ is the normalized advantage within each group. We also employ dynamic sampling, as proposed by DAPO¹⁷, to reject prompts that are too easy or too hard. We use a discount factor of 0.995 when computing the return for each intermediate step to encourage shorter trajectories.

These engineering details are essential and easy to miss. If they are not properly handled, RL training could collapse, or lead to worse policies where nothing immediately appears to be wrong.

We plan on sharing more about our RL system and empirical findings in future blog posts. Stay tuned!

Epilogue

If you’ve made it to the end of this post, we suspect you will enjoy using Navigator via our API. Navigator already powers all traffic on Scouts (e.g. here's one that tracks browser and computer use APIs).

Sign up here to get access to the API.

Citation

@misc{yutori2025navigator,
  author       = {The Yutori Team},
  title        = {Introducing Navigator},
  howpublished = {\url{https://yutori.com/blog/introducing-navigator}},
  note         = {Yutori Blog},
  year         = {2025},
}