FACTS Benchmark Suite: Systematically evaluating the factuality of large language models

Google introduces the FACTS Benchmark Suite, a comprehensive evaluation framework for measuring factuality of large language models across four distinct benchmarks.

• •Parametric Benchmark: 2,104 items (public + private) testing models' ability to answer trivia-style factual questions using internal knowledge without external tools
• •Search Benchmark: 1,884 items testing models' ability to use a shared web search tool to retrieve and synthesize information, often requiring multi-step retrieval
• •Multimodal Benchmark: 1,522 items evaluating factually accurate text generation in response to image-based questions requiring visual grounding and parametric knowledge
• •Grounding Benchmark v2: an updated version of the original benchmark testing context-grounded answering
• •