Introducing Smevals: A New Framework for Evaluating AI Models and Prompts
Smevals is a newly developed evaluation framework designed to assess the capabilities of various AI models, prompts, and harnesses. Developed by Jesse Vincent's Prime Radiant applied AI research lab, this tool allows users to create and run small evaluation suites against different model configurations and then grade the results. The framework aims to provide clear answers regarding the performance of diverse AI models.
Users can interact with Smevals through a command-line interface, starting with `uvx smevals docs` to access its documentation. Creating an eval suite involves organizing YAML files within a directory. Once an eval suite is set up, it can be executed against models using commands like `uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6`. The tool separates running evaluations from grading them, with grading performed via `uvx smevals grade path-to-eval/` against predefined checks.
To facilitate analysis, Smevals offers functionalities to serve results via a localhost web server or to generate static HTML reports using `uvx smevals build path-to-eval/`. The creator views Smevals as their third iteration on the concept of AI evaluation, expressing satisfaction with its current form and plans for future expansion and application to personal projects.
Smevals addresses a critical need for standardized and reproducible evaluation of large language models (LLMs) and their associated prompts. The development of such frameworks is essential for navigating the rapidly evolving landscape of generative AI, enabling researchers and developers to quantitatively compare model performance and identify areas for improvement. By separating execution, grading, and reporting, Smevals promotes a structured approach to model assessment. As AI systems become more integrated into various applications, robust evaluation tools like Smevals will be instrumental in fostering trust, ensuring reliability, and driving innovation by providing transparent insights into model behavior and limitations.
AI-generated to prompt reflection — not editorial opinion, not advice, not a statement of fact. How this works.