Logic Sketch Prompting Improves Accuracy and Consistency Across Leading Open-Weight LLMs

Global: Logic Sketch Prompting Improves Accuracy and Consistency Across Leading Open-Weight LLMs

Researchers who posted a new preprint on arXiv in December 2025 introduced a lightweight prompting framework called Logic Sketch Prompting (LSP) to address the unreliability of large language models (LLMs) on tasks that demand strict rule adherence, determinism, and auditability. The study evaluated LSP on two pharmacologic logic compliance tasks using three open-weight models—Gemma 2, Mistral, and Llama 3—aiming to determine whether the approach could deliver traceable and repeatable outputs without sacrificing performance.

Prompting Framework Overview

LSP incorporates typed variables, deterministic condition evaluators, and a rule‑based validator. These components work together to enforce logical constraints during generation, producing outputs that can be audited step‑by‑step. The framework is designed to be compatible with existing LLMs without requiring model fine‑tuning.

Benchmark Tasks

The authors selected two pharmacologic logic compliance tasks that simulate decision‑support scenarios common in clinical and regulated environments. Each task required the model to apply a series of explicit rules to determine the correct outcome, providing a rigorous test of deterministic reasoning.

Comparative Performance

Across both tasks and all three models, LSP achieved the highest accuracy, ranging from 0.83 to 0.89, and an identical F1‑score range of 0.83 to 0.89. By contrast, zero‑shot prompting recorded accuracy between 0.24 and 0.60, concise prompting between 0.16 and 0.30, and chain‑of‑thought prompting between 0.56 and 0.75. These figures illustrate a substantial performance gap in favor of LSP.

Statistical Significance

McNemar tests conducted on the results indicated statistically significant improvements for LSP over nearly all alternative prompting strategies, with p‑values less than 0.01 in most pairwise comparisons.

Implications for Regulated Domains

The findings suggest that LSP can enhance determinism, interpretability, and consistency—attributes essential for clinical decision support, regulatory compliance, and other safety‑critical applications where auditability is paramount.

Future Directions

The authors propose extending LSP to additional domains and exploring integration with larger, closed‑source models to assess scalability. Further research may also examine how the framework interacts with emerging alignment techniques.

This report is based on information from arXiv, licensed under Academic Preprint / Open Access. Based on the abstract of the research paper. Full text available via ArXiv.