LLMs Struggle with Secure PowerShell Script Generation, New Benchmark Shows

Global: Evaluation of LLMs for Secure PowerShell Script Generation

Researchers from an unnamed institution released a study on January 2026 that introduces a new benchmark, SecGenEval-PS, to assess large language models’ ability to generate secure PowerShell scripts, analyze vulnerabilities, and perform automated repairs. The work evaluates both proprietary and open-source models, highlighting the security challenges associated with scripting languages that often run with elevated privileges.

Benchmark Design

SecGenEval-PS comprises three task categories: secure script generation, security analysis of existing scripts, and automated repair of identified violations. The benchmark systematically measures model performance against these criteria, providing a structured framework for future research.

Baseline Model Performance

Initial experiments reveal that more than 60% of PowerShell scripts produced by leading models such as GPT‑4o and o3‑mini contain security flaws when no structured guidance is provided. The findings suggest that current LLMs, despite strong code generation capabilities in languages like Python and JavaScript, are not yet reliable for secure scripting in PowerShell.

PSSec Framework Introduction

To address the gap, the authors propose PSSec, a framework that combines data synthesis with targeted fine‑tuning. Central to PSSec is a self‑debugging agent that leverages static analysis tools together with the reasoning abilities of advanced LLMs to generate large‑scale triplets of insecure scripts, detailed violation analyses, and corresponding repairs.

Fine‑Tuning Methodology

The generated dataset is used to fine‑tune lightweight models—some as small as 1.7 billion parameters—through supervised fine‑tuning (SFT) and reinforcement learning (RL). This process equips the models with security‑aware reasoning, enabling them to produce safer PowerShell code while maintaining overall code quality.

Results and Efficiency Gains

Across multiple LLM families, including GPT and Qwen, models trained with PSSec match or exceed the performance of general‑purpose large models on the benchmark tasks. Notably, the fine‑tuned models achieve these results while reducing inference costs by more than an order of magnitude.

Implications

The study underscores the need for specialized security training when deploying LLMs for scripting environments and demonstrates that targeted fine‑tuning can deliver both safety and cost efficiency. The authors suggest that future work will explore broader scripting languages and integrate additional static analysis techniques.

This report is based on information from arXiv, licensed under Academic Preprint / Open Access. Based on the abstract of the research paper. Full text available via ArXiv.