Benchmarking Local AI Systems: Evaluating the Full Inference Stack

Selecting a local artificial intelligence model is often framed as a question of model performance: which model is best?

For practical deployment, that question is incomplete. A model does not operate independently of the system in which it is deployed. Its observed performance depends on the model architecture and parameters, but also on quantization, inference software, context configuration, hardware, prompt and tool interfaces, and the application logic surrounding the model.

For this reason, CT Cubed evaluates local AI as a system rather than benchmarking model weights in isolation. The relevant unit of measurement is the complete inference stack as it will be configured for operational use.

Several components materially affect that stack:

• Model parameters: The learned numerical parameters produced during model training. Model size is commonly expressed in billions of parameters, such as a 30-billion-parameter, or 30B, model.
• Quantization: The representation of model parameters at reduced numerical precision. Quantization can substantially reduce memory requirements and computational cost, although it may also affect model quality.
• Inference engine: The software responsible for executing model inference, such as llama.cpp or MLX.
• Context length: The maximum number of tokens available to the model for input, intermediate context, and generated output.
• Inference server: The software and hardware environment that exposes the model to an application, such as Ollama or LM Studio running on Apple silicon or NVIDIA GPUs.
• Application harness: The software surrounding the model that constructs prompts, exposes tools, maintains state, processes responses, and controls what actions the model is permitted to perform.

Changes to any of these components can affect throughput, memory utilization, tool-use behavior, response length, and task performance. Consequently, identifying a model by name and parameter count alone provides an incomplete characterization of its operational behavior.

Measuring the System

To evaluate these differences systematically, we developed Yardstick, an internal benchmarking framework for local AI deployments.

Yardstick evaluates a defined combination of model, quantization, inference environment, API configuration, context length, and hardware. The objective is to characterize the system that an application will actually use rather than infer performance from published model benchmarks.

The test suite measures several categories of performance, including:

• inference throughput;
• memory utilization;
• basic and multi-step tool use;
• code-related tasks;
• image-processing capabilities where supported;
• cybersecurity-domain knowledge;
• appropriate refusal behavior; and
• performance on representative application tasks.

Tool use is particularly important for agentic applications. A model that can generate an appropriate tool call in a simple test may not necessarily select, sequence, and interpret tools correctly during a multi-step task.

Safety behavior also requires more than determining whether a model will refuse a request. A useful model must distinguish between legitimate technical activity and requests that present unacceptable risk. Excessive compliance is undesirable, but excessive refusal can also make a model unsuitable for legitimate security engineering and research applications.

For that reason, Yardstick evaluates both inappropriate compliance and unnecessary refusal.

Runtime and API Independence

Yardstick was designed to minimize dependence on a specific model vendor, inference runtime, or API implementation.

Models are accessed through standardized API interfaces where possible, with alternate message formats supported when required by a particular server. This allows the same test methodology to be applied across environments such as Ollama, LM Studio, MLX-based servers, and other compatible inference platforms.

This separation is important because the local AI ecosystem changes rapidly. A benchmark tied directly to a single runtime provides limited value when the underlying serving infrastructure changes.

Maintaining independence between the benchmark and the inference environment also allows results from different configurations to be compared using a common methodology.

Experimental Controls

Benchmark results are only meaningful when the conditions under which they were collected are documented and controlled. Yardstick therefore records the configuration associated with each test, including the model, runtime, hardware, context configuration, and other available parameters. If a configuration value cannot be verified, it is recorded as unknown rather than inferred from another system. Comparisons are made between systems using the same test battery and, where the experiment requires it, the same hardware platform. The framework can execute test matrices across multiple models, quantizations, and context lengths without manual intervention, and test state is preserved so that interrupted experiments can resume without discarding completed results.

Scoring methods depend on the nature of the test. Objective results can often be evaluated programmatically, while more subjective tasks, such as determining whether an answer correctly addresses a cybersecurity question, are evaluated against a fixed rubric using a separate judge model. The purpose is not to eliminate judgment from the evaluation process, but to apply that judgment consistently across repeated experiments.

Example: Cubernaut API Enumeration

One application of this methodology involved Cubernaut, CT Cubed's locally hosted AI-assisted penetration-testing platform.

The experiment examined a specific task: identifying all API endpoints exposed by a target application.

The target contained 14 known endpoints, providing a defined ground truth against which each run could be evaluated. Cubernaut also had access to APIRecon, CT Cubed's API-enumeration tool, allowing the experiment to measure not only whether the model reached the correct result but also how it used the available tooling.

Three configurations were evaluated:

1. Qwen3 27B dense model using Ollama;
2. Qwen3 35B mixture-of-experts model using Ollama; and
3. Qwen3 27B dense model using LM Studio with MLX.

Each configuration was tested 10 times.

All tests were conducted on the same MacBook Pro equipped with an Apple M5 Max processor and 64 GB of unified memory. Holding the hardware constant removed one significant source of variability from the comparison.

All three configurations ultimately identified all 14 endpoints. Because task completion was equivalent, the experiment provided an opportunity to examine differences in efficiency and tool-use behavior.

Model Comparison Under the Same Runtime

The first comparison held the runtime and hardware constant while changing the model.

Under this configuration, the 35B mixture-of-experts model completed the task substantially faster while generating fewer tokens and making fewer tool calls.

Although the model has a larger total parameter count, mixture-of-experts architectures activate only a subset of their parameters during a given inference operation. Total parameter count therefore does not correspond directly to inference cost in the same way it does for a dense architecture.

For this experiment, changing the model while holding the runtime and hardware constant produced a substantial difference in efficiency.

Same Model Under Different Inference Environments

The second comparison held the Qwen3 27B dense model and hardware constant while changing the inference environment.

Both configurations achieved the same final task result, but their execution characteristics differed considerably.

In this experiment, the Ollama configuration generated tokens at a higher rate, required fewer tool interactions, and completed the task in less time. The LM Studio/MLX configuration generated substantially more tokens and used APIRecon in only five of the ten runs, despite the prompt explicitly identifying the tool. In the remaining runs, the model instead constructed its own HTTP requests using tools such as curl.

These observations are important because the underlying model remained nominally the same. However, they should not be interpreted as evidence that the runtime alone caused every observed difference. Inference configuration, model packaging, quantization, prompt templates, structured tool-call handling, and related implementation details can also influence model behavior.

From an operational perspective, that distinction reinforces the larger point: the complete deployed configuration must be measured.

Why System-Level Evaluation Matters

Model-selection decisions for local AI systems should be based on the behavior of the configuration that will actually be deployed.

A published model benchmark may provide useful information about general capability, but it does not establish how that model will perform on a specific hardware platform, under a particular quantization, through a particular inference server, or inside an application that expects reliable multi-step tool use.

For operational applications, those differences are consequential.

CT Cubed's Cubernaut and Terrain Trace platforms both use locally hosted AI models so that processing can remain within the operator's environment. Selecting models for these systems therefore requires more than identifying the model with the highest general benchmark score.

We evaluate candidate configurations against representative workloads and measure their behavior under the conditions in which they will be used.

The objective is not to identify a universally superior local AI model.

It is to identify the model, runtime, configuration, and hardware combination that provides the required capability for a specific application, and to establish that conclusion through measurement rather than assumption.