Open WebUI unifies chat, model management, file handling, web search, tool calling, image input, knowledge retrieval and API access within a single local AI interface. This hands-on review focuses on its Windows desktop build, using identical hardware and model files from our earlier Jan v0.8.6 testing to enable direct side-by-side comparison.

This is the second entry in our local LLM tools series, where we publish repeatable performance data from fixed test hardware across evaluated platforms.

Open WebUI is open-source and self-hosted, delivering a browser-native interface for local and remote AI models styled after mainstream commercial chat services. It connects to diverse inference backends and OpenAI-compatible endpoints, while its desktop package runs GGUF models locally via bundled llama.cpp. This broad feature set extends well beyond basic local chat once tools, presets, search, file support and API access are activated.

Our test rig reused the HP Z4 Rack G5 workstation, NVIDIA RTX 5000 Ada GPU and two Jan GGUF models from the prior Jan review. We locked in an 8,192-token context, 2,048 batch size and 100 GPU layers, matched sampling parameters for text benchmarks, and also validated presets, multimodal input, DDGS web search, URL retrieval and the local API.

Getting Open WebUI stable required far more effort than expected, mostly spent troubleshooting local llama.cpp and CUDA before benchmarking. We overcame broken default runtime downloads, missing CUDA binaries, excessive context allocation and several built-in tooling bugs.

Test System

Configuration matched our Jan testbed for comparable results; versioning for the desktop server, Python and llama.cpp was logged separately.

Installing Open WebUI Desktop

The Windows installer includes an environment panel for local llama.cpp deployment, Open Terminal toggle and custom install paths. We disabled Open Terminal, enabled llama.cpp and kept the default user-profile directory.

After Python and bundled packages finished installing, we selected the local backend and created an admin account before entering the main chat view. A notable quirk: the desktop wrapper and core Open WebUI server carry separate version numbers, with the server hosted under the user roaming folder within the bundled Python environment.

Adding Models and Configuring llama.cpp

Model files and inference runtime are decoupled: the Models page manages GGUFs, while the Inference Runtime section controls llama.cpp. We imported Jan V3.5 4B Q4_K_XL, Jan V2 VL High Q4_K_M and the V2 multimodal projection file; they appeared in the selector once the runtime was active.

The first installation failure occurred when pulling the “latest” llama.cpp release with CUDA 12.4 selected, as the target CUDA archive did not exist for that tagged version. Pinning build b11612 fixed the download issue, but llama.cpp still failed to launch without required CUDA runtime DLLs. We manually placed cudart64_12.dll, cublas64_12.dll and cublasLt64_12.dll into the build folder; llama-server then started normally on port 18881.

Model Settings and GPU Use

Settings are split between the desktop client and web UI, creating an initial learning curve. The desktop manages GGUF assets and llama.cpp runtime, while the web interface handles per-chat sampling, workspace presets, capabilities, tools, search and server administration.

Benchmark runs used 8,192-token context, 2,048 batch size, 100 GPU layers and disabled prompt caching. We aligned sampling parameters to our Jan test profile: temperature 0.8, Top K 40, Top P 0.95, Min P 0.05, frequency penalty 0, presence penalty 0, repeat last N 64 and repeat penalty 1.0.

We built dedicated workspace presets for benchmarking and disabled built-in tools. Unmodified model sessions automatically injected tool instructions into prompts; this extra context pushed the V2 VL request past the 8,192-token window and invalidated tests.

Local Generation Performance

The benchmark prompt was unchanged: “Explain RAID 5 and RAID 6 to someone building their first NAS. Include the main advantages and disadvantages of each, in about 400 words.” Web search stayed off, we discarded the first model load as warm-up, then captured three fresh chat runs per model.

Jan V3.5 4B hit 115.32 tok/s, while Jan V2 VL High reached 75.06 tok/s with far longer output driving higher latency. Against our prior Jan v0.8.6 results on identical hardware, Open WebUI was ~6.2% slower on V3.5 4B and ~8.5% slower on V2 VL High. The two applications used different llama.cpp builds, so this gap reflects full-stack behavior rather than raw llama.cpp performance.

Local Chat and Workspace Presets

Once runtime setup concluded, chat operation was smooth, and workspace presets became our most-used feature. A single GGUF can be reused across presets with independent system prompts, tool access, knowledge rules and generation parameters, avoiding duplicate model storage.

Presets resolved our benchmark context overflow. Raw V2 VL enabled automatic knowledge-file tool calls that bloated the prompt; a benchmark preset with all built-in tools disabled trimmed the prompt and stabilized testing. For regular use, presets can also activate web search and file tools for one profile while keeping another for plain chat.

Multimodal Input

Jan V2 VL High automatically loaded its multimodal projection file. We reused the nvidia-smi screenshot from our Jan test and queried for GPU model, driver, VRAM, utilization, temperature and power draw. The model returned all values accurately, passing our qualitative vision test.

Web Search

Web search is configured in admin settings, with provider, result count, URL fetch and SSL controls. We chose DDGS to skip extra API keys and built a dedicated Jan V3.5 4B search-enabled preset.

Initial tests revealed limitations for this small model. It successfully triggered search_web but would not fetch the returned GitHub page and relied on stale internal knowledge. Explicitly instructing the model to run fetch_url fixed the workflow, though fetched content ballooned the prompt to 17,689 tokens and overflowed our context limit. Capping fetched URL length at 8,000 characters kept prompts within bounds, and the model then correctly identified Open WebUI v0.12.2 from the release page.

Local API

API validation proved simpler. We enabled API authentication and queried the model endpoint from PowerShell; the response listed raw GGUF files plus all custom workspace presets, confirming presets are API-accessible, not only chat-only. A subsequent chat completion POST against our benchmark preset returned a correct RAID comparison summary. A minor rendering quirk showed one Unicode dash garbled in the PowerShell console, but the model output itself was intact.

Issues Encountered

Windows CUDA local inference requires manual intervention. Default llama.cpp installation fails; users must pin a working build and manually supply CUDA DLLs. Reinstalling the runtime wiped custom arguments, reverting to oversized native context slots — V2 VL peaked at 169,984 tokens and V3.5 at 192,000, nearly exhausting the 32GB VRAM and slowing remote desktop until we restored the 8,192-token cap.

Tooling also created context bloat. Web search demanded explicit fetch instructions and content limits to avoid overflow. While resolvable via settings and logs, preparing Open WebUI for repeatable benchmarking took substantially longer than Jan on the same workstation.

Conclusion

We validated local text generation, multimodal vision, DDGS search, URL fetching, presets and authenticated API access. Its throughput trailed Jan modestly, but setup overhead was much higher due to manual llama.cpp and CUDA fixes.

For users wanting a unified local hub for models, tools, search, files, presets and API pipelines, Open WebUI efficiently reuses one GGUF across benchmark, search-enabled and API profiles without redundant copies. Once configured, it supports a richer local workflow ecosystem, despite the more involved Windows deployment.

These results join our Jan benchmark data in the local LLM tools series; we will continue testing additional platforms with this fixed workstation, models and prompt.

Beijing Qianxing Jietong Technology Co., Ltd.
Sandy Yang/Global Strategy Director
WhatsApp / WeChat: +86 13426366826
Email: yangyd@qianxingdata.com
Website: www.qianxingdata.com/www.storagesserver.com
Business Focus:
ICT Product Distribution/System Integration & Services/Infrastructure Solutions
With 20+ years of IT distribution experience, we partner with leading global brands to deliver reliable products and professional services.
“Using Technology to Build an Intelligent World”Your Trusted ICT Product Service Provider!