Running a local coding assistant sounds simple until model quality, VRAM limits, latency, and container configuration start competing with one another. I built this workstation to find a practical balance between those constraints—not to create a benchmark machine, but to make everyday coding assistance predictable.

The goal is not to promise a universal performance profile. The numbers below are measurements from one machine and one workload. Reproduce them with the same model file, runtime revision, drivers, prompts, and sampling settings before drawing broader conclusions.

Publication note: This configuration targets a native Linux host with NVIDIA GPUs. Docker Desktop on macOS is not a substitute for Linux NVIDIA GPU passthrough.

Who this is for

#

This is for developers who enjoy operating their own inference stack and want to understand the trade-offs behind a local coding setup. If the goal is simply to get productive quickly, a managed service or desktop application is probably the better starting point.

Results at a glance

#

• Model: Qwen3.8-27B with a 196K-token runtime context.
• Best reported generation: 22.2 tokens per second under the documented workload.
• Maximum tested context: 196,608 tokens.
• Main lesson: memory planning and reproducible measurements matter more than a single impressive setting.

The workstation

#

The system is built around a six-core AMD Ryzen 5 9600X, 60 GB of DDR5 memory, two 16 GB GeForce RTX 5060 Ti cards, and dedicated NVMe storage for model files. The RTX 5060 Ti is part of NVIDIA’s Blackwell generation and has compute capability 12.0 (sm_120). NVIDIA also sells an 8 GB version, so the 16 GB specification matters here.

The CPU supports AVX-512, but support alone does not prove that a particular workload benefits from a specific vector instruction. Any performance attribution to AVX-512 should be treated as a hypothesis unless it is supported by profiling or an A/B build comparison.

Model choice

#

The model used in this workstation is Qwen3.8-27B, downloaded as an Unsloth GGUF quantization. Qwen3.8-27B is a multimodal model with a vision encoder, 64 language-model layers, a hybrid architecture, and a native context length of 262,144 tokens. The runtime configuration below selects a 196,608-token context, which is a lower limit than the model’s native maximum.

The quantization repository is not the official Qwen repository; it is an Unsloth distribution of GGUF files. Record the exact model URL, filename, and SHA-256 digest when reproducing the setup. A Q4 label is an approximate storage category: GGUF block scales, metadata, and mixed tensor precisions mean the actual file size is not exactly four bits per parameter.

The vision projector can be supplied with --mmproj, but multimodal behavior depends on the exact projector, llama.cpp revision, chat template, and client. Test an image request rather than assuming that every OpenAI-compatible client will handle vision input or reasoning fields identically.

Memory planning: measure, do not guess

#

Two 16 GB cards provide 32 GB of nominal VRAM, but that is not the usable capacity. Model weights, runtime buffers, activations, KV or recurrent state, CUDA allocations, and allocator fragmentation all compete for memory.

An early estimate used this calculation:

$$196{,}608 \times 139\text{ KB/token} = 13.50\text{ GiB}$$

That arithmetic is not valid: using 139 KiB per token would produce approximately 26.1 GiB, not 13.5 GiB. More importantly, Qwen3.8 uses a hybrid architecture, so a generic standard-attention KV formula is not a reliable description of every cached state. The correct way to report this value is to capture llama.cpp’s startup memory report and model metadata for the exact build and command line.

This is the part of local AI that is easy to underestimate: a configuration can appear to fit on paper and still fail during startup because runtime buffers, cache state, and allocator behavior consume the remaining memory. Report the workload, maximum context, number of requests, cold-start behavior, and observed per-device allocations.

Building llama.cpp with CUDA

#

The official llama.cpp documentation supports a CUDA build with GGML_CUDA=ON and allows explicit CUDA architecture selection. For this workstation, 120 is the relevant architecture. The build should be pinned to a known commit instead of cloning the moving master branch.

A simplified builder example is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

FROM nvidia/cuda:12.8.1-devel-ubuntu24.04 AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential ca-certificates cmake git \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /src
RUN git clone https://github.com/ggml-org/llama.cpp.git . \
    && git checkout <tested-llama-cpp-commit>

RUN cmake -S . -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=120 \
    && cmake --build build --config Release -j12 --target llama-server

FROM nvidia/cuda:12.8.1-runtime-ubuntu24.04

WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates libgomp1 \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /src/build/bin/llama-server /app/llama-server
ENTRYPOINT ["/app/llama-server"]

The CUDA version must be documented consistently. The example uses CUDA 12.8.1; it should not be described elsewhere as a CUDA 13 build. -march=x86-64-v4 is a portable AVX-512 baseline, not an exact Zen 5 target. If a compiler-specific znver5 target is used, record the compiler and verify that the generated binary is only deployed to compatible hosts.

CUDA Graphs and other tuning options can be useful, but they should be enabled only after confirming that the selected llama.cpp revision supports them and after measuring an A/B difference. A custom image should also be tested from a clean host, including llama-server --version, device discovery, model loading, and an inference request.

Compose service

#

Model files should remain outside the container image and be mounted read-only. The following example keeps the endpoint local to the host and requires an API key. Use a firewall or a reverse proxy if the service must be accessed from another machine.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

services:
  qwen-coder-workstation:
    image: llama.cpp:native
    container_name: qwen3.8-27b-native
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - /data/models:/models:ro
    command: >
      --model /models/Qwen3.8-27B-UD-Q4_K_XL.gguf
      --mmproj /models/mmproj-BF16.gguf
      --ctx-size 196608
      --n-gpu-layers 999
      --split-mode layer
      --host 0.0.0.0
      --port 8000
      --api-key ${LLAMA_API_KEY}
      --metrics
      --flash-attn on
      --parallel 1
      --cache-type-k q8_0
      --cache-type-v q8_0
      --batch-size 2048
      --ubatch-size 512
      --threads 6
      --threads-batch 12
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Flag names can change between llama.cpp revisions. Validate every option against the pinned server binary. In particular, do not assume that an MTP or n-gram speculative-decoding configuration documented for another serving engine has equivalent support in the selected llama.cpp build.

--parallel 1 configures a single concurrent sequence, which is appropriate for one interactive coding session. It is not a multi-user serving configuration. The service also listens on 0.0.0.0 inside the container, but the port binding above keeps it reachable only through localhost on the host.

Layer splitting and speculative decoding

#

Layer splitting can place different model layers on different GPUs and may be a better fit for this memory-constrained workstation than tensor parallelism. The actual device placement and transfer behavior should be taken from llama.cpp startup logs; avoid claiming that communication happens only once per forward pass or that a particular split is always optimal.

KV-cache quantization reduces memory compared with higher-precision cache storage, but it can involve quality and performance trade-offs. It does not prevent general long-context degradation. Compare q8_0 with a higher-precision cache when both configurations fit.

Speculative decoding can improve decode throughput when proposed tokens are accepted frequently enough to offset draft-generation overhead. N-gram speculation and model-based MTP are different techniques. The original workstation notes reported that an n-gram configuration did not create a measurable additional allocation in one run; that should be reported as an observation, not as a universal zero-VRAM guarantee.

Benchmarking the configuration

#

The original measurements were:

These numbers are useful as a starting point, but they are not a general guarantee. A reproducible benchmark should include:

• model filename and SHA-256 digest;
• llama.cpp commit, CUDA toolkit, driver, and container image;
• GPU clocks, power state, and temperature;
• context and prompt token counts;
• generated token count, seed, temperature, and sampling settings;
• warm-up policy and number of runs;
• separate prompt-processing and generation measurements;
• n-gram disabled versus enabled; and
• startup memory logs and /metrics output.

For example, the environment inventory can begin with:

1
2
3
4

llama-server --version
llama-server --list-devices
nvidia-smi
sha256sum /data/models/*.gguf

Report medians and percentiles where possible. Define draft acceptance using the server’s raw metrics rather than inferring it from a single token count. Likewise, a prefix-cache observation should identify the actual metric and calculation; it should not be relabeled as a universal “100% cache hit.”

Operational isolation

#

The workstation is reserved for interactive IDE assistants and agentic development. Batch jobs run elsewhere so that prompt processing and generation do not compete with unrelated workloads. This is an operational choice, not a property of the model or runtime, and it is one of the most practical ways to make latency predictable.

References

#

• Qwen3.8-27B model card
• Unsloth Qwen3.8 GGUF repository
• NVIDIA GeForce RTX 5060 Family
• NVIDIA CUDA GPU compute capabilities
• llama.cpp build documentation
• llama.cpp server documentation
• llama.cpp Docker documentation
• NVIDIA Container Toolkit
• Docker Compose GPU support

This post documents a specific workstation experiment. Hardware, model revisions, runtime flags, and benchmark results should be revalidated before replication.