Building a Local AI Coding Workstation with Qwen3.8-27B and llama.cpp

Running a local coding assistant sounds simple until model quality, VRAM limits, latency, and container configuration start competing with one another. I built this workstation to find a practical balance between those constraints—not to create a benchmark machine, but to make everyday coding assistance predictable.

The goal is not to promise a universal performance profile. The numbers below are measurements from one machine and one workload. Reproduce them with the same model file, runtime revision, drivers, prompts, and sampling settings before drawing broader conclusions.

Publication note: This configuration targets a native Linux host with NVIDIA GPUs. Docker Desktop on macOS is not a substitute for Linux NVIDIA GPU passthrough.

Follow-up (August 28, 2026): After publishing this post, I ran a multi-stage optimization and benchmark pass on the same workstation. The original measurements below remain the baseline; the follow-up compares asymmetric KV-cache quantization, a host-specific llama.cpp rebuild, native MTP speculation, a 3-bit quantization, and two different serving modes.

Who this is for

#

This is for developers who enjoy operating their own inference stack and want to understand the trade-offs behind a local coding setup. If the goal is simply to get productive quickly, a managed service or desktop application is probably the better starting point.

Results at a glance

#

• Original baseline: Qwen3.8-27B UD-Q4_K_XL at 196,608 tokens, generating at roughly 17.3–17.5 tokens per second.
• Best single-stream result: UD-Q3_K_XL with native MTP at 34.57 tokens per second cold and 32.79 tokens per second warm at 196,608 tokens.
• Maximum tested context: 262,144 tokens, the model’s native context length, at 32.70 tokens per second in single-stream mode.
• Best aggregate result: two approximately 128K streams at 25.61 and 24.47 tokens per second, or 48.95 tokens per second combined.
• Main lesson: quantization, cache representation, build configuration, and speculative decoding interact. There is no single “fastest” setting independent of workload and quality requirements.

The workstation

#

The system is built around a six-core AMD Ryzen 5 9600X, 60 GB of DDR5 memory, two 16 GB GeForce RTX 5060 Ti cards, and dedicated NVMe storage for model files. The RTX 5060 Ti is part of NVIDIA’s Blackwell generation and has compute capability 12.0 (sm_120). NVIDIA also sells an 8 GB version, so the 16 GB specification matters here.

The CPU supports AVX-512, but support alone does not prove that a particular workload benefits from a specific vector instruction. Any performance attribution to AVX-512 should be treated as a hypothesis unless it is supported by profiling or an A/B build comparison.

Model choice

#

The model used in this workstation is Qwen3.8-27B, downloaded as an Unsloth GGUF quantization. Qwen3.8-27B is a multimodal model with a vision encoder, 64 language-model layers, a hybrid architecture, and a native context length of 262,144 tokens. The runtime configuration below selects a 196,608-token context, which is a lower limit than the model’s native maximum.

The quantization repository is not the official Qwen repository; it is an Unsloth distribution of GGUF files. Record the exact model URL, filename, and SHA-256 digest when reproducing the setup. A Q4 label is an approximate storage category: GGUF block scales, metadata, and mixed tensor precisions mean the actual file size is not exactly four bits per parameter.

The follow-up also compared the 17.56 GB UD-Q4_K_XL file with the 12.24 GB UD-Q3_K_XL file. The smaller quantization made the 262,144-token configuration and two-slot configuration practical on this hardware, but the throughput and memory gains do not establish that Q3 has identical output quality. Validate coding accuracy, tool use, and long-context recall on the workloads that matter to you.

The vision projector can be supplied with --mmproj, but multimodal behavior depends on the exact projector, llama.cpp revision, chat template, and client. Test an image request rather than assuming that every OpenAI-compatible client will handle vision input or reasoning fields identically.

Memory planning: measure, do not guess

#

Two 16 GB cards provide 32 GB of nominal VRAM, but that is not the usable capacity. Model weights, runtime buffers, activations, KV or recurrent state, CUDA allocations, and allocator fragmentation all compete for memory.

An early estimate used this calculation:

$$196{,}608 \times 139\text{ KB/token} = 13.50\text{ GiB}$$

That arithmetic is not valid: using 139 KiB per token would produce approximately 26.1 GiB, not 13.5 GiB. More importantly, Qwen3.8 uses a hybrid architecture, so a generic standard-attention KV formula is not a reliable description of every cached state. The correct way to report this value is to capture llama.cpp’s startup memory report and model metadata for the exact build and command line.

This is the part of local AI that is easy to underestimate: a configuration can appear to fit on paper and still fail during startup because runtime buffers, cache state, and allocator behavior consume the remaining memory. Report the workload, maximum context, number of requests, cold-start behavior, and observed per-device allocations.

Building llama.cpp with CUDA

#

The official llama.cpp documentation supports a CUDA build with GGML_CUDA=ON and allows explicit CUDA architecture selection. For this workstation, 120 is the relevant architecture. The build should be pinned to a known commit instead of cloning the moving master branch.

A simplified builder example is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

FROM nvidia/cuda:12.8.1-devel-ubuntu24.04 AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential ca-certificates cmake git \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /src
RUN git clone https://github.com/ggml-org/llama.cpp.git . \
    && git checkout <tested-llama-cpp-commit>

RUN cmake -S . -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=120 \
    && cmake --build build --config Release -j12 --target llama-server

FROM nvidia/cuda:12.8.1-runtime-ubuntu24.04

WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates libgomp1 \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /src/build/bin/llama-server /app/llama-server
ENTRYPOINT ["/app/llama-server"]

The CUDA version must be documented consistently. The example uses CUDA 12.8.1; it should not be described elsewhere as a CUDA 13 build. -march=x86-64-v4 is a portable AVX-512 baseline, not an exact Zen 5 target. If a compiler-specific znver5 target is used, record the compiler and verify that the generated binary is only deployed to compatible hosts.

CUDA Graphs and other tuning options can be useful, but they should be enabled only after confirming that the selected llama.cpp revision supports them and after measuring an A/B difference. A custom image should also be tested from a clean host, including llama-server --version, device discovery, model loading, and an inference request.

Baseline Compose service

#

Model files should remain outside the container image and be mounted read-only. The following example keeps the endpoint local to the host and requires an API key. Use a firewall or a reverse proxy if the service must be accessed from another machine.

This is the original Q4/196K configuration. The optimized follow-up configuration appears later in the post.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

services:
  qwen-coder-workstation:
    image: llama.cpp:native
    container_name: qwen3.8-27b-native
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - /data/models:/models:ro
    command: >
      --model /models/Qwen3.8-27B-UD-Q4_K_XL.gguf
      --mmproj /models/mmproj-BF16.gguf
      --ctx-size 196608
      --n-gpu-layers 999
      --split-mode layer
      --host 0.0.0.0
      --port 8000
      --api-key ${LLAMA_API_KEY}
      --metrics
      --flash-attn on
      --parallel 1
      --cache-type-k q8_0
      --cache-type-v q8_0
      --batch-size 2048
      --ubatch-size 512
      --threads 6
      --threads-batch 12
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Flag names can change between llama.cpp revisions. Validate every option against the pinned server binary. In particular, do not assume that an MTP or n-gram speculative-decoding configuration documented for another serving engine has equivalent support in the selected llama.cpp build.

--parallel 1 configures a single concurrent sequence, which is appropriate for one interactive coding session. It is not a multi-user serving configuration. The service also listens on 0.0.0.0 inside the container, but the port binding above keeps it reachable only through localhost on the host.

Layer splitting and speculative decoding

#

Layer splitting can place different model layers on different GPUs and may be a better fit for this memory-constrained workstation than tensor parallelism. The actual device placement and transfer behavior should be taken from llama.cpp startup logs; avoid claiming that communication happens only once per forward pass or that a particular split is always optimal.

KV-cache quantization reduces memory compared with higher-precision cache storage, but it can involve quality and performance trade-offs. It does not prevent general long-context degradation. Compare q8_0 with a higher-precision cache when both configurations fit.

Speculative decoding can improve decode throughput when proposed tokens are accepted frequently enough to offset draft-generation overhead. N-gram speculation and model-based MTP are different techniques. The original workstation notes reported that an n-gram configuration did not create a measurable additional allocation in one run; that should be reported as an observation, not as a universal zero-VRAM guarantee.

Benchmarking the configuration

#

The original measurements were:

These numbers are useful as a starting point, but they are not a general guarantee. A reproducible benchmark should include:

• model filename and SHA-256 digest;
• llama.cpp commit, CUDA toolkit, driver, and container image;
• GPU clocks, power state, and temperature;
• context and prompt token counts;
• generated token count, seed, temperature, and sampling settings;
• warm-up policy and number of runs;
• separate prompt-processing and generation measurements;
• n-gram disabled versus enabled; and
• startup memory logs and /metrics output.

For example, the environment inventory can begin with:

1
2
3
4

llama-server --version
llama-server --list-devices
nvidia-smi
sha256sum /data/models/*.gguf

Report medians and percentiles where possible. Define draft acceptance using the server’s raw metrics rather than inferring it from a single token count. Likewise, a prefix-cache observation should identify the actual metric and calculation; it should not be relabeled as a universal “100% cache hit.”

Follow-up: optimizing the same workstation

#

The first version of this post established a useful baseline: the Q4 model fit across both 16 GB GPUs and could serve a single long-context coding session. The next question was whether the remaining memory and the runtime itself could be used more effectively.

I evaluated each stage on the live workstation with a standardized 600-token completion and deeper prompts, capturing llama.cpp timing output, raw token counters, and nvidia-smi telemetry. These are comparative measurements from one host, not a general benchmark of Qwen3.8 or llama.cpp. Several variables changed between stages, so the matrix shows an optimization path rather than a controlled experiment that attributes every percentage point to one flag.

Benchmark matrix

#

The focused comparison also measured the Native-v2 Q4 build at 211.6 prompt tokens per second, compared with 160.0 tokens per second for the baseline used in that run. The original post records a wider 167–241 tokens-per-second range across its earlier runs, so the prefill result should be treated as workload-specific rather than as a universal improvement factor.

What produced the improvement?

#

1. Asymmetric KV-cache quantization

#

The first low-risk change was to use a higher-precision type for keys and a smaller type for values:

1
2

--cache-type-k q8_0
--cache-type-v q4_0

At the same 196,608-token context, this reduced the observed base allocation from 27.70 GB to 24.99 GB, freeing about 2.71 GB without changing the measured 17.47-token-per-second generation rate. The run did not show an obvious quality regression, but “no regression observed” is not the same as a formal quality evaluation. The trade-off should be checked against representative code-generation and long-context tasks.

The server exposes separate cache-type controls for K and V, and the selected values must be supported by the CUDA FlashAttention build. The llama.cpp server options and CUDA build options are the source of truth for the exact revision being used.

2. A workstation-specific llama.cpp rebuild

#

The native-v2 image was compiled with CUDA architecture 120 for the RTX 5060 Ti, enabled all FlashAttention KV-cache quantization combinations, disabled CUDA peer copies for this topology, and used native host CPU optimization. The relevant build options are:

1
2
3
4
5
6
7

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=120 \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_CUDA_NO_PEER_COPY=ON \
  -DGGML_NATIVE=ON

The rebuild coincided with a rise from roughly 17.4 to 21.99–23.12 generation tokens per second, a prompt-evaluation result of 211.6 tokens per second in the focused comparison, and a reduction in model load time from 19.89 to 12.88 seconds. Those numbers are valuable operational evidence, but the rebuild changed several variables together. They do not prove that Blackwell code generation, CPU vectorization, or peer-copy behavior independently caused a particular share of the gain. Pin the llama.cpp commit and compare one build option at a time before generalizing the result.

3. Native MTP at the full 196K context

#

With the Q4 model, native multi-token prediction (MTP) and compressed draft-cache types made speculative decoding practical at the full 196,608-token context:

1
2
3
4

--spec-type draft-mtp
--spec-draft-n-max 2
--spec-draft-type-k q4_0
--spec-draft-type-v q4_0

The run accepted 58.9% of drafted tokens and produced 25.39 tokens per second, reducing the 600-token completion from 35.30 seconds to 23.58 seconds. The important detail is that MTP uses the model’s own prediction heads; it is not interchangeable with an external draft model. Acceptance rate and net throughput still depend on the prompt, sampler, model revision, and server build.

I also tested cascading MTP with ngram-mod. It produced longer bursts on repetitive boilerplate, JSON, and templating, but the measured result was slower than pure MTP: 22.97 versus 25.39 tokens per second, with 46.2% versus 58.9% draft acceptance. For novel coding and reasoning, pure MTP was the better setting in this test; the hybrid mode remains interesting for highly repetitive output.

4. Moving from Q4 to Q3

#

The largest practical improvement came from changing to UD-Q3_K_XL. The model file fell from 17.56 GB to 12.24 GB, and the 196,608-token MTP run reached 34.57 tokens per second cold and 32.79 tokens per second warm. The 600-token completion took 17.92 seconds, and total observed VRAM fell to 23.51 GB.

That is approximately twice the original baseline decode rate while leaving about 9.11 GB of total VRAM headroom. It is also the point where the setup becomes more flexible: the freed memory can be spent on context length, concurrent slots, or simply a larger safety margin. The cost is that a lower-bit quantization is another quality variable, so I would choose it after validating output quality rather than treating the speedup as free.

Two useful serving modes

#

The Q3 result supports two different workstation designs. They optimize for different user experiences, and the “best” mode depends on whether one request needs the largest possible context or whether two requests should make progress at the same time.

With two slots, the shared 262,144-token context budget works out to approximately 131,072 tokens—roughly 128K—for each stream. The single-stream mode preserves the model’s full native 262,144-token window. In both cases, the numbers are aggregate or per-stream measurements from this host; they are not promises about latency under arbitrary concurrent workloads.

Optimized production configuration

#

The following is the active single-stream configuration from the follow-up. It uses Q3, asymmetric KV caches, native MTP, the full native context length, and compressed draft state. The model and projector remain host-mounted and read-only. Bind the host port to localhost unless the service is deliberately protected by authentication and a firewall.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

services:
  qwen3.8-27b:
    image: llama.cpp:native-v2
    container_name: qwen3.8-27b-native
    restart: unless-stopped
    ports:
      - "127.0.0.1:8002:8000"
    volumes:
      - /media/chris/games-media/qwen3.8-27b-q3:/models:ro
    command: >
      --model /models/Qwen3.8-27B-UD-Q3_K_XL.gguf
      --mmproj /models/mmproj-BF16.gguf
      --image-min-tokens 1024
      --ctx-size 262144
      --parallel 1
      --n-gpu-layers 999
      --host 0.0.0.0
      --port 8000
      --cont-batching
      --flash-attn on
      --metrics
      --cache-type-k q8_0
      --cache-type-v q4_0
      --spec-type draft-mtp
      --spec-draft-n-max 2
      --spec-draft-type-k q4_0
      --spec-draft-type-v q4_0
      --reasoning-preserve
      --batch-size 4096
      --ubatch-size 1024
      --threads 6
      --threads-batch 12
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

For the dual-stream mode, change --parallel 1 to --parallel 2 while retaining the total --ctx-size 262144. Recheck the startup report after every change: memory placement, slot sizing, and available flags can change with the llama.cpp revision.

Operational isolation

#

The workstation is reserved for interactive IDE assistants and agentic development. Batch jobs run elsewhere so that prompt processing and generation do not compete with unrelated workloads. This is an operational choice, not a property of the model or runtime, and it is one of the most practical ways to make latency predictable.

References

#

• Qwen3.8-27B model card
• Unsloth Qwen3.8 GGUF repository
• NVIDIA GeForce RTX 5060 Family
• NVIDIA CUDA GPU compute capabilities
• llama.cpp build documentation
• llama.cpp server documentation
• llama.cpp Docker documentation
• NVIDIA Container Toolkit
• Docker Compose GPU support

This post documents a specific workstation experiment. Hardware, model revisions, runtime flags, and benchmark results should be revalidated before replication.