Large mixture-of-experts (MoE) models are usually associated with datacenter hardware. I had the opposite question: how far could an experimental Qwen3.8 Flash-Next checkpoint be pushed on a Proxmox host with two consumer GPUs, legacy dual-socket Xeon processors, and 96 GB of system memory?

The answer is surprisingly encouraging: the hardware can run the model, although not at datacenter speed and not without compromise. The interesting part is understanding which compromises help and which ones merely move the bottleneck somewhere else. This is an empirical experiment built on an open, unmerged llama.cpp pull request—not a turnkey production recipe.

Important: The model, GGUF files, and llama.cpp support are moving targets. Pin the exact model files and source commit before attempting to reproduce these results.

Who this is for

#

This is for engineers who want a practical lab for exploring model architecture, memory placement, and inference bottlenecks. If the goal is reliable production serving, start with a released runtime and hardware designed for the model’s memory requirements.

Why try this?

#

The host hardware was already available, and Proxmox provided a convenient boundary for isolating the experiment. The goal was not to make legacy hardware behave like a datacenter server. It was to learn whether the model’s hybrid architecture could make a very large checkpoint useful under severe memory pressure.

The model and the constraint

#

The checkpoint is Qwen3.8 Flash-Next, an experimental model using the qwen4_exp architecture. The official model accounting describes approximately 125B language-model parameters, 51B n-gram embedding parameters, and 4B multi-token-prediction parameters. The Unsloth GGUF repository labels the quantized model as approximately 177B, while the official accounting is closer to 180B. These figures use different accounting conventions and should not be treated as interchangeable.

The selected Unsloth UD-IQ4_XS GGUF is approximately 93.7 GB according to the repository’s current listing. The exact size and number of shards must be recorded with the model revision and checksums.

The model has a native context length of 262,144 tokens. Its hybrid design includes recurrent linear-attention components, sparse attention, per-layer n-gram embeddings, and a vision encoder. Gated DeltaNet reduces the conventional attention-cache requirement for some layers, but it does not make the complete model’s memory usage universally constant: sparse-attention and other implementation state still matter.

The hardware used for this experiment was:

The two GPUs provide 24 GB of aggregate VRAM, but they do not form one unified memory pool. Layer placement, tensor placement, PCIe topology, and transfers between host memory and devices remain important.

Experimental llama.cpp build

#

At the time of the experiment, upstream llama.cpp did not provide a released build for this architecture. The test used PR #27742, which remained an open and unmerged experimental pull request at the time of review. The PR should therefore be treated as a source snapshot, not as stable upstream support.

A simplified build sequence is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

git clone https://github.com/ggml-org/llama.cpp.git /opt/llama.cpp-qwen-next
cd /opt/llama.cpp-qwen-next
git fetch origin pull/27742/head:pr-27742
git checkout pr-27742

git rev-parse HEAD

cmake -B build-cuda \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build-cuda --config Release -j16 \
  --target llama-server llama-cli llama-gguf-split

The resulting commit SHA, CUDA toolkit, NVIDIA driver, kernel, and compiler versions are part of the benchmark configuration. A moving PR branch cannot provide a reproducible build by itself.

GPU offload versus CPU-resident experts

#

One experiment used --cpu-moe, which keeps MoE weights in host memory. The measured run used approximately 9.8 GB on GPU 0 and 7.1 GB on GPU 1 and generated about 1.24 tokens per second.

Keeping the MoE weights in host memory was possible, but it made synchronization and host-to-device transfers the dominant cost in this test. The exact number of transfers depends on graph partitioning, tensor placement, batching, and backend implementation.

A second configuration allowed a limited number of layers to be offloaded:

1

-ngl 10 --split-mode layer --tensor-split 1,1

In the reported runs, generation increased to approximately 3.06–3.37 tokens per second. -ngl 10 permits up to ten layers to be offloaded; it does not by itself guarantee that ten complete layers are placed identically on both GPUs. The actual placement should be confirmed using the startup logs from the pinned build.

NUMA interleaving

#

The host exposes two NUMA nodes with the following reported distance matrix:

1
2
3

node   0   1
  0:  10  21
  1:  21  10

NUMA distance values describe relative topology costs. They are not direct latency multipliers, so a distance of 21 should not be described as a measured 2.1× latency penalty.

The experiment used:

1

numactl --interleave=all /path/to/llama-server ...

This asks Linux to distribute newly allocated pages round-robin across the NUMA nodes. It does not eliminate remote access, and it does not prove that all eight memory channels are saturated. In this workload, the observed generation rate increased from approximately 2.58 to 3.04 tokens per second. To verify whether the improvement comes from better memory placement, compare it with NUMA and memory-bandwidth counters.

Thread-count experiments

#

The two Xeon processors provide 16 physical cores and 32 logical processors. The tested configuration found that 12 worker threads performed better than the alternatives:

1

-t 12 --threads-batch 12

The result should not be attributed automatically to Hyper-Threading port contention. A process using 16 threads is not necessarily scheduled on sibling logical CPUs; placement depends on affinity and scheduler behavior. The defensible conclusion is that 12 threads produced the best measured result for this workload. Higher thread counts did not improve throughput and may have increased contention for shared resources.

The reported optimized result was approximately 3.77 tokens per second, compared with a 2.58-token-per-second baseline. That is a 46.1% increase relative to that baseline:

$$\frac{3.77 - 2.58}{2.58} \approx 46.1%$$

It should not be called an overall improvement unless the other variables were held constant.

Context-length scaling

#

A conventional transformer KV cache generally grows linearly with the number of cached tokens, while attention computation can become increasingly expensive as context grows. The hybrid recurrent components in Qwen3.8 Flash-Next reduce conventional cache requirements for some layers, but the complete memory behavior depends on recurrent state, sparse-attention state, buffers, and the exact implementation.

The following values were reported by the original case study. Units and measurement method should be retained with the raw logs before publication.

The reported run showed approximately 1.9 GB of additional total VRAM usage between 32K and 262K. That is an observation from this build and configuration—not proof that the complete architecture has constant memory usage. “Stable” should mean something measurable, such as a specified number of successful runs without CUDA OOM, correct output, and a defined prompt and generation length.

Speculative decoding and n-gram embeddings

#

The model includes n-gram embedding parameters and an MTP component. Those are architectural model features; they are not the same thing as llama.cpp’s external ngram-mod speculative-decoding mechanism.

The original experiment reported:

• MTP configuration: not validated against the exact experimental build.
• ngram-mod: approximately 3.91 tokens per second versus 3.88 tokens per second in a comparison run.

These results are too close to interpret without repeated runs and variance estimates. In practical terms, this test did not establish a meaningful speedup. Any claim that external drafting is redundant would require controlled acceptance-rate and throughput measurements.

Service configuration

#

A service can be launched with the tested parameters, but the original example requires security changes before network use. Binding to 0.0.0.0 exposes the API on every interface, and running the service as root increases the impact of a vulnerability. A safer default is localhost binding behind an authenticated, TLS-terminating reverse proxy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

[Service]
Type=simple
User=llama
WorkingDirectory=/opt/llama.cpp-qwen-next
ExecStart=/usr/bin/numactl --interleave=all \
  /opt/llama.cpp-qwen-next/build-cuda/bin/llama-server \
  --model /models/Qwen3.8-Flash-Next-UD-IQ4_XS/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  --host 127.0.0.1 \
  --port 8000 \
  --ctx-size 262144 \
  --n-predict 16384 \
  --n-gpu-layers 10 \
  --split-mode layer \
  --tensor-split 1,1 \
  --parallel 1 \
  --flash-attn on \
  --batch-size 2048 \
  --ubatch-size 512 \
  --threads 12 \
  --threads-batch 12 \
  --threads-http 2 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --reasoning-preserve \
  --chat-template-kwargs '{"reasoning_effort":"xhigh"}' \
  --alias qwen3.8-flash-next
Restart=on-failure
RestartSec=3
NoNewPrivileges=true
PrivateTmp=true

Flag spellings can change in an experimental checkout; validate the command against the built binary. The model server should run under a dedicated account with only the required model-file, cache, and NVIDIA-device permissions. If LXC device permissions require a privileged design or root, document that as a deployment limitation.

LimitMEMLOCK=infinity is not a substitute for NVIDIA driver setup, LXC device mapping, or cgroup configuration. Likewise, a readiness loop that waits for /dev/nvidia0 only proves that one device node exists; it does not verify that both GPUs, CUDA, and the model backend are usable.

What I would do differently

#

I would capture the exact PR commit, model shard checksums, startup logs, and benchmark prompts before changing any tuning parameter. That small investment would make it much easier to separate a real improvement from normal run-to-run variation.

Reproducing the experiment

#

A useful replication record includes:

• the exact Qwen3.8 Flash-Next model revision and every GGUF shard checksum;
• the exact PR #27742 commit SHA;
• Proxmox, kernel, LXC, CUDA, driver, and compiler versions;
• whether the container is privileged or unprivileged;
• the LXC device-node and cgroup configuration;
• GPU PCIe topology and NUMA placement;
• CPU affinity, governor, and thread settings;
• prompt length, generation length, batch sizes, cache types, and sampling parameters;
• warm-up count, measured-run count, and whether results are median or single-run values; and
• VRAM measurement source, startup logs, and correctness checks at long context.

Keep prompt-processing speed, generation speed, time-to-first-token, and total throughput as separate metrics. Label the results as observations from the author’s system unless they have been independently reproduced.

Conclusion

#

This experiment demonstrates that a very large, experimental MoE model can be made to run on hardware that would not normally be considered suitable for it. The practical gains came from combining partial GPU offload, NUMA-aware memory allocation, and careful thread selection.

The more important lesson is methodological: experimental inference requires separating measured behavior from architectural assumptions. llama.cpp PR #27742, the model files, and the server flags should all be pinned. With that discipline, a Proxmox cluster becomes a useful lab for exploring how hybrid model architectures behave outside the datacenter.

If you have similar hardware, start with a smaller model or a released llama.cpp version before attempting this experiment. If your goal is to learn how hybrid models behave under memory pressure, this setup is a useful lab—but it is not a substitute for production inference infrastructure.

References

#

• Qwen3.8 Flash-Next model card
• Unsloth Qwen3.8 Flash-Next GGUF repository
• llama.cpp PR #27742
• llama.cpp repository
• llama.cpp CLI documentation
• Proxmox VE documentation
• numactl manual
• NVIDIA CUDA Toolkit

This post documents an experimental deployment. Hardware, model revisions, source commits, runtime flags, and benchmark results must be revalidated before replication.