Running Qwen3.6-27B on a Home Server: Complete Guide to Building a Local LLM

Efforts to run large language models (LLMs) on home servers are quietly gaining traction among engineers. Concerns about reliance on cloud APIs and an increasing focus on data privacy are driving this trend. Ryō Igarashi has published a detailed account of his experience on Zenn, documenting the technical process of running Alibaba’s open-weight model “Qwen3.6-27B” with GPUs accessible to regular households.

Reasons to Choose a Local LLM

In the introduction to his article, Igarashi shares his internal conflict between his interest in AI technology and his discomfort with major AI companies. He cites ethical concerns such as fear-inducing statements by Anthropic’s CEO, xAI Grok’s involvement in the U.S. attack on Iran, and the environmental impact of data center construction. These concerns led him to distance himself from mainstream AI trends and turn to local LLMs as a means of staying updated with the technology.

This perspective partially resonates with geohot’s critique of LLM hype. While running models on personal hardware involves significant costs and effort, Igarashi aims to leave behind a record of his trials and errors, hoping that his documentation will help others experiment with local LLMs in the future.

Criteria for Model Selection

As of June 2026, the highest-performing open-weight models with less than 100 billion parameters are Google’s Gemma 4 and Alibaba’s Qwen3.6. Both were released in April 2026 and have achieved scores on multiple benchmarks, including SWE-bench, comparable to Claude Opus 4.5. Igarashi chose Qwen, which initially piqued his interest in local LLMs.

Qwen3.6 comes in two variants. The Qwen3.6-27B is a dense model that uses all 27 billion parameters for inference. The Qwen3.6-35B-A3B is a MoE (Mixture-of-Experts) model that utilizes only around 3 billion parameters relevant to the input for inference. The MoE model is faster in inference and can run well on limited computing resources. However, Igarashi determined that the dense model outperforms the MoE model for Qwen3.6 and opted for the 27B model.

Calculating Required VRAM

Qwen3.6-27B has 27 billion parameters, each stored as a 16-bit floating-point number. This means the model itself requires 54GB of memory, which is impractical for a typical home setup. To mitigate this, Igarashi utilized a technique called quantization, converting the parameters to 8-bit floating-point numbers, resulting in the “Qwen3.6-27B-FP8” model.

According to vLLM’s documentation, the approximate VRAM requirements can be calculated using the following formulas:

KV Cache ≈ batch_size × seq_len × hidden_size × num_layers × 2 × dtype_size Model Weights ≈ num_parameters × dtype_size Activation Memory ≈ 10%-30% of Model Weights Required VRAM ≥ KV Cache + Model Weights + Activation Memory

Substituting the parameters for Qwen3.6-27B-FP8 into the formulas, Igarashi calculated the required VRAM to be approximately 43.69GB. Based on this result, he targeted a GPU setup with at least 44GB of VRAM.

Hardware Procurement

To meet the VRAM requirements, Igarashi purchased two used GPUs on Mercari. While he did not specify the exact models, he assembled a configuration with a combined VRAM of 48GB. The GPUs were connected via NVLink, pooling memory to enable inference for large-scale models that would otherwise be impossible. The total cost amounted to approximately 250,000 yen.

This procurement method offers valuable insights. Buying a brand-new high-end GPU, such as an RTX 5090, would require a budget exceeding 600,000 yen. By leveraging the secondhand market, Igarashi significantly reduced costs while achieving practical performance, demonstrating how local LLM adoption barriers can be lowered.

Setting Up the Inference Server

After securing the hardware, Igarashi began setting up the inference server. He chose vLLM as the framework. vLLM is an open-source library optimized for large language model inference, featuring efficient memory management through PagedAttention and improved throughput via continuous batch processing.

The setup process was relatively straightforward. Igarashi ran vLLM on a Docker container and loaded the Qwen3.6-27B-FP8 weights downloaded from Hugging Face. The host operating system was Ubuntu 24.04 LTS, and CUDA version 12.8 was used. Once the inference server was operational, it published an OpenAI-compatible API endpoint, enabling easy integration with existing applications.

Performance Testing and Practicality

After deploying Qwen3.6-27B-FP8, Igarashi reported that, depending on the prompt, the model produced responses comparable in quality to cloud-based Claude Opus 4.5. However, he noted that the model still lagged behind cloud APIs in tasks requiring natural Japanese expression or complex reasoning.

Regarding inference speed, the server achieved practical response times for short-context queries. However, when processing long contexts of up to 262,144 tokens, the KV cache heavily strained the VRAM, significantly slowing down the response time.

Operating Costs and Sustainability

According to Igarashi’s calculations, running the vLLM server 24/7 would incur a monthly electricity cost of approximately 5,000 yen, assuming GPU idle power consumption of 150W and maximum consumption of 350W during operation. Compared to the pay-as-you-go pricing of cloud APIs, local operation becomes more cost-effective with higher usage frequency.

However, considering the initial hardware investment of around 250,000 yen and the engineering costs for setup and maintenance, short-term cost recovery is challenging. Igarashi views the project as “a means of technical skill development and an expression of ethical stance,” rather than purely a cost-saving measure.

Editorial Opinion

In the short term, detailed documentation like this article could lower the technical barriers to adopting local LLMs. The specific examples of vLLM setup steps and VRAM calculations are invaluable resources for developers pursuing similar projects. The easing of semiconductor shortages and the expansion of the secondhand GPU market could further support this trend.

From a long-term perspective, the key to local LLM adoption lies in how closely their performance can match that of cloud APIs. The rapid improvement in benchmark scores of open-weight models, exemplified by Gemma 4 and Qwen3.6, suggests that sub-100B models are closing the gap with Opus 4.5-level performance. Organizations prioritizing data privacy and latency may increasingly shift to in-house operations within the next one to three years.

As an editorial team, we are keen to observe whether the methods demonstrated in this article remain confined to individual engineers’ technical interests or extend to corporate AI infrastructure strategies. Given practical constraints like GPU availability and electricity costs, the question remains whether local LLMs will remain a niche pursuit or evolve into a broader trend.

References

• “ローカル LLM を構築した”, by Ryō Igarashi — Zenn, 2026-07-14T08:13:16.000Z (ARR)
• Source URL: https://zenn.dev/neet/articles/11bafab8645995