llama.cpp on an FPGA: Work in Progress
I’m building a backend for llama.cpp that runs LLM inference on an FPGA card, an AMD Alveo U280. It is work in progress, and the code is private for now. This post records how fast it is today.
Why an FPGA
When a model writes text, each new token needs all of the model’s weights read from memory once. So decode speed, the speed at which words appear, is set mostly by how fast the hardware reads memory, not by how fast it computes. The U280 carries 8 GB of HBM, stacked high-bandwidth memory, rated at 460 GB/s. The goal is to decode as fast as that memory allows, for any model that fits, at any context length.
Where it stands
The card works as an ordinary llama.cpp backend: models come from the usual GGUF files, and speed is measured with llama.cpp’s own llama-bench. Twelve models from four families run on it, with outputs checked against llama.cpp’s CPU backend. The same GGUF files on three llama.cpp backends:
| model | size GiB | tg128, t/s | pp512, t/s | ||||
|---|---|---|---|---|---|---|---|
| FPGA | CPU | RTX 4090 | FPGA | CPU | RTX 4090 | ||
| Qwen3 | |||||||
| ggml-org/Qwen3-0.6B-Q4_0.gguf | 0.39 | 302.3 | 111.0 | 776.4 | 272 | 1,261 | 46,094 |
| ggml-org/Qwen3-4B-Q4_K_M.gguf | 2.32 | 72.5† | 17.5 | 259.7 | 83† | 230 | 15,905 |
| Qwen/Qwen3-8B-Q4_K_M.gguf | 4.68 | 44.9 | 10.9 | 165.2 | 52 | 163 | 11,418 |
| Llama | |||||||
| bartowski/Llama-3.2-1B-Instruct-Q4_K_M.gguf | 0.74 | 202.4 | 50.9 | 716.6 | 275 | 923 | 38,219 |
| bartowski/Llama-3.2-3B-Instruct-Q4_K_M.gguf | 1.87 | 92.7 | 22.6 | 329.5 | 114 | 318 | 20,065 |
| bartowski/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf | 4.58 | 47.7 | 11.2 | 171.6 | 54 | 163 | 11,462 |
| Gemma 3 | |||||||
| ggml-org/gemma-3-270m-it-qat-Q4_0.gguf | 0.22 | 529.4 | 112.4 | 878.8 | 1,642 | 3,081 | 57,202 |
| ggml-org/gemma-3-1b-it-qat-Q4_0.gguf | 0.66 | 239.8 | 58.4 | 515.1 | 488 | 795 | 34,729 |
| ggml-org/gemma-3-4b-it-qat-Q4_0.gguf | 2.35 | 84.4 | 21.0 | 240.5 | 118 | 243 | 18,292 |
| GPT-2 | |||||||
| mradermacher/gpt2.Q4_K_M.gguf | 0.10 | 959.1 | 206.0 | 1,836.6 | 1,213 | 5,435 | 94,267 |
| mradermacher/gpt2-medium.Q4_K_M.gguf | 0.25 | 402.6 | 63.5 | 939.9 | 415 | 2,112 | 57,861 |
| mradermacher/gpt2-large.Q4_K_M.gguf | 0.51 | 218.6 | 47.3 | 582.6 | 213 | 931 | 35,498 |
| † From a later build of the same version. | |||||||
llama-bench with its defaults, version 1.1.0, October 2026. tg128 is 128 tokens generated from an empty cache (decode); pp512 is a 512-token prompt (prefill). Tokens per second, the mean of five runs. Size is llama-bench's model size; all models are 4-bit quantized. The CPU is the same host's two Xeon Gold 5218, 32 cores. The FPGA columns are shaded.
Decoding Llama 3.1 8B, the card reads 224 GB of weights a second, against the memory’s rated 460 GB/s. An RTX 4090 is still faster on every model; its memory is rated at 1,008 GB/s. Prefill is still slower on the card than on the CPU; this work is about decode.
Next
Keeping this speed as the context grows, up to the longest context the card’s memory holds.