TL;DR

• Question: Can you fine-tune a language model on AMD consumer GPUs, and what does it take?
• What we did: We fine-tuned Gemma 4 with LoRA on two AMD Radeon RX 7900 XTX cards using Unsloth on ROCm: on one card and on two, in 16-bit and 4-bit, and with a smaller and a larger model.
• Answer: Yes. Every setup trained, and on our test task the model's score went from 54% to about 80% in around five minutes.
• What to watch for: The install needed only two small AMD-specific fixes. The real trouble was four bugs that broke training without any error message, and none of them came from AMD. Checking the loss in the first 10 steps caught all four.
• Our recommendation: Start with Unsloth and 16-bit LoRA on one card. Add the second card when you want faster runs, and switch to 4-bit when the model doesn't fit.

The full experiments, with numbers and charts, are in the Appendix. New to some of the terms? See the Glossary.

In Epoch 0 I said this blog would learn the way models do: take a small step, check how wrong you were, adjust, repeat. In Epoch 1, we took that first step. We got Gemma 4 running on two AMD Radeon RX 7900 XTX cards with llama.cpp, serving our agents. That answered one question: can this box run the model?

The next question is harder: can it train the model?

On NVIDIA, fine-tuning is a well-worn path. Most kernels, libraries and tutorials are built for CUDA first: Flash Attention, xformers, bitsandbytes, even the nvidia-smi in every guide. On AMD, the path is much less clear.

For us, it isn't optional. Our project is air-gapped, so if we want to adapt a model to our own data, the training has to happen on the same machine.

Two things in 2026 made this worth trying. Unsloth released an official AMD install path, and bitsandbytes 0.50 added a ROCm backend, which is what 4-bit QLoRA needs.

So we took the Epoch 1 box and gave it something to learn.

Picking a task we can grade

Before training anything, we needed a way to tell whether training actually helped. That means a task with a clear score, where every answer can be checked against a reference.

We picked text-to-SQL. The model gets a table schema and a question in plain English, and writes one SQL query. Every question comes with a gold (reference) query, so we can compare the model's answer against it directly. The answers are also short, one line of SQL, so a full training run takes about five minutes instead of hours. That let us repeat every run several times.

Here's one example from the dataset:

Schema:   CREATE TABLE table_name_64 (pali VARCHAR, tibetan VARCHAR)
Question: What is the pali word for phrag dog in tibetan?
Answer:   SELECT pali FROM table_name_64 WHERE tibetan = "phrag dog"

The data is b-mc2/sql-create-context. We shuffled it once with seed 42, kept the last 300 rows aside as the test set, and trained on 5,000 of the rest. That split never changed.

We measured two things:

• Exact match: after normalising case, quotes and spacing, is the model's query the same as the gold query? This is our main score.
• Executes: does the query run in SQLite without an error? Every model, even the untrained one, scored ~98% here, so it couldn't tell runs apart.

Why not run both queries and compare the results? The dataset has table definitions but no rows, so there's nothing to return.

The model is Gemma 4 E4B, the small one: about 4B parameters active per token, 8B in total. Before any training, it scored 54.0% exact match. That's the number to beat.

The hardware is the same box from Epoch 1:

Every run used the same training recipe:

Getting it running: the easy part

We expected the install to be the hard part. It wasn't.

PyTorch ships ROCm builds (wheels) that bring their own ROCm runtime. We installed them, imported torch, and both GPUs showed up. bf16 worked. PyTorch's built-in attention (SDPA) worked. That covers most of what training needs.

Only two things needed fixing, and both were AMD-specific.

Installing Unsloth broke ROCm's Triton. Unsloth depends on triton and xformers. When pip installs them, it pulls the CUDA builds from PyPI, and those overwrite the ROCm Triton that came with PyTorch. The fix is a small overrides file that tells the installer to skip both packages:

# overrides.txt
triton; sys_platform == "never"
xformers; sys_platform == "never"

uv pip install --override overrides.txt "unsloth[amd]"

The CPU's built-in graphics showed up as a third GPU. Our Ryzen CPU has a small integrated GPU (gfx1036), and ROCm lists it next to the two 7900 XTX cards. One environment variable hides it:

export HIP_VISIBLE_DEVICES=0,1

Two smaller issues came from the software stack, not from AMD:

• Unsloth swaps google/… model names for its own mirror copies and downloads those instead. Setting use_exact_model_name=True keeps the model you asked for.
• On a fresh machine, load_dataset() in offline mode fails until the dataset has been downloaded once.

We also went without two things. Flash Attention 2 and xformers weren't available in our setup on gfx1100, so Unsloth used SDPA instead. We had nothing to compare against, so we can't say what that cost us. Every number in this post was measured without them.

You'll see a few scary-looking warnings in the logs too. All of them are harmless, and we lost time on each before learning to ignore them:

• Attempting to use CK on an unsupported architecture
• failed loads of torchao's CUTLASS kernels
• expandable_segments not supported on this platform

The plot twist: training ran, and learned nothing

Our first training run used the standard Hugging Face tools: TRL's SFTTrainer with a PEFT LoRA adapter. It started fine and logged its first loss: 102.

For this task, a healthy first loss is around 1. A loss of 102 means the model is being scored on the wrong thing. But nothing crashed, and nothing warned us. If we hadn't looked at that one number, the run would have finished, the eval would have shown no improvement, and we'd have concluded that "fine-tuning doesn't work on AMD".

Getting to a correct run took four fixes. None of these bugs raised an error:

1. Loss 102: The loss included the prompt instead of only the SQL answer. We want the loss on the SQL answer only, so TRL measures the prompt's length and masks it out. But TRL 0.24 expects apply_chat_template(tokenize=True) to return a list, and transformers 5.5 returns a BatchEncoding object instead. TRL read every prompt as 2 tokens long, so almost the whole prompt counted toward the loss.
2. Loss stuck at 14: some layers read empty memory. In Gemma 4 E4B, layers 24–41 don't compute their own keys and values (K/V). They reuse the K/V from earlier layers, passed along through the KV cache. Training turns the cache off (use_cache=False), so those 18 layers had nothing to read.
3. Gradients 8× too big. We accumulate gradients over 8 steps, so the loss should be divided by 8. Gemma 4's forward accepts the num_items_in_batch argument that should handle this scaling, but ignored it.
4. 36 LoRA modules that never learned. Those same 18 layers never run their key and value projections, so the LoRA adapters attached to them never got a gradient (36 modules, 72 weight matrices). On one GPU, this fails silently. On two GPUs, DDP refuses to start, which is how we noticed.

Here's the twist: none of these are AMD bugs. They come from two libraries disagreeing about a return type, and from how Gemma 4 is built. The same code would fail the same way on an NVIDIA card.

Unsloth's Gemma 4 patches already fix the first three. When we switched to Unsloth, we got the same accuracy as our hand-patched Hugging Face setup, and training ran 1.7× faster (321 s vs 548 s). For the fourth issue, we removed the unused K/V projections from the LoRA target list, reducing the target modules from 294 to 258.

We also made a rule and followed it for the rest of this post: run a 10-step smoke test before every long run. If the first loss is around 1 (we saw 0.9 to 1.4) and going down, start the real run. If it's around 14 or 100, stop and debug.

What we found

With training fixed, we ran four experiments. Each one changed a single thing, and each setup ran with three seeds, so every number below is an average of three runs.

So yes: training works on these cards, on one GPU or two, in bf16 or 4-bit.

One more thing stood out. Every setup landed at about 80%. When we read the answers, about 9% of the test questions turned out to have wrong or inconsistent gold answers, and most of what fine-tuning "fixed" was the dataset's writing style. So 80% is close to the ceiling of this test, not a limit of AMD or the model.

The full setups, tables and charts for each experiment are in the Appendix at the end.

The AMD scorecard

Here's everything we ran into, sorted by whose problem it really was:

• AMD: specific to ROCm or AMD hardware.
• Stack: Gemma 4, transformers, TRL, PEFT, Unsloth or datasets. These would happen on NVIDIA too.
• Host: our particular machine.

✅ Works as-is

🔧 Works with a workaround

❌ Not used: Flash Attention 2 and xformers weren't available in our stack on gfx1100. SDPA took their place in every run.

Not tested: FSDP or splitting one model across both cards, the 26B and 31B models, NVIDIA head-to-head.

The verdict: on an RX 7900 XTX, the AMD-specific problems were packaging issues, and you fix them once. What actually cost us time, four silent training bugs and a model type our library didn't recognise, would have hit us on any GPU.

If you already own these cards, our recommended default is Unsloth with bf16 LoRA. Add DDP across both cards when you want faster runs, and switch to QLoRA when the model doesn't fit in bf16.

Seven more traps that give you wrong numbers without an error

Epoch 1 ended with seven traps that give you wrong numbers without any error. A crash costs you an hour; a wrong number that looks right costs you the decision. Training added seven more:

Loss after Epoch 2

If Epoch 1 was the first real gradient step, this was the second. Here's what the loss told us, and where it points next:

• The AMD question had a software answer, again. Only two issues were AMD's, both fixed at install. The rest would have followed us to any GPU.
• Smoke-test the loss. Four bugs trained without an error. Ten steps and one number caught all of them.
• Three seeds, or it didn't happen. On 300 test questions, a 2-point gap between two single runs is noise. Every claim in this post is an average of three.
• Check what 4-bit can actually shrink. QLoRA compresses the model's main layers, but not its embedding tables. In E4B, 44% of the weights are embeddings, so 4-bit saved only 3.4 GB. In the 12B, embeddings are just 8%, so almost the whole model shrank and it trained in 9 GB.
• Bigger wasn't better here. The 12B fit on one card and matched E4B, at 2.3× the training time. Five setups landed on the same number.
• Read the misses, not just the score. About 9% of the test set can't be reliably matched, and two-thirds of the consistent fixes were style. A score can be stable and still measure the wrong thing.

Links

Gradient Decent

• Epoch 0: Initializing the Weights
• Epoch 1: Try It, Time It, Then Buy It

Tools and docs

• Unsloth: installing on AMD GPUs
• Unsloth: fine-tuning Gemma 4
• Unsloth: multi-GPU training
• bitsandbytes 0.50 installation (ROCm backend)
• PyTorch ROCm 7.2 wheels
• Google: fine-tuning Gemma with QLoRA in Hugging Face

Models and data

• google/gemma-4-E4B-it
• google/gemma-4-12B-it
• b-mc2/sql-create-context

Taking small steps toward slightly less wrong. ∇

Appendix: the experiments in detail

The main post gives the short version. Here are the full setups, tables and charts for each experiment, plus our deep dive into the answers the models got wrong.

Five experiments, one chain of questions

We planned the experiments as a chain: each one answers a question and sets up the next. Before running each one, we wrote down what it had to prove.

Each experiment changes one thing and keeps everything else fixed.

We ran each setup with three seeds: 3407, 42, and 1234. With 300 test questions, one answer changes the score by a third of a percentage point. A two-point difference between single runs could be random variation, so we report the mean and spread across seeds.

Why these training settings

We didn't tune any of these. The goal was to test the platform, not to get the best possible score, so we picked common starting values and kept them the same for every run. That way, any difference between experiments comes from the one thing we changed.

Because these settings were chosen for the smaller E4B model and never tuned, the 12B results in Exp D show what the 12B does with the same recipe, not the best it could do.

Exp A — It learns, every time

Objective: Train the model with LoRA for 300 steps and check that its score goes up by at least 20 points. It has to happen on all three seeds, not just once.

Why 20 points? On a test set of 300 questions, a score can move by about ±5 points just by chance. A 20-point gain means 60 more correct answers, about four times what noise alone could produce. If every seed clears that bar, we know the training really worked and the gain isn’t luck.

Setup: Unsloth · bf16 · one RX 7900 XTX · 34.9M trainable parameters (0.43% of the model).

It worked. Exact match went from 54% to about 80%, in a little over five minutes per run. The best and worst seeds were 2.7 points apart, and all three cleared the +20 target.

Peak VRAM moved by up to 2 GB between seeds, even with identical settings. So for the rest of this post, we don't read anything into VRAM gaps smaller than that.

So what changed in the answers? Here's a question the base model got wrong:

What is the pali word for phrag dog in tibetan?

-- before: looks up the wrong column
SELECT tibetan FROM table_name_64 WHERE pali = 'phrag dog'
-- after
SELECT pali FROM table_name_64 WHERE tibetan = "phrag dog"

The base model swapped the two columns. After fine-tuning, it picks the right ones.

Here's a different kind of fix:

-- before
SELECT player FROM table_name_4 WHERE score = '71'
-- after
SELECT player FROM table_name_4 WHERE score = 71

Both queries run fine. In fact, the column is a VARCHAR (text), so the base model's quoted '71' is arguably the more correct one. Keep this example in mind; it comes back later.

Exp B — QLoRA works on AMD now, but saves less than you'd think

QLoRA stores the frozen base model in 4-bit (NF4) to save memory, and trains the small LoRA adapters in bf16 on top. It relies on bitsandbytes, which for a long time only ran on CUDA.

Objective: check that 4-bit QLoRA trains cleanly through bitsandbytes' ROCm backend, and measure what it saves.

Setup: same as Exp A plus --load-in-4bit · frozen base weights in NF4 · LoRA adapters train in bf16.

• No NaNs. Unsloth's AMD guide warns that bitsandbytes 0.49.2 and older can produce NaNs in 4-bit. On 0.50.2 we saw none, in training or in generation.
• Same accuracy. On the 4-bit base it was trained with, QLoRA scored the same as bf16 LoRA.
• Use the adapter with the same base it was trained on. The QLoRA adapter was trained on top of the 4-bit model. When we attached it to the original bf16 model instead, every seed scored about 2.8 points lower. The adapter had learned to work with the 4-bit weights, so it fits that version of the model best.

The surprise was memory. We expected 4-bit to save about 5 GB at peak. It saved 3.4 GB, and training took 40% longer. To see why, look at what E4B is made of:

44% of E4B is embedding tables. 3.49B of its ~8B parameters are lookup tables: a 262k-token vocabulary plus Gemma's per-layer embeddings. QLoRA only shrinks the decoder's linear layers to 4-bit. The embeddings stay in bf16. So only the decoder's 4.03B parameters get smaller:

$$\text{most we can save} \approx 4.03\text{B} \times (2 - 0.5)\ \text{bytes} \approx 6\ \text{GB} \qquad \text{measured after loading: } 14.9 - 9.4 = 5.5\ \text{GB}$$

During training, activations and optimizer state take the same space in both setups. That's why the 5.5 GB saved after loading shrinks to 3.4 GB at peak.

In Epoch 1, the lesson was to count active parameters, not total ones. This is the same idea for memory: count what quantisation can actually touch. For E4B, that's only about half the model. Keep that 44% in mind for Exp D.

Exp C — Two cards, no tuning

Next question: does the second GPU help? We used DDP. Each card holds a full copy of the model, trains on its own half of the batch, and the two cards average their gradients after every step. On AMD, that averaging runs over RCCL, AMD's version of NVIDIA's NCCL.

Objective: measure what the second card buys, and check that the accuracy doesn't change.

Setup: the same bf16 LoRA · launched with torchrun --nproc_per_node 2 · batch 4 per card, no accumulation, so the effective batch is still 8 (Unsloth multi-GPU docs).

RCCL worked out of the box, with no environment variables and no tuning. Accuracy stayed the same, and a run took about three minutes instead of five: 1.63× faster. We expected about 1.7×. The small gap is most likely the time spent syncing gradients over PCIe.

Each card used about 2 GB more than the single-GPU run, at the same batch size of 4. DDP keeps its own gradient buffers, which is the likely reason.

The problems we did hit were on the CPU side, not the GPUs:

• RAM ran out while loading. Both processes load the model at the same time, and each holds about 13 GB in CPU RAM while doing it. Together they overflowed our 30 GB machine into swap, and loading slowed to about one second per tensor.
• dist.barrier() made both processes hang. The usual way to load one process at a time was deadlocked, because Unsloth runs its own collective operations inside from_pretrained.
• A plain file lock (fcntl.flock) fixed it. One process loads, then the other.

Exp D — Going bigger: 12B on one card

Objective: find out whether Gemma 4 12B fine-tunes on a single 24 GB card, and whether it beats E4B.

Setup: QLoRA · one card · the exact E4B recipe: same learning rate, steps and batch. We didn't tune anything for the bigger model, on purpose.

The dependency trap, again

The first attempt to load the 12B failed with KeyError: 'gemma4_unified'. The 12B checkpoint uses a model type that transformers 5.5 doesn't know about.

Epoch 1 had the opposite problem: an unpinned transformers update broke vLLM. This time, our pinned version was too old.

We didn't want to touch the environment that all our E4B numbers came from. So we built a second one, venv-12b, identical except for a newer transformers (5.17), tokenizers and Unsloth. We diffed the two package lists to make sure nothing else had changed.

The side quest that became a rule

To check the new environment, we re-ran one E4B seed in it. Same accuracy, but it took 616 s instead of 311 s. Twice as slow.

We spent a good while on this. We turned off Unsloth's compilation (no change), compared the two stacks step by step, and tried a cold compile cache. The real cause was much simpler: the 12B model was downloading in the background the whole time, 23 GB over a slow connection.

With nothing else running, the new stack took 302 s against the old stack's 314 s. No slowdown at all.

New rule: no downloads and no other jobs during a timed run.

Results

The 12B fits easily: 9 GB at peak on a 24 GB card. In bf16, its ~13B parameters alone would need about 26 GB, so 4-bit is what makes this run possible. QLoRA pays off here because only 8% of the 12B is embeddings (see the parameter chart in Exp B), so almost all of it gets quantised.

It even uses less memory than E4B in QLoRA. Part of that is the newer software stack: an E4B control run in it used 2 GB less. The other part is that the 12B has no per-layer embedding tables. E4B's 3.49B bf16 embedding parameters take about 7 GB, more than the 12B's whole 4-bit decoder (about 5.4 GB).

But the bigger model isn't more accurate. It starts 3 points higher (57% vs 54%), ends at the same place within noise, varies more across seeds, and takes 2.3× longer to train. The loss curves show why:

The 12B starts with a lower loss (1.07 vs 1.26), but by around step 30 all four curves sit on top of each other. Model size wasn't what kept the score at 80%. One caveat: we reused E4B's settings, so a tuned 12B might do better.

Still, five setups and two model sizes landing on the same number made us suspicious of the number itself.

Read the misses, not just the score

Every run saved its answer to each of the 300 test questions. So we lined up all 15 fine-tuned runs (A, B on both bases, C and D, three seeds each) and compared them question by question:

32 questions were wrong in every single run, whatever the setup, seed or model size. We read all 32 and sorted them by cause:

The 16 bad gold answers break down like this:

• four COUNTs where the question asks for a value
• four arbitrary AVGs or SUMs
• four that are broken or nonsense SQL
• two that need information that isn't in the schema or the question
• two with the wrong value or operator (laps = 585 for "more than 585 laps")

So what does that mean for the score?

• At least 16 gold answers are simply wrong, and another 10 can't be matched reliably. The quoting cases only match if the model happens to guess the dataset's inconsistent style, and the equivalent queries never match.
• That's 26 of 300 questions (8.7%). So the practical ceiling for exact match on this test set is about 91%, not 100%.
• It's an estimate. We only checked the always-wrong questions. Some bad gold answers are probably hiding among the 68 mixed ones too.
• At ~80%, every setup is really competing over those 68 mixed questions. That's where the seed-to-seed spread comes from.

Two of the six real errors also trace back to the dataset. Training taught the model that "how many" means COUNT so well that it adds COUNT even when the column already holds a count (countries_sampled, played). The same habit probably earns it points on some of the bad COUNT gold answers too.

Then we looked at the other side: what did fine-tuning fix? The base E4B model got 138 questions wrong. Fifty of those were fixed by all 15 runs, including the three 12B runs, even though they started from a different base model:

Remember the '71' from Exp A? Of those 50 fixes, about two-thirds were the dataset's house style, not better understanding. The base model's SQL was often fine; it just wasn't written the way this dataset writes SQL. Fine-tuning taught it those habits, and exact match rewards habits.

One more sign of this: the base model wrote 116 of its 300 answers across several lines, the way a person would. After fine-tuning, all 4,500 answers (15 runs × 300) are on one line, like the dataset.

This doesn't make the fine-tuning fake. But it changes what "54% → 80%" means. Part of it is the model understanding the questions better, and a big part is the model learning to match one particular grader. In Epoch 1, reading the outputs showed our benchmark was measuring speed, not answer quality. Here, reading the misses showed our score was partly measuring the dataset.

Putting the experiments together

Looked at side by side, the four experiments tell one story:

• The training itself is solid on AMD. Every setup learned, every seed landed within a few points of the others, and the GPU gave the same scores when we re-ran the evaluations.
• The choices change cost, not accuracy. bf16 or 4-bit, one card or two, 4B or 12B: all of them ended at about 80%. What changed was time and memory. Two cards made runs 1.63× faster, 4-bit made the 12B fit in 9 GB, and the bigger model took 2.3× longer for no gain.
• The ceiling came from the test, not the hardware. About 9% of the test set can't be reliably matched, and much of what training "fixed" was the dataset's style. To see real differences between setups, the next step is a cleaner test set and an evaluation that checks meaning, not exact wording.

So the details back up the main answer of this post: yes, you can fine-tune on AMD. The numbers you get are only as trustworthy as the test you measure them with.

Glossary