LoRA Fine-Tuning Qwen3.5 with mlx-lm: 162 Wrong Answers, Then 18

I spent three hours on LoRA fine-tuning a 4B model on my Mac mini, and it came out worse than it went in.

On 200 hard multiple-choice answers it got 162 wrong. The same model without training got 55 wrong.

The second run changed one setting, the learning rate. It got 18 wrong. That puts it level with the untrained 9B model on the same Mac, which got 20 wrong.

This is the build log of both runs with mlx-lm. The learning rate I picked first is a normal one in most LoRA guides. What I got wrong was what that number means in mlx-lm.

Why I fine-tuned a 4B model at all

Since late September I have been benchmarking typed decisions: pick a category, give a score, answer yes or no.

The test set is 450 Dutch business texts in three tests: helpdesk emails, search queries and sales call fragments. The setup is in Jev vs 25 lines of Python.

Each test has an easy, a medium and a hard tier of 50 texts.

On the 200 choice answers of the hard tier, the untrained Qwen3.5 4B running locally made 55 errors. The 9B made 20. Jev, the hosted decision model, made 12.

The 9B is the better local model and also the slower one. On the hard tier its median call took 1.5 seconds against 0.9 for the 4B. So the question was simple.

Can a few hours of training on my own easy and medium texts close the gap for the small model?

The setup

I wrote the split, the hyperparameters and four hypotheses down before training. The main one: the trained 4B beats the untrained 4B in at least two of the three tests.

• Base model: mlx-community/Qwen3.5-4B-4bit on a Mac mini M4 with 24 GB, mlx-lm 0.31.3.
• Data: train on the 300 easy and medium texts. 30 go to validation, 270 to training. That gives 900 training examples, one per question per text.
• Test: the 150 hard texts. The model never sees them in training.
• Format: each example is the exact evaluation prompt with thinking off. The completion is one answer token: a letter, a digit for the score questions, or ja/nee. With mask_prompt, only the completion counts in the loss.
• LoRA: rank 8, scale 20 (the mlx-lm default), the last 16 layers, every linear layer in them. That is 8.1 million trainable parameters out of 4.2 billion, 0.19 percent.
• Optimizer: AdamW, learning rate 1e-4, constant. Batch 1 with gradient accumulation over 4, gradient checkpointing on. 1,800 micro-steps, so 450 updates and two epochs.

One guard in my script turned out useful. It raises an error if the tokenized prompt is not an exact prefix of the tokenized prompt plus answer.

If the chat template inserts something between the two, the mask points at the wrong tokens and nothing tells you.

Batch 4 without gradient checkpointing ran out of GPU memory on the first step.

Qwen3.5 has a vocabulary of 248,320 tokens, so the logits for one 481-token sequence take about 0.48 GB in fp32. That is the likely cause.

I can't prove it, because I changed the batch size and turned on gradient checkpointing in the same step. Peak memory after the fix was 10.5 GB.

The run took 3 hours and 2 minutes.

Run 1: a loss that looked like progress

Validation loss went from 5.87 before training to 0.70 at step 600 and 0.39 at step 1,200. It ended at 0.48. Training loss jumped once, from 0.49 to 3.18 around step 300, and then settled.

I read that as a model that learned something. Then I ran it on the hard texts.

The model had stopped reading. Every choice answer was the letter A or E. Search intent: A, 50 out of 50. Sales signal: E, 50 out of 50. The helpdesk category was only ever A or E, out of six options.

That is not the textbook majority-class collapse.

The training labels were close to balanced per question: 13 to 16 examples per class for the helpdesk category and the sales signal. There was no majority to fall back on.

The adapter learned a letter preference that ignored both the text and the option list.

The business consequence sits in one question. The helpdesk test asks whether an email needs a human. Of the 20 hard emails that did, the trained model missed all 20. The untrained 4B missed 7.

Four things in mlx-lm and MLX I should have read first

I went through the source to find out why. Line numbers below are for mlx-lm 0.31.3, the version I ran. Four findings, from the one that hurt most to the one that only cost me time.

1. scale is alpha divided by rank, not alpha

In the original LoRA paper the adapter output is multiplied by alpha/r. In Hugging Face PEFT you set lora_alpha and PEFT divides by the rank.

PEFT's default is r=8, lora_alpha=8, so a scaling of 1.

In mlx-lm you set scale, and the layer applies it directly. This is the forward pass in mlx_lm/tuner/lora.py, lines 95 to 98:

There is no division by the rank. The default scale: 20.0 at rank 8 (mlx_lm/lora.py, line 74) is the same as lora_alpha=160 in PEFT.

With Adam, scale times learning rate is roughly the step size on the adapter output. The LoRA paper says tuning alpha is "roughly the same as tuning the learning rate".

The mlx-lm defaults pair scale 20 with a learning rate of 1e-5, a product of 2e-4. I took 1e-4 from PEFT-style guides and kept scale 20. Product 2e-3, ten times the default.

What makes this easy to miss: --learning-rate is a CLI flag, but rank and scale can only be set in a YAML config.

Copy a learning rate from a PEFT or Unsloth guide into the CLI and the scale of 20 stays, silently.

Since 0.32.0, a config that says alpha raises an error pointing out that mlx-lm uses scale (PR #1671). A config that only sets the learning rate gets no warning.

2. MLX's AdamW skips bias correction by default

Adam keeps running averages of the gradient and its square. Both start at zero, so the first updates are biased. PyTorch's AdamW always corrects for that.

MLX's Adam and AdamW have bias_correction: bool = False as the default (mlx/optimizers/optimizers.py, lines 498 and 570).

Without correction, every step in my run came out larger than with it, the early ones most.

My own calculation, for the default betas: 3.2 times larger at update 1, 6.5 times at update 10, 3.7 times at update 75 and still 1.7 times at update 450.

The loss jump showed up in the log at micro-step 300, which covers updates 51 to 75. The overshoot was larger before that, while the loss was still falling.

So the timing alone doesn't pin the jump on bias correction.

I also had no warmup, a constant learning rate and no gradient clipping.

That last one is not a setting I skipped: mlx-lm's trainer has no clipping option, also not in 0.32.0 (issue #1658).

Mosbach et al. (ICLR 2021) link Adam without bias correction in BERT fine-tuning to failed runs that end at majority-class accuracy.

3. The loss mask counts one padding token per example

My completions are three tokens: the answer, <|im_end|> and a newline. Yet Trained Tokens in the log rose by exactly 4 per micro-step.

This is the mask in mlx_lm/tuner/trainer.py, lines 92 and 93:

A sequence of length L occupies positions 0 to L-1. steps <= L also includes position L.

Batches are always padded past the longest sequence (lines 156 to 161), so that position exists. It holds token id 0.

The same lines are in 0.32.0 and on main. The off-by-one was reported upstream in August as PR #1697, which was closed without a merge.

My log shows the same thing independently: three completion tokens, four trained tokens per step.

I haven't measured what the extra token does to training. For reading the validation loss it matters.

The reported loss is an average over four tokens, and three of them become trivial fast. The answer token hides in the average.

Here is how I read it now. Multiply the validation loss by the masked tokens per example. That gives an upper bound on the loss of the answer token.

Run 1: 0.48 times 4 gives a bound of 1.93 nats per answer. Uniform guessing on this validation mix costs about 1.34.

The real answer loss is lower than the bound, but I had nothing that showed the run was healthy.

The same mask explains the speed line in the log. Tokens/sec showed 0.6, because it counts masked tokens only. The real rate was about 39 processed tokens per second.

4. mlx_lm.server ignored my adapter

I evaluate through mlx_lm.server, so I started it with --adapter-path.

In 0.31.3 the server resolves the model name first and then looks up the adapter under the resolved name, which is not there (mlx_lm/server.py, lines 388 and 389).

It serves the base model without a warning.

This is issue #1248, fixed in 0.32.0. The workaround is to send "adapters": "<path>" in every request.

I checked that it worked: with the adapter, the top-5 tokens differed from the base model on 4 of 4 test prompts.

Run 2: learning rate 1e-5

For the second run I changed the learning rate to the mlx-lm default of 1e-5 and kept every other setting. The batch order changed too, because it was not seeded. More on that below.

I decided on this run after seeing run 1, so it is an exploratory arm, not one of my pre-registered tests. It took 3 hours and 4 minutes.

There was no loss spike this time. Validation loss went to 0.046 at step 600, 0.034 at step 1,200 and 0.037 at the end.

That is more than ten times lower than run 1. The upper bound on the answer loss is now 0.15 nats.

Against the untrained 4B the gain holds up in all three tests: the bootstrap intervals stay above zero. Of the errors, 41 were made only by the untrained model and 4 only by the trained one.

Against the 9B I can't show a difference. Run 2 made 12 errors the 9B did not, and the 9B made 14 that run 2 did not. That is a draw (McNemar p = 0.85).

Against Jev it is 13 against 7, p = 0.26, also not shown.

The predictions are spread again. For search intent: 12 informational, 13 commercial, 14 transactional and 11 navigational, against 13, 12, 12 and 13 in the answer key.

On the "needs a human" question, run 2 missed 1 of the 20 emails and sent 4 to a human that didn't need it. The untrained 4B missed 7.

Calibration barely moved. The expected calibration error is 0.053, against 0.046 for the untrained 4B and 0.059 for the 9B.

The hard texts are also longer than anything in training: prompts up to 1,824 tokens, where the longest training example was 472. The adapter held up on those.

On speed, the trained 4B stays a 4B. The 500 calls on the hard tier took 615 seconds, against 1,005 for the 9B.

The adapter may cost a little: 589 seconds for the untrained 4B, measured on another day. Other work ran on the same Mac, so treat these timings as a snapshot.

Read also: Jev vs OpenAI's Decisions API vs Claude Haiku 5.5: the hosted models on the same 450 texts

What this does not show

The texts come from one source. Training and test texts were written for this benchmark by the same model family and labelled with the same definitions.

The model can learn the conventions of this dataset. That is what fine-tuning is for, but it says nothing about texts from somewhere else.

One run per setting. My script calls train_model() directly, and mlx-lm only seeds the batch shuffle inside run() (mlx_lm/lora.py, line 320).

So run 1 and run 2 also differ in the order of the examples, while the LoRA weights started identical. Dodge et al. (2020) found data order matters about as much as weight initialisation in fine-tuning variance.

One change, several suspects. Lowering the learning rate shrinks the effective step, including the bias-correction overshoot. This run can't tell them apart, or tell either from the missing warmup.

The 4-bit base stays open. Unsloth advises against 4-bit LoRA on Qwen3.5 because of larger than normal quantization differences.

That advice is about bitsandbytes QLoRA, and MLX quantizes differently, so it may or may not carry over.

Every run here used the 4-bit base, because that is what fits next to everything else on a 24 GB Mac. Run 2 works anyway. Whether a bf16 base does better, I don't know.

A checklist for LoRA fine-tuning in mlx-lm

1. Start from the mlx-lm defaults: learning rate 1e-5 with scale 20. If you take a learning rate from a PEFT or Unsloth guide, set scale to match in a YAML config. PEFT's default alpha/r is 1.
2. Put the run in a YAML file, because rank, scale, the schedule and the optimizer options only exist there:

The schedule counts optimizer updates, not micro-steps.

1,800 micro-steps with accumulation 4 are 450 updates: 50 for warmup, 400 for the decay. learning_rate is ignored once lr_schedule is set.

This config follows from the code, but I have not trained with it yet.

1. Read the validation loss per answer. Multiply by the masked tokens per example and compare with the log of the number of options. If the bound is above guessing, stop and look.
2. Check the prediction histogram on held-out data, next to the untrained model on the same items. One letter everywhere is visible in five minutes. A loss curve won't show it.
3. Check the prompt-prefix yourself. mlx-lm's chat dataset doesn't verify that the tokenized prompt is a prefix of prompt plus answer. If it isn't, mask_prompt masks the wrong tokens.
4. Prove the adapter is loaded. The same prompt with and without the adapter must give different logprobs. On 0.31.3, send adapters per request or upgrade to 0.32.0.
5. Seed the shuffle yourself with np.random.seed() if you call train_model() directly.

The 9B is training now

Before run 2 was evaluated, I wrote down one rule: if the trained 4B made fewer errors than the untrained 4B's 55, the 9B would train overnight with the same settings.

It made 18, so the 9B started at 17:16. This time the batch order is seeded.

The untrained 9B made 20 errors. Jev made 12, OpenAI's Decisions API 6. I'll add the result here tomorrow. The cost side of running this locally, in Dutch, is in AI lokaal draaien als beslismodel.

What I take from it

The setting I changed was one number. Finding it took reading four files in mlx-lm and one in MLX. A library default is a set of values that belong together, and I changed one of them.

If you want to know whether a small local model can take over a decision in your own process, measured on your own data, that is the kind of work I do: AI consultancy.