Teaching FLUX What I Look Like (Without Losing Its Mind)

The Goal

Here's what I wanted: to see what I would look like if I doubled my weight, without having to eat McDonald's every day for six months.

I'm kidding (mostly). The real goal was to type "Arthur without baldness" and get a picture of me with hairs, not some random hairy dude. But when my friend Vassilis showed me his fine-tuned model generating a morbidly obese version of himself, I knew I had to have one too. Some things you just need to see.

The technical term for this is "personalization": teaching a pre-trained model that a specific word (my name) should map to a specific appearance (my face). You give the model a few dozen photos of yourself, and it learns to associate your name with your visual features: the shape of your face, the way your hair falls, that one expression you apparently make in every single photo.

If you've heard of DreamBooth, this is the same broad idea. DreamBooth typically fine-tunes many weights with a class prior to keep the model general. I took a lighter approach: only adjust a small set of parameters via LoRA adapters. Fewer trainable parameters means less overfitting, less catastrophic forgetting, and critically, training that finishes before my coffee gets cold.

The Dataset Problem

Every ML project eventually becomes a data problem. Mine was no different.

I needed 40-50 photos of myself with good captions. The photos were easy enough (I raided my camera roll and recruited anyone who's ever taken a picture of me). The hard part? The captions.

Writing 50 descriptions like "Arthur standing on the left, wearing sunglasses, sunny day, urban background" sounded like my personal hell. I'd rather debug memory leaks.

Enter Grok (My Low Budget Caption Intern)

So I built a small captioning agent using Grok's vision API. I picked one well-lit selfie with a neutral expression as my reference image, and sent it alongside each photo to caption so the VLM could tell which person in the frame is "Arthur".

Then I tell it some rules:

This isn't about perfect prose but about consistent signals. If half my captions say "a man" and half say "Arthur," the fine-tuning gets confused.

Example Captions

Here are some real examples from my dataset:

"Arthur smiles while holding an American flag in one hand and a colorful Hazy IPA beer can in the other, sitting at a wooden table in a brightly lit kitchen with shelves of kitchenware and supplies in the background."

Very American. Very on-brand.

"Arthur in the center making a funny face with peace sign while posing with two blonde women on a grassy hillside overlooking the Golden Gate Bridge and ocean, sunny daytime selfie with hazy clouds."

When there are other people, the caption explicitly states where I am in the frame. This helps the model learn that "Arthur" refers to a specific person, not just "anyone in the photo."

Fine-Tuning

Alright, enough about data. Let's talk about the fun part: training.

I'm using FLUX.1-dev (from Black Forest Labs) with LoRA adapters, orchestrated via PyTorch Lightning. If that sentence meant nothing to you, don't worry. I'll break it down.

The 30-Second Version (For People Who Have Meetings)

1. Load each image, resize to 1024×1024, normalize to [-1, 1]
2. Encode through the frozen VAE to get latents
3. Add noise to the latents at a random timestep
4. Ask the transformer to predict how to denoise
5. Compute loss between prediction and ground truth
6. Update only the LoRA weights (everything else is frozen)

The Slightly Longer Version (For People Who Skipped the Meeting)

LoRA: Training Millions Instead of Billions

FLUX has about 12 billion parameters. Training all of them on my 50 photos would be:

1. Computationally ridiculous (you'd need serious hardware)
2. Dangerous (the model would forget how to draw anything except my face, this is called "catastrophic forgetting" and yes, it's as dramatic as it sounds)
3. Slow (I have other things to do)

LoRA (Low-Rank Adaptation) sidesteps this elegantly. Instead of updating the original weights, we insert small trainable matrices next to them. With rank 32 and $\alpha = 16$, I'm training about 0.1% of the total parameters. The original model stays intact. I'm just adding a small "adapter" on top.

Looking back, I was probably a bit too conservative with the hyperparameters. We could definitely improve the training speed by using a higher LoRA rank, a more aggressive learning rate, or a larger batch size. But hey, it worked, and I'd rather have a slow training session that succeeds than a fast one that explodes.

We only touch the query, key, value, and output projections in the attention layers. That's where the magic happens. It's how the model learns to associate "Arthur" with my face, glasses, and questionable fashion choices.

Flow Matching: Not Your Grandmother's Diffusion (Or Your 2022 Stable Diffusion)

FLUX uses flow matching instead of the DDPM-style diffusion you might be familiar with. The key difference is in how we corrupt and denoise.

Think of it as a linear interpolation between the clean image and pure noise. The model learns to predict the velocity, the direction from noise to clean image. The loss is just MSE between the predicted velocity and the true velocity.

Guidance: Telling the Model How Hard to Listen

If you've used Stable Diffusion, you've probably cranked up the "guidance scale" slider. Higher values make the model follow your prompt more strictly (at the cost of diversity and sometimes quality). In classic classifier-free guidance, this works by running the model twice (once with your prompt, once without) and interpolating the predictions.

FLUX does something cleverer. Instead of running two forward passes, it embeds the guidance scale directly into the model. The transformer receives the guidance value as an additional input, and learns during pre-training how to modulate its behavior based on this signal.

Training Details

For the hyperparameter nerds (I know you're out there):

Training runs on a single GPU. With gradient checkpointing enabled, it fits comfortably in ~50GB VRAM. No need for a multi-GPU rig or cloud credits that make you cry.

Results

After training, I can prompt the model with fun scenarios:

• "Arthur as a professional boxer"
• "A picture of Arthur as a wizard, casting a spell"
• "Arthur dressed as a 1920s gangster, leaning against a vintage car"

And the outputs actually look like me, not just some generic guy who happens to have a beard.

The model learned what "Arthur" looks like, and it generalizes to new scenarios. I can put myself in situations I've never photographed, and the likeness holds up.

Gallery: Arthur in Every Scenario Imaginable

Here are some of my favorite generations. The model handles everything from professional settings to complete fantasy:

The likeness is remarkably consistent across wildly different scenarios. Boxer Arthur? Still me. Wizard Arthur? Still me (with magic hands). 1920s Gangster Arthur? Still me, but with better fashion sense.

Watching the model learn

One of the most satisfying parts of this project was watching the model gradually learn my face. I saved sample images every 25 epochs, and you can literally see "Arthur" emerge from the noise.

The progression is fascinating:

• Epoch 0 (Initial): A stylized illustration of some random dude. No beard, no glasses, wrong face shape entirely.
• Epoch 100-300: The model starts adding a beard and glasses but the face is still generic.
• Epoch 400-600: The face shape starts to converge.
• Epoch 800+: The likeness stabilizes. More training mostly refines details rather than making dramatic changes.

The Identity Replacement Effect

Here's something I didn't expect: when I regenerate images using the exact same captions from my training data, the model faithfully recreates the scene but replaces other people with strangers.

Original training image with my girlfriend:

Regenerated with the same caption:

The model learned "Arthur posing with a female friend in a glass elevator" but it has no memory of who that friend was. It just generates a plausible person who fits the description.

Same thing happens with my best friend at the Acropolis:

Original:

Regenerated:

This makes sense when you think about it: my friend only appears in one or two training images, while I appear in all 50. The model has learned a robust representation of "Arthur" but treats everyone else as interchangeable background characters.

If I wanted the model to remember my girlfriend or friends, I'd need to fine-tune on them too. Give them their own trigger words and enough training examples.

A bit of math

Feel free to skip this section if equations make you sleepy.

LoRA

Instead of updating the original weights, we insert small trainable matrices next to them:

$W$ is the frozen original weight matrix, $B$ and $A$ are small trainable matrices of rank $r$, and $\alpha$ is a scaling factor.

Flow matching

Given a clean latent $z$ and noise $\epsilon \sim \mathcal{N}(0, I)$, we linearly interpolate to build a noisy version $z_\sigma = (1 - \sigma)z + \sigma \epsilon$, where $\sigma = 0$ is the clean image and $\sigma = 1$ is pure noise. The model learns to predict the velocity $v = \epsilon - z$, with the weighted MSE loss:

The weighting $w(\sigma)$ emphasizes different noise levels. The rest is engineering: gradient accumulation, mixed precision, checkpointing.

Final Thoughts

What's Next

I'm planning to:

• Add grounding examples: Include some images generated by the base model (without "Arthur") during training. This should prevent the model from thinking all men are me.
• Combine LoRAs: Try stacking my identity LoRA with style LoRAs. Can I make a Studio Ghibli Arthur? An oil painting Arthur? Or better yet, fine-tune a model on one of my friends and stack both identity LoRAs. Can I generate Arthur and his friend together in new scenarios? We'll see.

The Recipe

If you want to try this yourself, here's the recipe:

1. Collect 30-50 diverse photos of yourself (different angles, lightings, backgrounds, outfits)
2. Caption them well (use a VLM with a reference image for consistency)
3. Fine-tune with LoRA (rank 32 is a good starting point)
4. Be patient

The math is elegant, but the results depend on data quality more than anything else. Spend more time on your dataset than on hyperparameter tuning.

And if this works for you, you'll have a model that can generate you wearing a tuxedo, insanely obese, or boxing with your own name on your belt. The possibilities are limited only by your imagination (and the model's ethical guardrails).

Got questions? Want to see more examples? Feel free to reach out!