Here are instructions how to backport patches, listed as missing backports by syzbot, to stable kernel versions.
Start by reading the documentation.
For stable kernel rules, read the official documentation here.
For backporting and conflict resolution more specifically, read the page here.
You will want to test the C / syzbot reproducers with QEMU. Set it up in your local development environment. Instructions can be found here and here and here.
Also, a comfortable level with git is essential.
Stable kernel versions can be found here.
Clone the Linux kernel stable tree. All trees are listed here. Pick the line that says stable — click browse — select summary tab — look for the git clone URL.
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-stable
Once you have the stable tree cloned, you can checkout different releases by listing the branches and checking out a specific version.
git branch -a
git checkout remotes/origin/linux-6.1.y
You may want to clone the stable tree to several local folders. It makes it easier to compare git history for a couple of stable kernel versions at the same time.
In addition to browsing git logs locally, there is an online tool to browse Linux kernel's version history here.
As mentioned, the list of missing backports for Linux kernel can be found here.
Find a commit to backport from that list. There are a few things to take into account while doing that. Check the following before spending significant amount of time on a commit:
git log --grep 'reverts commit 09c5ba0aa2fcfdadb17d045c3ee6f86d69270df7'
dfn:drivers/usc/core/driver.c dfa:scnprintf
In short, a patch has to be in mainline Linux kernel before it can be backported to stable kernel versions. Similarly, a patch has to be in newer stable kernel versions first, before it can be applied to older versions.
For the above reason, if you have picked a commit to backport to stable kernel version 5.15, you need to check that the commit is present in all newer versions. At the time of writing this, the stable kernel versions are: 6.12, 6.6, 6.1, 5.15, 5.10 and 5.4, so you would look for the commit (or backport commit) in versions 6.12, 6.6. and 6.1 before sending a patch to 5.15.
Once you know which versions have the commit, you can start working on the backport. For example, if you find the commit in versions 6.12 and 6.6, but not in versions 6.1. or 5.15, you start the backporting with version 6.1.
Send an email to syzbot to see if the issue is present, without any changes. This step is optional, but especially useful if it's been a while since the bug has been reported.
Include the following line in the email body, with stable repo address and latest commit hash of the stable release:
#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git 0bc96de781b4da716c8dbd248af4b26d8d8341f5
More instructions for how to communicate with syzbot here.
You should receive a reply from syzbot within an hour (sometimes in less than that, sometimes more). If syzbot can reproduce the issue, the next step is to reproduce it locally. You can use the image from syzbot assets to reproduce it, but you also need to be able to reproduce it with custom built kernel.
Checkout stable kernel version you want to backport to, download and copy
the .config file from syzbot assets to the source folder,
and compile the kernel with that.
sudo cp ~/backports/71309069d4a84523ff89/config ~/source/linux-development/linux-stable/.config
cd ~/source/linux-development/linux-stable
sudo make olddefconfig
sudo make -j8
su -c "make modules_install install"
The output from that is an image. Copy the path and set it to environment variable in a new terminal window.
export KERNEL_IMG=/boot/vmlinuz-5.10.242
export RFS_IMG=/home/hannelotta/backports/71309069d4a84523ff89/disk-98f47d0e.raw
Start QEMU.
qemu-system-x86_64 -m 2G -kernel ${KERNEL_IMG} -append "console=tty0 root=/dev/sda1 earlyprintk=serial net.ifnames=0" -drive file=${RFS_IMG},format=raw -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 -net nic,model=e1000 -enable-kvm -nographic -pidfile vm.pid 2>&1 | tee vm.log
Open a new terminal window. Compile the C reproducer, SSH into QEMU and execute the program.
gcc -o repro repro.c
scp -P 10021 -o "StrictHostKeyChecking no" -r ./repro root@localhost:~/
ssh root@localhost -p 10021 -o "StrictHostKeyChecking no"
./repro
Check the QEMU terminal window to see the results.
If you managed to reproduce the issue with custom built stable kernel, you can apply the commit.
In the stable kernel source folder, check that the commit is found:
git show cc249e4cba9a6002c9d9e1438daf8440a160bc9e
If you can't find the commit, add mainline kernel as another remote in your stable repository.
git remote add mainline git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Now git show should find the commit.
Apply the commit by cherry-picking it.
git cherry-pick -s cc249e4cba9a6002c9d9e1438daf8440a160bc9e
If the commit applies as is, this is easy. If it doesn't apply — as is more likely —
this is the part where you study the code, understand what it does, and
resolve any conflicts. Use git add and git cherry-pick --continue
until the conflicts have been resolved.
When the commit has been applied, test if it fixes the issue. Compile the kernel again, start QEMU, and execute the reproducer.
If the issue does not reproduce locally any longer, you can send the patch to syzbot.
Create the patch file using git format-patch -1.
Add the same line to the email body as earlier, and include the patch file
as an attachment in the email.
If syzbot replies and no issues were triggered, looks like the fix works.
If the issue is still present, or another issue is being triggered, additional patches might be required, or something went wrong with applying the commit.
In addition to syzbot testing, test the change locally. Boot the
stable kernel version with your local .config file,
because the .config file provided by syzbot most likely
will not work as is in your environment. Execute the changed code paths in
some way. Use debugging tools, tracing, logs and tests, and / or other tools
and commands to make sure. This part is not trivial and will be different
depending on the kernel area.
If all looks good, prepare the commit for sending as a backport patch.
Same patch submitting rules apply for backports than for mainline Linux kernel patches, with additional details for backports.
Add the upstream commit ID to the commit message, which is required by stable kernel rules. You can do this by amending the commit manually.
git commit --amend
Type [ Upstream commit <commit-id> ] as the first line
in the commit message (that is after the commit title and one empty line).
If you resolved any conflicts, include a short summary of those on the lines
above your Signed-off-by line. This is also required, and gives
context and confidence when the stable patches are reviewed. If you're
unsure what to put in the summary, it's better to include too much detail
than be too vague in the summary section.
Create a patch, and include the stable kernel version in the prefix.
git format-patch -1 --subject-prefix="PATCH 6.1.y"
Check the patch.
./scripts/checkpatch.pl --git <commit-id>
In the mainline kernel source folder, execute the script to get maintainers. This is to get the latest information about maintainers.
./scripts/get_maintainer.pl <patch-filename>
Send the patch to stable kernel mailing list, CC maintainers and mailing lists.
git send-email --to=stable@vger.kernel.org --cc=maintainer@example.com,linux-kernel@vger.kernel.org <patch-filename>
Note that if you need to backport the commit to several stable versions, repeat these steps for all the versions. You may want to have all the patches applied, tested and ready before sending any of them to the mailing list.
Signed-off-by line.
./scripts/checkpatch.pl.
git format-patch.
./scripts/get_maintainer.pl from mainline repo.
git send-email.