Development statistics for the 7.2 kernel

LWN.net needs you!

Without subscribers, LWN would simply not exist. Please consider signing up for a subscription and helping to keep LWN publishing.

Linus Torvalds released the 7.2 kernel on August 17, after noting that the number of fixes coming in was still "bigger than I would have wished for". In fact, 7.2 was one of the busiest development cycles in the kernel's history, adding nearly 600,000 lines of code. It's time to look at some statistics to get a handle on how the kernel's development community is changing.

There were 16,418 non-merge commits brought into the mainline for the 7.2 release, exceeding the (already busy) 7.1 release; it is, in fact, the second-busiest release in the kernel's history. Only the 6.7 release brought in more commits but, while 6.7 was dominated by the merging of the bcachefs development history, the volume for 7.2 is more broad-based.

These commits were contributed by 2,652 developers, a big increase over the previous record (2,479 in 7.1). If one looks at the history of developer involvement since the beginning of the Git era, the result is:

(This plot, like much of the information here, comes from the subscriber-only LWN Kernel Source Database). There has been a clear trend over that entire history: the number of developers contributing to each release increases over time. But the slope of that line has clearly changed.

The most active developers working on 7.2 were:

Most active 7.2 developers
By changesets
---
Uwe Kleine-König
Rosen Penev
Ian Rogers
Johan Hovold
Arnaldo Carvalho de Melo
Chuck Lever
Eric Dumazet
Ville Syrjälä
Johannes Berg
Krzysztof Kozlowski
Thorsten Blum
Namjae Jeon
Christoph Hellwig
Sean Christopherson
Jakub Kicinski
SJ Park
Mike Rapoport
Dmitry Baryshkov
Eric Biggers
Rafael J. Wysocki

|

By changed lines
Matthew Stewart
Hawking Zhang
Miguel Ojeda
Ian Rogers
Darren Ye
Ethan Nelson-Moore
Harry Wentland
Uwe Kleine-König
Ingo Molnar
Chuck Lever
Eric Biggers
Jakub Kicinski
Rodrigo Siqueira
Johannes Berg
Arnd Bergmann
Emil Tsalapatis
Vivek Aknurwar
Rob Clark
Sabrina Dubroca
Svyatoslav Ryhel

|

Uwe Kleine-König [KSDB] was the biggest contributor of commits to the 7.2 release with cleanups through the driver subsystem. Rosen Penev [KSDB] contributed a long list of LLM-authored bug fixes throughout the architecture-specific, driver, networking, and crypto subsystems, among others. Ian Rogers [KSDB] worked extensively on the user-space perf tool, as did Arnaldo Carvalho de Melo [KSDB]. Johan Hovold [KSDB] made numerous improvements, mostly in the SPI and USB subsystems.

Matthew Stewart [KSDB] reached the top of the lines-changed column by following the time-honored tradition of adding more massive, machine-generated amdgpu header files; Hawking Zhang [KSDB] did likewise. Miguel Ojeda [KSDB] brought the Rust zerocopy crate into the kernel source. Darren Ye [KSDB] added the MediaTek mt8196 sound driver.

Just over 8% of the commits in 7.2 carried Tested-by tags, while 48% included Reviewed-by tags. The review tags, in particular, are showing a mild downward trend in recent releases:

Whether that decline reflects an actual decrease in the amount of review taking place is unclear at best. In any case the most active testers and reviews for this cycle were:

Test and review credits in 7.2
Tested-by
---
Dan Wheeler
Luiz Capitulino
James Clark
Sarthak Sharma
Venkat Rao Bagalkote
Tommaso Merciai
David Riley
Jie Gan
Geert Uytterhoeven
Arthur Kiyanovski
Marek Szyprowski
Randy Dunlap
Shivaprasad G Bhat
Rinitha S
Vitaly Prosyak

|

Reviewed-by
Konrad Dybcio
Dmitry Baryshkov
Andy Shevchenko
Geert Uytterhoeven
Krzysztof Kozlowski
Frank Li
Ilpo Järvinen
Christoph Hellwig
Jan Kara
Christian König
Linus Walleij
Alex Deucher
Simon Horman
Jeff Layton
Ido Schimmel

|

Dan Wheeler [KSDB] is, as always, the top tester by virtue of his work with AMD graphics-driver patches. On the review side, Konrad Dybcio [KSDB] added his tag to 231 changes while Dmitry Baryshkov [KSDB] tagged 225; the same two, who both tend to focus on drivers and devicetree files for Qualcomm devices, were at the top of that list for the 7.1 cycle as well.

Work on the 7.2 kernel was supported by 249 employers that we were able to identify; the most active of those were:

Most active 7.2 employers
By changesets
---
(Unknown)
Intel
Google
Red Hat
AMD
Qualcomm
(None)
NVIDIA
Meta
(Consultant)
SUSE
Renesas Electronics
IBM
Kylin
NXP Semiconductors
Oracle
Arm
BayLibre
Microsoft
Bootlin

|

By lines changed
AMD
(Unknown)
Google
Qualcomm
Intel
Red Hat
Meta
(None)
NVIDIA
MediaTek
NXP Semiconductors
BayLibre
Oracle
Microsoft
Renesas Electronics
Linaro
(Consultant)
IBM
Texas Instruments
SUSE

|

The most significant change this time around, arguably, is the significant increase in developers with unknown affiliation. This is partially, at least, a reflection of the ongoing flood of new developers entering the kernel community.

New developers and their tools

Each of the previous two kernel releases had set a record for the number of first-time developers contributing changes; 7.2 has broken that record again, with 613 contributors [KSDB] who had not been seen here before. Here, once again, the trend is clear:

The most active new developer was Bryam Vargas [KSDB], who contributed 43 commits fixing problems in the input, NVMe, F2FS, AppArmor, Landlock, SELinux, NTFS, WiFi, ATA, SCSI, networking, Ceph, and Thunderbolt subsystems, among others. As has been observed here before, it would be difficult for a seasoned kernel developer to reach deeply into that many subsystems over the course of a single development cycle — at least, if they wanted to understand what they are doing. For a previously unknown developer to do this is surprising at best. Vargas's patches do not carry Assisted-by tags, though, so we are left to believe that this work was done without LLM assistance.

Overall, 1,111 changes in 7.2 (under 7% of the total) carried Assisted-by tags — a significant increase over the 301 seen in 7.1. The real count of LLM-assisted patches in 7.2 must surely be far higher than that, though. Many developers do not supply those tags (often — but not always — out of ignorance of the rules), maintainers rarely ask about LLM use, and at least one maintainer actively strips those tags when applying patches. Accurately tracking commits developed with LLM assistance may turn out not to be possible, but it sometimes feels like the kernel community does not care enough to really try.

The other way in which LLM usage is making itself felt in kernel development is in the area of patch review. There are 252 patches in 7.2 with Reported-by tags crediting the Sashiko review system [KSDB]; 568 commits mention Sashiko in their changelogs. Other LLMs are also clearly being used to find kernel bugs, though, with real effects.

The 7.2 release contains 4,830 commits with Fixes tags, indicating that they fix a bug introduced by a previous commit. To see just how significant an increase is, consider this plot, an updated version of one that has been seen here before (most recently for the 6.17 development cycle):

The heavy green line plots how many commits in each release contained Fixes tags, while the heavy brown line is the number of commits that were identified by Fixes tags in subsequent commits. The crossover point is at 5.9, suggesting that, as of that release, the kernel community started fixing more bugs than it introduced. That conclusion will not hold, though; it will take years for all of the post-5.9 bugs to be identified and fixed, and that line will shift as it happens.

The noteworthy part of the plot, though, is at the right-hand side, where the number of commits carrying Fixes tags increases dramatically, after having been roughly level for quite a few releases. That, it seems, is what happens once large numbers of people start turning LLMs toward the task of finding bugs in existing software. This is going to be an interesting trend to watch. The number of bugs cannot be infinite — even in the kernel — so the rate of bug detection and fixing must eventually fall off. The existing bugs will have been fixed, and LLM review will have reduced the number of new bugs introduced. At least, that is the hope.

If that drop-off does not happen, there will be some interesting questions to answer. It seems reasonably clear that the spread of LLMs has enabled people to create kernel patches who would not have been capable of doing so before. If that is true, it is not much of a stretch to conclude that, in many cases, the submitters of these patches do not fully understand what those patches do. Perhaps the LLMs are good enough that this understanding is no longer necessary, but it could also be that we are creating a pile of code that nobody has ever really understood, but which will need to be maintained in the coming years.

The Linux kernel is aimed at production use, not research or experimentation. But there is currently a massive experiment being run on the kernel's development processes and community. This community has absorbed new processes and tools many times over its 35-year history, and will almost certainly find a way to do that this time as well. But that absorption has not always been without turbulence along the way.

Meanwhile, there are nearly 14,000 non-merge changesets sitting in the linux-next repository and poised to be pulled for the 7.3 release; over 1,100 of them contain Assisted-by tags. It would seem that the high level of activity that has characterized recent releases will not be ending anytime soon. As always, keep an eye on LWN as we follow the next development cycle (and those that come after).

Index entries for this article
Kernel
Kernel

---