Many readers will be aware that an unusually high number of features have been reverted from the PostgreSQL 19 branch after the beta period began. Having some reverts is not unusual, maybe one per cycle could be expected. But this time around, about 8 to 10 significant features, depending on how you count, have been reverted, and some of them quite late in the beta period. I have been on the receiving end of some of that, as the developer or committer of some of those features, and have gotten some questions about it, and so I want to take a moment to reflect on this. I’ll try to extrapolate a little bit to similarly affected work that I was not actively involved in, but these are just my opinions at this point.
So what happened?
I don’t think we, meaning all the developers affected, have had enough time to fully reflect on all this. But here are some thoughts. It’s probably a combination of these, and a different combination in the case of each reverted feature.
Maybe we just didn’t do a good enough job. Maybe the code was just not good enough. We’ll learn and improve. It’s probably some of that, and I’m mentioning it here to not give the impression that it was only the other factors that I’m going to mention.
It could also be a coincidence. Maybe we had been on such a good roll that we attempted to tackle several features that turned out to be too hard, or at least too hard to finish in time.
Some of the reverted features had significant architectural defects that would have been hard to fix quickly and during beta. But there was also a long tail of relatively harmless issues involving various edge cases. In past times, many of these would not have surfaced until much later and we would have fixed them bit by bit over the subsequent five years. But with LLM-assisted code review, these kinds of things can get found much faster, and then you’re staring at a list of like forty defect reports. And even if each of them requires only a three-line fix on average, there is a fixed overhead to dealing with each of these, and then you run out of time. Most if not all of the affected reverted features were written a long time ago when LLM-assisted code review was not as useful as it is now, meaning that the effective quality bar has been raised after the time the feature was written and committed. Unless the capabilities of LLM-assisted code review keep improving at the current rate, this situation should sort itself out for future development cycles.
A corollary of that is that, at least for the most part, as far as I can tell, this situation is not due to “vibe coding”. This code was written well before that was a possibility.
Because of the Vulnpocalypse, many senior developers were busy between, say, April (the start of feature freeze for PostgreSQL 19, and also the start of the Vulnpocalypse) and August (the most recent PostgreSQL security release) with working on fixes for security issues. Because of that, the usual additional review and testing that happens during the beta period was reduced, and then picked up again in August. That’s possibly why a lot of new issues were discovered quite late in the beta period and why so many reverts happened so late, which is what made all of this so notable. Again, some issues were more fundamental but there was also a long tail of smaller issues, and I feel with more time, say, two months, these could have been addressed (given adequate focus, but note that the Vulnpocalypse is ongoing). But the community preferred trying to stick to the release schedule rather than delaying to get more fixes in. (And you never know how long a long tail really is.)
The complexity of the system has made some kinds of features extremely complex to get right, and there is not enough scaffolding in the code to support that. Consider, as an abstracted example, some SQL-level query language feature. Does it work with domains? How about with domains over a composite type? Where one of the fields is also a domain? And the domain has a not null constraint? And one of the columns was dropped and re-added? And this is part of a table partition that has been detached and reattached with a different column order? And this is part of a view that is called from a security-definer function from a trigger? And so on. No one can manually test this or even enumerate these test cases. But LLM-assisted fuzzing can find problems with this really quickly. So this probably needs to be part of feature development in the future. But we should also think about making the internal interfaces more robust so that each different combination of contexts doesn’t open up so many new possibilities of interactions.
I like to say, in PostgreSQL, “everything works with everything”. This is part of the appeal, that you can construct cases like this and expect them to work. I don’t want to give up on that at all. But it does impose a significant penalty on feature development that everyone needs to be aware of.
As a final point, the PostgreSQL development community has been contemplating for some time the possibility of committing features with some kind of “experimental” status and then maturing them in the main tree. This would also be one possibility to address the previous point. This would have been a helpful tool for at least some of the reverted features, but we haven’t yet figured out a way to do this, and some of the work suffered from misunderstandings related to this. This is a concrete point to work on.
So what’s next?
First of all, from my personal perspective, thank you to everyone who has reached out in person or online to express support and encouragement. That’s the most important thing. I’m confident that many of the reverted features will come back before too long.
PostgreSQL 19 will still be a great release. REPACK CONCURRENTLY, logical replication of sequences (itself a previously reverted feature), and planner hinting with pg_plan_advice, just to pick a few headline items, will make this release very attractive. And it will be very robust, after all the additional scrutiny.
It’s difficult to predict things now. A few months ago I said, PostgreSQL 20 might have no new features, just bug fixes. It turned out that things moved even faster, and PostgreSQL 19 now, kind of, has half as many major features as then expected, but now we also already have a few nice 80%-ready features queued up for PostgreSQL 20 that just need those remaining 80% to get done!
Instagram Engineering
Shopify Engineering