dlab Open Source Week: Frontier AI on Your Own Hardware — Tim Dettmers

In one of my classes I asked the question I was afraid to ask but I just needed the answer to: “Who is afraid of not getting a job after graduating?” About eighty percent of the 150 people in the room raised their hands. That is roughly 120 students answering, in one motion, that they do not believe there is a place for them in the future.

The other story arrives by email. PhD students who cannot wait to graduate, because they want to join a frontier lab and they have concluded that research in academia is meaningless. They are counting the years until they can leave.

I believe both stories are wrong, and wrong for the same reason. They assume the future of research belongs to whoever has the most GPUs. I think the opposite is true. Academia is probably about to have a renaissance, and the most exciting work of the next decade will happen in university labs — not in spite of their limited resources, but because of them.

This week is our argument for that claim, and we are making it in code rather than in prose.

This post has six parts: why a lab like ours now publishes ecosystems instead of papers; what is actually in this open-source week; why the pessimism I keep running into is mistaken; what to let go of, and what to hold on to; what research will look like once you have let go of it; and why the renaissance happens in academia.

The unit of research is no longer the paper

Something changed in the last year, and most of us have not updated our habits to match it.

With agents, research per projects have become easy and quick. Work that used to take a year of engineering and experimentation now takes weeks, sometimes days. Here is the part that took me longer to see: when every individual project becomes easy, piecemeal work stops being good research. A paper here, a paper there, each one self-contained, each one asking the reader to stitch the pieces together themselves — that is a format from a world where every piece was expensive.

The difficulty did not disappear. It moved. It is no longer hard to publish a paper. It is hard to publish a coherent ecosystem.

The unit of research is the ecosystem.

That is what Open Source Week is for. When my students and I started, we set out to build components that build on each other rather than merely coexist, so that each piece makes the next one more useful. My lab and I believe in using our academic freedom to bring the best AI tools to everyone for free. Something that you can do uniquely at universities. Concretely, that meant building open systems, making models cheaper to run locally, making local models stronger, building local systems that replicate frontier performance in deep and autonomous research, and creating new methods for for building domain-specific reinforcement learning environments.

All of it sits at the intersection of three things: inference-serving frameworks, agent harnesses and work, and the combination of the two into autonomous research systems. And all of it has to be easy to use, because open source that only experienced researchers can run is not open source. Accessibility has two halves — the resources you need and the expertise you need — and only one of them is fixed by hardware. A couple of GPUs, or a MacBook, can be enough. The expertise requirement is a design problem, and you solve it by abstracting away every technical detail the user does not need to think about. That is where most of our effort went, and it is most visible in the agent harness.

I am not going to give away everything before the open-source week starts, so here is what I can tell you now.

If you ask me what a small lab can do today, wee will show you three things: frontier autonomous research, the most efficient test-time scaling I know of, and auto-compaction that is far more efficient than what Claude Code or Codex implement.

Start with the harness, because it is what makes everything else usable.

You have probably heard about agent sessions that run for hours, days, or even weeks. For most people, and especially for anyone who has never worked with agents, it is a mystery how that is achieved. You point our harness at a repository — an inference framework with CUDA kernels, say — and you tell it to optimize the kernels. Then you leave. It keeps improving them through the parts where progress is slow and the work is frustrating, and it keeps going until you come back. No feedback will be provided along the way, so the agent has to figure things out on its own whenever it is unclear or unsure.

That is what we did with the Mac and Metal implementations of our inference framework. One command set the agent loose on the kernels. What came back was quantized inference of a Qwen 3.6 35B-A3B model at 450 tokens per second, with high-quality output at 1.5 bits per weight. A half-precision model needs sixteen bits for every weight; at 1.5 bits, the same model runs in about a tenth of the memory, and it runs fast enough to feel like a local process rather than a remote service.

Then there is the theme in the title of this post. What happens when the models that used to be out of reach fit on the hardware you already own?

Qwen 3.8 at 27 billion parameters has been the popular local model. Our framework lets you run its larger sibling, Qwen 3.8 Flash Next at 125 billion parameters, on a single 24 GB GPU — the card in a normal desktop machine. With AMD Strix, an NVIDIA DGX Spark, or a MacBook with 128 GB of memory, you can run DeepSeek V4.1 — a 550B model. You will not have to manage context length either: compression and context handling are automatic, and inference stays fast even at long contexts.

Then there is the part I am most excited about.

We combined these pieces and pushed further into autonomous research, and on the way we built a new information retrieval technique with a precision I have not seen before. The system beats deep research systems from frontier labs, and it produces better autonomous research results than Sakana AI’s system or Google’s ScientistOne. It runs entirely locally, with no internet access at all.

Using it is simple. Let me give you the experiment I ran.

I asked the agent to find a problem worth working on in the domain of bioinformatics — because I do not know much about it — and the criteria were specific. Progress had to be fast. The evaluation had to be cheap enough to run on the hardware we already had. And it had to be a fresh problem, with active research published in the last four weeks, so that we would be working on something the field has not settled. The agent came back with three problems. We took the first, and within about two hours it had established a new lower bound on heuristic methods, developed and tested the best heuristic method in the literature, moved closer to expensive methods trained with AI models, and found issues in the data sources that everyone uses to evaluate this problem. We did not reach state of the art on the overall problem. Still: two hours of work on a machine in my lab produced four results, and one of them questions the evaluation data the whole area depends on.

The system is not a demo that we trot out for blog posts. My students use it every day. Before it lived inside the harness, it lived in a Slack bot, and it was flaky enough that the bot would go down at times. I did not have an email system that alerts me to the Slack bot going offline, but I had the next best thing: my students often wrote me “Tim, there is something with the slack bot and it does not work anymore. Can you help?” In a collaborative setting I used it after recording a meeting: it generated research questions from the recording, evaluated the ideas discussed against the literature, and sorted the promising directions from the unpromising ones. Then created a google doc and sent it to the students. I did that for two meetings. The students liked it, but it was cumbersome since it had a manual component of me copy pasting two pieces in the pipeline, so I stopped. For the next two meetings I did not use it — and then the students asked me with anticipation if we can again use the system because they found it to be so useful to make sense of their research.

That is the only evaluation of a research tool I trust: people ask for it after you stop giving it to them.

The last piece is the one we use the most and talk about the least. How do you keep an agent working after the conversation would have ended?

Our answer is an auto-compaction technique called CliffCompaction. We have used it in the lab for months, and I, for one, want to never run an agent without it. It is considerably more powerful than the auto-compaction in Claude Code or Codex. Sessions with it run for millions of tokens, and some of mine have run past a hundred million. It also cuts overall cost by about fifty percent. One of our partners deployed it inside their company and measured a forty-five percent reduction in their total AI budget — nearly half of what they spend on AI, gone, without giving anything up. On KernelBench it reaches state of the art, beating methods far more complicated than ours, AlphaEvolve-style approaches and hierarchical memory systems among them, by a wide margin. We will published a strong version. We already parts of the next one autocompaction technique, and it is better.

It moves both sides of the cost/capability trade-off at once: sessions that run longer, and a bill that runs smaller. In other words, the agent stops forgetting what it was doing, and you stop paying for the forgetting.

Long sessions, lower bills.

That brings me to the test-time scaling part of the list, because it falls out of the same trick. Auto-compaction cuts cost by about fifty percent, and you can reinvest the saving: instead of one rollout, buy several with the same budget. We have found the first practical method that turns multiple rollouts into significant improvement at the same cost, and while it is not practical for everyday engineering work yet, the leap to that level will not be difficult. Combined with local deployments, which are often underutilized, we believe this leads to a future where anyone can run many parallel agents on any single problem.

Why the pessimism is wrong

So why are 120 of those 150 students afraid?

Three things are happening at once, and only one of them is about AI. The first is a belief that AI will take everyone’s job. The second is a poor understanding of what AI does to work. The third is contagion: self-defeating ideas spread from person to person, and a room full of people who have heard the same pessimistic sentence ten times will produce an eleventh hand.

Let’s start with what AI does to work, because that is the part we can actually reason about.

Start with the profession everyone expected to go first. The prediction was that software engineers would lose their jobs first, and the recent trend went the other way: demand for software engineers is higher than ever. A software engineer with good agents produces new products, maintains existing systems, and expands them far more efficiently, so a company gets more value for every dollar it spends on that engineer. What the job requires has changed. It needs strong agent skills and, often, deeper specialization than before — the “software engineer” job no longer exists — and both are now within reach: agent skills come with time, and deep specialization, which used to take years, is quick to acquire with agents.

We live in an economy of incremental improvements. The next phone is not much better than the last one; the improvements are real but small, and they get harder to notice every year. Many services have converged the same way. A ride from Lyft or Uber is not a fundamentally different experience than it was, and it probably never will be, because there is not much left to change. You can only make the same thing slightly better so many times before someone stops paying for the next version.

Progress comes in two forms, and they behave nothing alike. Call the pair improvement/capability: improvement makes what you already have slightly better, capability gives you something you did not have at all. An economy that only produces the first kind has a ceiling, no matter how hard everyone in it works. So look at what happens when technology delivers the second kind.

Self-driving cars are the obvious example, and the interesting part is how unevenly they will arrive. Driving in complicated places like Europe or Asia is decades away. But in grid-like structures with well-behaved traffic — much of the United States — it will work much sooner, and it will change a great deal over the next two decades. Robotics is probably on a similar path: robots in households will free up work the way the washing machine did, and the point of freeing up work is not the work. It is the life you get to lead instead.

The same logic applies to the device in your pocket. The next phone might not be much faster, because chips do not get much faster anymore. It might be a very different device.

Take the case that people bring up when they tell me AI makes products worse, because it deserves an honest hearing. It is a cliché by now that bolted-on AI destroys the experience of the product it is bolted onto, and the AI features in Microsoft’s products are almost universally severely disliked. I think that reaction is correct, and I also think it is a verdict on the integration, not on the technology. A well-designed AI experience changes how you interact, how you work, and how you structure your day. That version exists, and you can feel the difference in the places where it has been done well. ChatGPT is the prime example.

Adoption of ChatGPT in the general population has been slow in the US. It is also undeniable that people catch up, and when they catch up it makes a difference in their lives — not a small difference, a structural one. How we produce knowledge and how we consume it will be changed permanently.

That change will bring turmoil and complexity, and I do not want to pretend otherwise. But turmoil is not where the story ends. Demand for new experiences and new products drives revenue, revenue drives hiring, and hiring is what people mean when they say job security. The pessimistic reading stops at the turmoil and forgets everything that follows it.

Let go of how you work. Not who you are.

The future that is coming is a dramatic shift, and for many people it will be shocking, disappointing, and disillusioning. It does not have to be.

The most useful thing you can do right now is to let go. Let go of how you did things. Let go of the sequence you were taught, where you learn the basics and how you do things more generally, finally, the problems. Let go of the idea that your value is stored in what you have already learned, because the tools you learned are being rewritten while you use them.

Letting go of how you do things is not the same as letting go of yourself, and the difference is the whole point. Identity is that which has to stay straight: what you do is negotiable, and who you are is not. Each of us does things to be engaged and to enjoy life, and when life changes, the way you spend your days changes with it — be it a new job, or starting a family, or so many other things. But people stay close to their own personality — not because they are stuck, but because they like it. That is who they are. We are all flawed, we all love someone, and we hold certain things to be important for ourselves and for others. Technology does not touch any of that. Once you are firm on who you are, everything else becomes negotiable, and being able to negotiate everything else is what lets you adapt quickly to whatever comes next.

So what does letting go look like in practice?

If you are an academic, you have to let go of papers. Not of writing them, and not of caring about them — the paper is still how we communicate. What has to go is the paper as the unit of achievement, the thing that gets counted and compared. If the ecosystem is the unit of research, then building something that other people can build on has to count for more than the next increment.

If you are a student, you have to let go of the idea that you first acquire skills and basic knowledge and then solve problems. The order reverses. In the apprenticeship model — which is what a PhD already is, at its best — you do not read a textbook so that you can solve a problem later. You attach the problem, and you learn, build understanding and intuition along the way, and you spend your attention on the hard part instead of the part that can now be looked up.

Hard problems will be more common than ever. Not because the world will be harder, but because everything that is not hard will be automated away. The skill that matters is the one a PhD teaches and almost nothing else does: staying with a problem that does not yield.

That is the skill set worth investing in.

What will research look like, and how do you train for it?

If agents change research this much, what does research look like from here, and how should we train a student to do it?

I have been living with that question for about a year, and I do not have a clean answer but it appears to be dawning on me. Starting a faculty job comes with more responsibilities than doing research as a PhD student, and that leaves less room to dive deeply into a skill and then hand it to my students. But I decided to neglect something to make that time — for working with agents myself, for working out what research with agents should look like, and for finding out what actually makes it productive. That trade is the one I keep making.

After about a year of it, particular ways for PhD students to work have emerged. For example, students should focus on the ecosystem as a unit of work. Students should work on many projects in parallel. Students should embrace the method of attacking a problem first and understanding it as you go.

I am designing a four-week short course at CMU for next week, and a full course for next semester, with the aim of putting the whole thing on YouTube so that anyone can build these agent skills.

With the right agent skills, the future does not look dire. It looks exciting. There is a transition period, and I will not pretend it is comfortable, but once it is behind you the possibilities are endless. Not all of that excitement holds up: in the first weeks of using agents, almost none of it does. What is real arrives later, once you reach a sober understanding of what agents can actually produce. From there the excitement holds up, and it turns into rapid progress in research.

Open Source Week is what that progress looks like.

The renaissance is in academia

So where should you do that work?

My answer is: in a university lab, and sooner than you think.

The reason is the one I started with. Agents let you find and work on problems where you do not need many resources, and yet the impact can be enormous. That space — problems that are cheap to attack and valuable to solve — is vast, and it is completely uncontested, because everyone with resources is competing somewhere else: on scale, on problems that need thousands of GPUs, on the things only the largest labs can attempt. What a small lab has instead is creativity, time, and the freedom to work on problems that frontier labs cannot work on. That turns out to be a different kind of advantage than the one everybody is chasing.

Open Source Week is our attempt to show what we can do when believing in this story — and I think we did well! It has been so much fun to work on all of this with my students. We are a small lab with a couple of GPUs. The open-source week will be delayed by a day (still finish up that draft), but from tomorrow we are putting out two open-source projects and four papers, and the point is that they arrive together, as one package in which each piece makes the others more useful.

It did not happen by plan. The week came together because my students and I want the same thing: to contribute to open source and to build things that people can use — not necessarily products, though some of them are research products, but new knowledge and techniques that did not exist before. I wanted to release four weeks ago. The student projects have been finished for more than a month, and the students waited, patiently, so that we could release everything as one ecosystem instead of scattered announcements over several weeks. Bringing it all together has been far more overwhelming than I expected — coordinating six releases is a different skill from doing research, and it is not one I have practiced. It hope it will be worth it.

If this works, research becomes accessible at a level that has not existed before, and it proves something that I think students badly need to hear right now. A couple of people with a couple of GPUs can build systems that compete with the frontier. The renaissance does not require anyone’s permission. You can just do things.

So the next time I ask 150 students who is afraid of not getting a job, I hope fewer hands go up. The work is there. The future in academia is very bright, and I am excited about the years ahead.

---