TypeSafe.AI recently launched its first model, Jev. They claim it’s the first model of a new kind (a System One model) that is fundamentally different from the LLMs we’ve been working with (and hyping up) over the last several years.
This new model looks particularly well suited to many everyday use cases, such as classification (e.g. topic modelling for NPS comments) or LLM-as-a-judge tasks. So, naturally, I decided I had to try it out.
In this article, we’ll look at what makes System One models different from LLMs and put some of TypeSafe.AI’s claims to the test in practice, using intent classification for customer support requests as an example.
How System One models differ from LLMs
While LLMs are trained to predict the next token and generate the long-form text and conversations we’ve all been enjoying, System One models are built to evaluate a state and produce structured answers. Similar to LLMs, System One models can take natural language as input, so there’s no real difference there.
You can find more details about the training process in the documentation. The interesting part about Jev is its post-training approach. LLMs are mostly post-trained with RLHF (Reinforcement Learning from Human Feedback), which teaches them to align with human preferences. But this can also lead to sycophancy and confident-sounding hallucinations that we’ve all seen in practice. That works well for chatbots, but for use cases that involve decision-making, TypeSafe suggests a different approach.
For Jev, they use RLCD (Reinforcement Learning for Calibrated Decisions), which trains the model to return both decisions and probabilities.
Let’s look at how the model works and what its input and output look like.
The input to the model is called a state. The state defines the context you want to provide to the model together with the questions you want it to answer. It can be as simple as a single message, for example a customer request like “Is it possible for me to change my PIN number?”, or it can be a JSON object.
a map with several fields, such as {"message": "Is it possible for me to change my PIN number?", "user_id": 123}
an array representing a sequence of messages or records, such as ["Hello! How can I help you?", "Is it possible for me to change my PIN number?"]
The best practice is to use an object for the state with clear field names, so it’s easier for the model to reason about the context.
Along with the context, we can also pass one or several questions to the model. Since System One models don’t generate free-form responses, we need to specify the expected type of answer using one of the available primitives:
Choice works when the answer should be one of several predefined options. For example: what is this customer request about — delivery, billing, or account access?
Score can be used when the answer comes from an ordered set of values. For example: what is the sentiment of this customer message — negative, neutral, or positive?
Noul can be used for yes/no questions. For example: has the customer’s problem been solved in this chat?
In all of these cases, we get not only the answer itself, but also probabilities for all possible values. Confidence is an important part of Jev, because it tells us whether the model is confident in its decision (for example, when most of the probability is concentrated on one value) or uncertain, when several options have roughly similar probabilities.
These confidence levels can be particularly useful when we need to make decisions based on the model output. Take auto-replies to customer questions as an example: if the model can classify a customer request into one of the known categories with high confidence, we can send an automatic reply. Otherwise, we can route the message to a human support agent.
Practice
Jev claims to be 193.6× faster, 444.6× cheaper, and less prone to hallucinations. Let’s see how those claims hold up in practice by comparing it with our good old LLMs.
For this experiment, I chose OpenAI, since TypeSafe’s benchmark shows Jev performing on par with OpenAI’s Terra model.
I also decided to use a publicly available dataset of banking intents released by PolyAI under the CC BY 4.0 licence. It’s quite an interesting classification problem, with 77 different intent classes (which is a lot).
Here’s a sample of the data.
Image by author
Using Jev
Let’s start by making a call to the Jev model. To get access, you’ll need to register (there’s currently a waitlist) and obtain an API key. In my case, it took about half a day to receive an invite.
For our classification task, we’ll use the Choice primitive for the question. We don’t have any additional descriptions for the categories, so we’ll leave those as null.
From there, we just need to make an HTTP request and pass all the required information (the state and the question primitive).
As a result, we get a JSON object containing the answer and the probabilities for all possible options. In this case, we can see that the model is absolutely confident that the correct answer is card_linking.
Using OpenAI
For comparison, we’ll use OpenAI’s Luna and Terra models. To make the setup comparable, we’ll also specify an output schema for the OpenAI models, so their responses are constrained to the same structured format.
Unfortunately, OpenAI models don’t currently return log probabilities, so there’s no straightforward way for us to get a confidence score from the model and compare it directly with Jev.
Comparison
TypeSafe positions Jev as roughly on par with Terra and slightly ahead of Luna, while being substantially faster and cheaper, especially for workflows involving multiple decisions or model calls.
Let’s see how that translates to our use case, even though this task is fairly simple and requires just a single call. On accuracy, Jev performs noticeably worse than both OpenAI models: 79.0%, compared with 83.9% for Terra and 86.2% for Luna. The difference is statistically significant.
Image by author
We can also see that Jev uses significantly more tokens (about 2× more input tokens and 40× more output tokens) largely because it returns probabilities for all 77 intents. So even with the lower per-token pricing, I’m not convinced it ends up being dramatically cheaper than Luna for this particular use case.
However, the speed improvement is significant indeed: Jev is almost 2× faster.
Image by author
Image by author
What’s really impressive is how well calibrated the confidence scores are: accuracy consistently increases for higher-confidence buckets.
Image by author
I also experimented a bit to understand why Jev wasn’t performing as well on this task. My best guess is that the large number of classes was the main issue. When I reduced the task from 77 labels to just 7, the results improved significantly and were roughly on par with the OpenAI models.
Image by author
You can find all the code on GitHub.
We’ve looked at how this new model works and put it into practice, so it’s time to wrap up and summarise the experience.
Summary
I really like the direction System One models are taking, because I can see them being useful for quite a few tasks I deal with at work, such as LLM-as-a-judge evaluations or intent classification.
As we’ve seen, the quality isn’t always on par with LLMs, but the well-calibrated confidence scores are a big advantage. They could make it possible to use a fast and cheap model for easier cases, while routing more uncertain ones to frontier models. I’d be very interested in trying this kind of setup in a real production workflow.
At the same time, TypeSafe’s headline claims make these models sound almost miraculous (100×+ cheaper and faster). I can believe that this is achievable for some workflows, especially those involving many small decisions and repeated calls, but I’d still expect the real-world gains to vary quite a lot by use case. So it’s worth testing on your own task rather than assuming the benchmark numbers will translate directly.
Thank you for reading. I hope this article was insightful. Remember Einstein’s advice: “The important thing is not to stop questioning. Curiosity has its own reason for existing.” May your curiosity lead you to your next great insight.
Instagram Engineering
Shopify Engineering