, as in life, it’s important to know what you’re working with. Python’s dynamic type system appears to make this difficult at first glance. A type is a promise about the values an object can hold and the operations that apply to it: an integer can be multiplied or compared, a string concatenated, a dictionary indexed by key. Many languages check these promises before the program runs. Rust and Go catch type mismatches at compile time and refuse to produce a runnable binary if they fail; TypeScript runs its checks during a separate compile step. Python does no checking at all by default, and the consequences play out at runtime.
In Python, a name binds only to a value. The name itself carries no commitment about the value’s type, and the next assignment can replace the value with one of a completely different kind. A function will accept whatever you pass it and return whatever its body produces; if the type of either is not what you intended, the interpreter will not say so. The mismatch only surfaces as an exception later, if at all, when code downstream performs an operation the actual type doesn’t support: arithmetic on a string, a method call on the wrong kind of object, a comparison that quietly evaluates to something nonsensical. This leniency is often in fact a strength: it suits rapid prototyping and the kind of exploratory, notebook-driven work where the shape of a value is something you discover as you go. But in machine learning and data science workflows, where pipelines are long and a single unexpected type can silently break a downstream step or produce meaningless results, the same flexibility becomes a serious liability.
Modern Python’s response to this is type annotations. Added to Python in version 3.5 via PEP 484, annotations are syntax for specifying the types you intend. A function gets type information by attaching it to its arguments and return value with colons and an arrow:
def scale_data(x: float) -> float:
return x * 2
The annotation is not enforced at runtime. Calling scale_data("123") raises no error in the interpreter; the function dutifully concatenates the string with itself and returns "123123". What catches the mismatch is a separate piece of software, called a static type checker, which reads the annotations and verifies them before the code runs:
scale_data(x="123") # Type error! Expected float, got str
Static checkers surface type annotations directly in the editor, flagging mismatches as you write. Alongside established tools like mypy and pyright, a newer generation of Rust-based checkers (Astral’s ty, Meta’s Pyrefly, and the now open-source Zuban) are pushing performance much further, making full-project analysis feasible even on large codebases. This model is deliberately separate from Python’s runtime. Type hints are optional, and checking happens ahead of execution rather than during it. As PEP 484 puts it:
“Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.”
The reason is historical as much as philosophical. Python grew up as a dynamically typed language, and by the time PEP 484 arrived there were decades of untyped code in the wild. Making hints mandatory would have broken that overnight.
A type checker does not execute your program or enforce type correctness while it runs. Instead, it analyses the source code statically, identifying places where your code contradicts its own declared intent. Some of these mismatches would eventually raise exceptions, others would silently produce the wrong result. Either way, they become visible immediately. A mismatched argument that might otherwise surface hours into a pipeline run is caught at the point of writing. Annotations make a function’s expectations explicit: they document its inputs and outputs, reduce the need to inspect its body, and force decisions about edge cases before runtime. Once you’re used to it, adding type annotations can be highly satisfying, and even fun!
Making structure explicit
Dictionaries are the workhorse of Python data work. Rows from a dataset, configuration objects, API responses: all routinely represented as dicts with known keys and value types. TypedDict (PEP 589) provides a lightweight way to write such a schema down:
from typing import TypedDict
class SensorReading(TypedDict):
timestamp: float
temperature: float
pressure: float
location: str
def process_reading(reading: SensorReading) -> float:
return reading["temperature"] * 1.8 + 32
# return reading["temp"] # Type error: no such key
At runtime, a SensorReading is just a regular dict with zero performance overhead. But your type checker now knows the schema, which means typos in key names get caught immediately rather than surfacing as KeyErrors in production. The PEP highlights JSON objects as the canonical use case. This is a deeper reason TypedDict matters in data work: it lets you describe the shape of data you do not own, such as the responses that come back from an API, the rows that arrive from a CSV, or the documents you pull from a database, without having to wrap them in a class first. PEP 655 added NotRequired for optional fields, and PEP 705 added ReadOnly for immutable ones, both useful for nested structures from APIs or database queries. TypedDict is structurally typed rather than closed: by default a dict can carry extra keys you didn’t list and still satisfy the type, which is a deliberate choice for interoperability but occasionally surprising. PEP 728, accepted in 2025 and targeting Python 3.15, lets you declare a TypedDict with closed=True, which makes any unlisted key a type error.
Categorical values are another kind of implicit knowledge that data science code carries around constantly. Aggregation methods, unit specifications, model names, mode flags: these often live only in docstrings and comments, where the type checker cannot reach them. Literal types (PEP 586) make the set of valid values explicit:
from typing import Literal
def aggregate_timeseries(
data: list[float],
method: Literal["mean", "median", "max", "min"]
) -> float:
if method == "mean":
return sum(data) / len(data)
elif method == "median":
return sorted(data)[len(data) // 2]
# etc.
aggregate_timeseries([1, 2, 3], "mean") # fine
aggregate_timeseries([1, 2, 3], "average") # type error: caught before runtime
A small note on syntax. list[float] here is the modern form for what older code wrote as typing.List[float]. PEP 585 (Python 3.9+) made the standard collection types generic, which means the lowercase built-ins now do the same job without needing an import from typing. The capitalised versions still work, but most modern code has moved to the lowercase forms, and the examples in this article do too.
Returning to Literal, it is most useful deep in a pipeline, where a typo like "temperture" might not raise an exception but will produce silently wrong results. Constraining the allowed values catches these mistakes early and makes valid options explicit. IDEs can also autocomplete them, which reduces friction over time. Unlike most types, which describe a kind of value (any string, any integer), Literal describes specific values. It is a simple way to make “this must be one of these options” part of the function signature.
When a structure becomes complex enough that the type itself is hard to read at a function signature, type aliases can bring much needed concision:
from typing import TypeAlias
# Without aliases
def process_results(
data: dict[str, list[tuple[float, float, str]]]
) -> list[tuple[float, str]]:
...
# With aliases
Coordinate: TypeAlias = tuple[float, float, str] # lat, lon, label
LocationData: TypeAlias = dict[str, list[Coordinate]]
ProcessedResult: TypeAlias = list[tuple[float, str]]
def process_results(data: LocationData) -> ProcessedResult:
...
An alias can also clearly document what the structure represents, not just what Python types it happens to be composed of. This pays dividends when someone tries to read the code six months later (and that someone will often be you!).
Making choice explicit
Real data and real APIs rarely deliver one type and one type only. A function might accept a filename or an open file handle. A configuration value might be a number or a string. A missing field might be a value or None. Union types let you say so directly:
from typing import TextIO
def load_data(source: str | TextIO) -> list[str]:
if isinstance(source, str):
with open(source) as f:
return f.readlines()
else:
return source.readlines()
The | syntax was added by PEP 604 and is available from Python 3.10. Older code uses Union[str, TextIO] from the typing module, which means exactly the same thing.
By some margin the most common union is the one where None is one of the alternatives. Measurements fail, sensors aren’t installed yet, APIs return incomplete responses, and a function that returns either a result or nothing is everywhere in data work. The modern way to write it is float | None:
def calculate_efficiency(fuel_consumed: float | None) -> float | None:
if fuel_consumed is None:
return None
return 100.0 / fuel_consumed
The type checker will now flag any code that tries to use the return value as a definite float without first checking for None, which prevents a large class of TypeError: unsupported operand type(s) crashes that would otherwise have surfaced at runtime.
An older syntax, Optional[float], means exactly the same thing as float | None and shows up everywhere in pre-3.10 code. The name is worth pausing on, though, because it’s easy to misread. It sounds like it describes an optional argument, one you can leave out of a call, but it actually describes an optional value: the annotation permits None as well as the named type. These are different properties, and both exist in Python:
def f(x: int = 0): # argument is optional; value is *not* Optional
def f(x: int | None): # argument is required; value is Optional
def f(x: int | None = None): # both
The misreading was severe enough to shape later PEPs. PEP 655, when it added NotRequired for potentially-missing keys in a TypedDict, considered and rejected reusing the word Optional on the grounds that it would be too easy to confuse with the existing meaning. The X | None syntax sidesteps the problem entirely.
Once you’ve declared a parameter as float | None, the type checker becomes precise about what you can do with the value. Inside an if value is None branch, the checker knows the value is None; in the else branch, it knows the value is float. The same “type narrowing” happens after an assert value is not None, an early raise, or any other check that rules out one of the alternatives.
def calculate_efficiency(fuel_consumed: float | None) -> float:
if fuel_consumed is None:
raise ValueError("fuel_consumed is required")
# Inside this block, the type checker knows fuel_consumed is float
return 100.0 / fuel_consumed
When the checker genuinely cannot determine a type, typing.cast() lets you override it. The most common case is values arriving from outside the type system. For example, json.loads() is annotated to return Any, because it can produce arbitrarily nested combinations of dicts, lists, strings, numbers, and None, depending on the input. If you know the expected shape of the data, cast lets you assert that knowledge to the checker:
from typing import cast
raw = json.loads(payload)
user_id = cast(int, raw["user_id"]) # The type checker now treats user_id as an int.
cast does not convert the value or check it at runtime; it simply tells the type checker to treat the expression as a given type. If raw["user_id"] is actually a string or None, the code will proceed without complaint and fail later, just as if no annotation had been present. For that reason, frequent use of cast or # type: ignore is usually a sign that type information is being lost upstream and should be made explicit instead.
Making behaviour explicit
Data work involves passing functions as arguments constantly. Scikit-learn’s GridSearchCV takes a scoring function. PyTorch optimisers take learning-rate schedulers. pandas.DataFrame.groupby().apply() takes whatever aggregation function you hand it. Homegrown pipelines often compose preprocessing or transformation steps as a list of functions to be applied in sequence. Without annotations, a signature like def build_pipeline(steps): is silent about what steps should look like, and the reader has to guess from the body what shape of function will work.
Callable lets you specify what arguments a function takes and what it returns:
from typing import Callable
# A preprocessing step: takes a list of floats, returns a list of floats
Preprocessor = Callable[[list[float]], list[float]]
def build_pipeline(steps: list[Preprocessor]) -> Preprocessor:
def pipeline(x: list[float]) -> list[float]:
for step in steps:
x = step(x)
return x
return pipeline
The general form is Callable[[Arg1Type, Arg2Type, ...], ReturnType]. When you genuinely don’t care about the arguments and only the return type matters, Callable[..., ReturnType] accepts any signature, which is occasionally useful for plug-in interfaces, though most of the time being specific is the point. Callable does have limits. It can’t express keyword arguments, default values, or overloaded signatures. When you need to type a callable with that level of detail, Protocol can do the job by defining a __call__ method. But for the overwhelmingly common case of “a function that takes X and returns Y”, Callable is the right tool and reads cleanly at the signature.
Duck typing is one of the things that makes Python feel fluid: if an object has the right methods, it can be used in a given context regardless of its inheritance hierarchy. The trouble is that this fluency disappears at the function signature. Without type hints, a signature like def process(data): tells the reader nothing about what operations data must support. A typed signature using a concrete class like def process(data: pd.Series): rules out NumPy arrays and plain lists, even if the implementation would happily accept them.
Protocol (PEP 544) resolves this by typing structurally rather than nominally. The type checker decides whether an object satisfies a Protocol by inspecting its methods and attributes, not by walking up its inheritance chain. The object never has to inherit from anything, or even know the Protocol exists.
from typing import Protocol
class Summable(Protocol):
def sum(self) -> float: ...
def __len__(self) -> int: ...
def calculate_mean(data: Summable) -> float:
return data.sum() / len(data)
import pandas as pd
import numpy as np
calculate_mean(pd.Series([1, 2, 3])) # ✓ type checks
calculate_mean(np.array([1, 2, 3])) # ✓ type checks
calculate_mean([1, 2, 3]) # ✗ type error: lists have no .sum()
pd.Series doesn’t inherit from Summable, and neither does np.ndarray. They satisfy the protocol because they have a sum method and support len(). A plain Python list does not, since sum on a list is a free function rather than a method, and the type checker catches that distinction precisely. The shift from nominal to structural typing is small in syntax and substantial in spirit. Nominal types describe what an object is; structural types describe what it can do. Protocol lets you ask whether an object can do something, which is almost always the question that matters in data work, without committing to what it is.
Two practical points are worth knowing. The standard library already ships many of the protocols you’d actually want, in collections.abc and typing: Iterable, Sized, Hashable, SupportsFloat, and a long list besides. You’ll find yourself importing these far more often than defining your own. The other point is about runtime behaviour: protocols are erased by default, which means isinstance(x, Summable) will raise unless the protocol is decorated with @runtime_checkable. The default reflects a deliberate trade-off, since structural checks at runtime are slow, and the design assumes most uses are at type-check time. When you do need isinstance against a Protocol, the decorator is a single line and the cost is paid only where you ask for it.
Data science is largely about transformations, and a well-typed transformation preserves information about what’s flowing through it. The challenge is expressing “whatever type comes in, the same type comes out” without resorting to Any, which simply switches the type checker off for that variable. TypeVar is the construct that addresses this:
from typing import TypeVar
T = TypeVar('T')
def first_element(items: list[T]) -> T:
return items[0]
x: int = first_element([1, 2, 3]) # ✓ x is int
y: str = first_element(["a", "b", "c"]) # ✓ y is str
z: str = first_element([1, 2, 3]) # ✗ type error: returns int, not str
T is a type variable: a placeholder that the checker resolves to a concrete type at the call site. Calling first_element([1, 2, 3]) binds T to int for that call, and the return annotation T is read as int accordingly. Call it with a list of strings, and T becomes str. The link between input and output is preserved without committing the function to any particular type. Once you have a way to say “the type that came in is the type that goes out”, reaching for Any becomes a visible admission rather than a default. Generic typing pushes you, gently, toward writing functions that actually preserve their input shape, rather than ones that quietly lose it somewhere in the middle.
For reusable pipeline stages, this extends naturally to generic classes:
from typing import Generic, Callable
T = TypeVar('T')
class DataBatch(Generic[T]):
def __init__(self, items: list[T]) -> None:
self.items = items
def map(self, func: Callable[[T], T]) -> "DataBatch[T]":
return DataBatch([func(item) for item in self.items])
def get(self, index: int) -> T:
return self.items[index]
batch: DataBatch[float] = DataBatch([1.0, 2.0, 3.0])
value: float = batch.get(0) # type checker knows this is float
Completely unconstrained TypeVars are rarer in practice than you might expect. Often you want to say “any numeric type” or “one of these specific types”, and TypeVar accommodates both: TypeVar('N', bound=Number) accepts Number and any of its subtypes, while TypeVar('T', int, float) accepts only the listed types. Most of the time you’ll be consuming generics rather than writing them, since the libraries you depend on do the heavy lifting: list[T] is generic in its element type, and NumPy’s typed-array facilities (NDArray[np.float64] and friends) are generic in their dtype. But when you’re writing reusable utilities, particularly anything that wraps or batches data, reaching for TypeVar is what lets the wrapping be transparent to whoever uses it downstream.
Debugging generics can be opaque, since the inferred T isn’t visible at the call site. Most type checkers support reveal_type(x), which prints the inferred type at type-check time:
batch = DataBatch([1.0, 2.0, 3.0])
reveal_type(batch) # type checker prints: DataBatch[float]
It is the quickest way to understand a type error appearing where you don’t expect it.
Practical considerations
Despite their many benefits, annotations have limits. The type system cannot express everything Python can do: dynamic frameworks, decorators that change function signatures, and ORM-style metaprogramming all sit awkwardly within it, and libraries that lean on these patterns often need separate type-stub packages and checker plugins (django-stubs, sqlalchemy-stubs) to be checked at all. Annotations also add overhead. The type checker will sometimes disagree with code you know to be correct, and the time spent persuading it is time you weren’t spending on the actual problem. # type: ignore accumulates in real codebases for honest reasons, often because an upstream library’s types are incomplete or inaccurate.
Even your own code will rarely be fully typed, and that is fine. PEP 561 set out two official ways for libraries to ship type information, either inline with a py.typed marker or as a separate foopkg-stubs package. NumPy ships its annotations inline; pandas distributes them as pandas-stubs. Both projects have annotated their public APIs but openly acknowledge gaps: the pandas-stubs README notes that the stubs are “likely incomplete in terms of covering the published API”, and full coverage of the latest pandas release is still in progress. The same dynamic plays out in your own codebase. Coverage starts narrow and grows where the value is highest.
A sensible response is to pick your battles. Begin with the functions where there is most uncertainty about what is coming in, such as API responses or anything that reads from a database. Coverage grows outward from there. The same gradient applies to how strictly the checker enforces your annotations; basic checking catches obvious mismatches, while stricter modes can require annotations on every function and reject implicit Any types. Mypy, by default, skips functions that have no annotations at all, which means the most common surprise among new users is enabling the tool and finding it has nothing to say about the code they haven’t annotated yet. Pyright and the newer Rust-based checkers all check unannotated code by default, though mypy users can get the same behaviour by setting --check-untyped-defs. Whichever level you pick, continuous integration (CI) is the natural place to enforce it, since a check on every commit catches errors before they reach the main branch and sets a single standard for the team.
Against the costs are concrete wins. A wrong key in a TypedDict is caught at the keystroke rather than as a KeyError days later. A function signature with types tells the next reader what it expects without their having to read the body. Knowing when and how best to add annotations is a craft, and like any craft it rewards practice. Used well, type annotations turn assumptions about your code into things the checker can verify, making your life easier and more certain in the process. Happy typing!
References
[1] G. van Rossum, J. Lehtosalo and Ł. Langa, PEP 484: Type Hints (2014), Python Enhancement Proposals
[2] E. Smith, PEP 561: Distributing and Packaging Type Information (2017), Python Enhancement Proposals
[3] Ł. Langa, PEP 585: Type Hinting Generics In Standard Collections (2019), Python Enhancement Proposals
[4] J. Lehtosalo, PEP 589: TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys (2019), Python Enhancement Proposals
[5] D. Foster, PEP 655: Marking individual TypedDict items as required or potentially-missing (2021), Python Enhancement Proposals
[6] A. Purcell, PEP 705: TypedDict: Read-only items (2022), Python Enhancement Proposals
[7] Z. J. Li, PEP 728: TypedDict with Typed Extra Items (2023), Python Enhancement Proposals
[8] M. Lee, I. Levkivskyi and J. Lehtosalo, PEP 586: Literal Types (2019), Python Enhancement Proposals
[9] P. Prados and M. Moss, PEP 604: Allow writing union types as X | Y (2019), Python Enhancement Proposals
[10] I. Levkivskyi, J. Lehtosalo and Ł. Langa, PEP 544: Protocols: Structural subtyping (static duck typing) (2017), Python Enhancement Proposals
Instagram Engineering
Shopify Engineering