GoKawiilWhen you write typed Python, you expect your type checker to follow the rules of the language. But how closely do today's type checkers actually follow the Python typing specification?
In this post, we look at what typing spec conformance means, how different type checkers compare, and what the conformance numbers don't tell you.
Python's type system started with PEP 484. At the time, the semantics of the type system were mostly defined by the reference implementation, mypy. In practice, whatever mypy implemented became the de-facto specification.
Over time, more type checkers appeared: Pyright (from Microsoft), Pytype (from Google), and Pyre (from Facebook), to name a few. Meanwhile, the type system itself continued to evolve through many different PEPs.
This created a problem: the semantics of the type system were scattered across many documents, and different type checkers implemented slightly different interpretations. To address this, the typing community began consolidating the rules into a single reference: the Python typing specification. The spec describes the semantics of typing features and includes a conformance test suite that type checkers can run to measure how closely they follow the spec.
The typing specification includes a conformance test suite containing roughly a hundred test files. Each file encodes expectations about where a type checker should and shouldn't emit errors. The suite covers a wide range of typing features, from generics to overloads to type aliases. Within each test, lines are annotated to indicate where an error is expected. When a type checker runs on a file, two types of mismatches can occur:
False positives: The type checker reports an error on a line that is not marked as an error in the test. This means the checker rejected code that the specification considers valid. For example, given this valid code: class Movie ( TypedDict , extra_items = bool ) :
name : str
m : Movie = { "name" : "Blade Runner" , "novel_adaptation" : True }
If the type checker doesn't support extra_items , it will flag "novel_adaptation" as an unknown key, even though extra_items=bool explicitly allows extra boolean-valued keys.
... continue reading