Polars: Python speeds up DataFrames with Rust and multiple threads

Thu, 24 Sep 2026 00:00:00 +0000

Python remains one of the most comfortable languages for exploring and transforming data. But DataFrame convenience does not always translate into speed: as rows and columns grow, memory and CPU costs become obvious. Polars addresses that problem with a Rust-based DataFrame library exposed through a Python API.

What is Polars?

Polars is an analytical query engine and DataFrame library for tabular data. Its core is written in Rust, uses an Apache Arrow-compatible memory model, and combines vectorized, columnar processing with query optimization.

The API offers two styles. Eager mode executes each operation immediately, which is useful for interactive exploration. Lazy mode builds a query plan: Polars can reorder operations, remove unused columns, and push filters down before executing the result.

It is not a universal replacement for a database or distributed system. Its strength is using one machine—laptop or server—efficiently while working with CSV, Parquet, JSON, and other sources without requiring a cluster.

Parallelism: multiple threads without writing concurrency code

Unlike many traditional Python workflows, Polars does not leave all heavy computation on the interpreter’s main thread. Its engine is implemented in Rust and uses a thread pool to distribute work across available CPU cores. The engine is vectorized, columnar, and multi-threaded, so a filter, aggregation, or join can automatically use several cores.

In practice, users write ordinary Polars expressions and the engine decides how to parallelize them. There is no need to create a ThreadPoolExecutor, serialize every partition, or coordinate results manually. Python is the interface; intensive computation runs in the native core.

The current API exposes the pool size through pl.thread_pool_size(). Polars chooses a reasonable value automatically, but in a shared environment—such as a worker with an existing CPU limit—you can set POLARS_MAX_THREADS before importing the library:

POLARS_MAX_THREADS=4 python analyze_sales.py

This should not be confused with distributed execution: multiple threads use the cores of one machine. For data larger than memory, the lazy engine can also stream a plan when supported; horizontal scaling is provided by a separate distributed layer.

The company behind the project

Polars began in 2020 as a personal project by Ritchie Vink, initially to learn about query engines, Apache Arrow, and Rust. On August 3, 2023, Vink and Chiel Peters announced the creation of Polars Inc., an Amsterdam-based company focused on providing engineering and long-term stability for the ecosystem.

The company did not replace the community project: Polars remains open source under the MIT license. Its role is to fund engine development, improve connectors, and offer products for running queries at larger scale, including Polars Cloud. It is an interesting separation: a company can build commercial services around a project while keeping the open core available to everyone.

A Python example

Install the library with pip and build a small transformation. scan_csv returns a LazyFrame, so operations are planned and collect() executes the optimized plan.

pip install polars

import polars as pl

sales = pl.scan_csv("sales.csv")

result = (
 sales
 .filter(pl.col("status") == "paid")
 .with_columns(
 (pl.col("units") * pl.col("price")).alias("revenue")
 )
 .group_by("category")
 .agg(
 pl.col("revenue").sum().alias("revenue"),
 pl.col("units").sum().alias("units")
 )
 .sort("revenue", descending=True)
 .collect()
)

print(result)

The code states what we want and lets Polars decide how to scan columns and divide the work. With a large file, lazy mode can avoid reading unused columns and push the filter into the scan. For Parquet, pl.scan_parquet("data/*.parquet") follows the same pattern.

When should you choose it?

Polars is a strong fit for transformation pipelines, large exploratory analyses, model preparation, and services that need quick responses without immediately adopting a distributed system. If a project depends heavily on the pandas ecosystem, migration requires reviewing some operations and types; both libraries can still coexist, and Polars can convert to pandas when needed.