DuckDB: the analytical duck that just landed at AWS

Wed, 23 Sep 2026 00:00:00 +0000

Some tools appear in data pipelines because a large company imposes them. Others spread because they solve an everyday problem with an elegance that is hard to ignore. DuckDB belongs to the second category. It is a small, fast, embeddable analytical database that can query local or remote files with SQL. In recent years it has become a common part of Data Engineering, Data Science, and Analytics Engineering workflows.

It is now at the center of important news: Amazon has brought DuckLabs into AWS, the Amsterdam company built around the team developing DuckDB. The deal was announced in August 2026 and DuckLabs confirmed that it closed on August 31. The important point for users is that Amazon did not buy the open-source project itself: DuckDB remains under the independent DuckDB Foundation and under the MIT license.

What is DuckDB?

DuckDB is a relational database management system designed for analytical workloads—OLAP. Unlike a traditional server database, it runs inside the application process: it can be imported as a library from Python, R, Go, Java, Rust, Node.js, or C++, or used through its CLI.

SELECT country, sum(amount) AS total
FROM 'sales/*.parquet'
GROUP BY country
ORDER BY total DESC;

There is no need to start a cluster or create an intermediate table first. DuckDB can query CSV, Parquet, JSON, and other formats directly, including data accessible through HTTP(S) or S3. Its vectorized engine processes batches of values per operation and is optimized for scanning, filtering, aggregating, and joining large datasets.

The comparison that explains it best is SQLite: both are simple, embedded, and portable, but SQLite is primarily designed for transactional workloads while DuckDB is designed for analytics. It is not a universal replacement for PostgreSQL, an event queue, or a distributed warehouse. Its strength is occupying a specific niche and doing it exceptionally well.

Who maintains it?

DuckDB started in the database architecture group at CWI Amsterdam, with Hannes Mühleisen and Mark Raasveldt among its creators. Professional development was organized in DuckDB Labs, later renamed DuckLabs.

The key governance piece is the DuckDB Foundation, a Dutch non-profit foundation that owns the intellectual property and trademarks of the Duck Stack. Its board includes DuckDB’s creators and Peter Boncz, a database-systems researcher. DuckLabs provides the engineering team; the Foundation protects the project and its open license.

Why is it open source?

DuckDB is distributed under the MIT license, a permissive license that allows people to use, study, modify, and redistribute the software, including inside commercial products. DuckDB’s own documentation explains that the project began in a Dutch public research environment and that its authors see making the results freely available as a responsibility to society.

That freedom is not only a moral or academic matter. For a tool placed in the middle of a pipeline, being able to inspect the engine, compile it for different architectures, and add extensions reduces lock-in risk. It also lets universities, companies, tool vendors, and individual users work from the same foundation.

Why has it become so popular in Data Engineering?

DuckDB’s popularity is not explained by one feature, but by how several of them fit together:

No infrastructure to get started: it installs as a library and requires no server administration.
SQL over files: a directory of Parquet or CSV files becomes queryable without first loading it into a central system.
Analytical speed: columnar, vectorized execution fits aggregations, joins, and full scans.
Portability: it works on a laptop, in a container, in a function, in a browser through WebAssembly, or on a large server.
A strong Python and R fit: it connects DataFrames and SQL without unnecessary data movement.
Interoperability: Parquet, JSON, HTTP(S), S3, extensions, and lakehouse formats are part of its ecosystem.
Reproducibility: a database file or SQL script can travel with a project and run in CI, locally, or in a notebook.

This changes the order of some traditional workflows. Instead of uploading every dataset to a warehouse before exploring it, an engineer can run an initial transformation locally over lake objects, validate a hypothesis, and materialize only what deserves to become a permanent process. DuckDB does not eliminate the warehouse; it makes the work before, beside, or between warehouse queries cheaper and faster.

Amazon’s acquisition: what actually happened

The terminology matters. AWS acquired DuckLabs, not the DuckDB project as if it were a closed application. According to Amazon, Hannes Mühleisen and Mark Raasveldt will continue leading the team and the project’s technical direction. According to DuckLabs’ announcement, the team remains together in Amsterdam, DuckDB, DuckLake, and Quack remain under the Foundation, and the software remains under the MIT license.

AWS’s interest is easy to understand: DuckDB is an effective layer for working close to data, especially in S3, and can connect files, notebooks, applications, lakehouses, and analytical services. The combination gives AWS access to specialized talent and gives the project much greater resources, infrastructure, and commercial reach.

Will DuckDB’s design change?

It is too early to state which concrete design decisions will change. Public announcements say that the license, governance, and immediate roadmap do not change. We can still outline reasonable hypotheses:

Deeper cloud ecosystem integration. We may see more work around S3, lakehouse formats, catalogs, permissions, and AWS services. The challenge will be improving that integration without turning DuckDB into a privileged client for one provider.
More investment in remote operation. DuckDB began as an embedded engine, but its ecosystem is already exploring protocols and distributed scenarios. AWS may accelerate features that move from local exploration to remote data without losing the simplicity of the model.
More pressure on extensions and compatibility. The Foundation and DuckLabs have indicated that they want to grow the community and extension system. That can bring more connectors and formats, but it also requires stable APIs, security, and a coherent user experience.
Vendor neutrality as a strategic design decision. Paradoxically, for AWS to gain long-term value, DuckDB must remain useful outside AWS. The adoption that made it valuable depends on working locally, in other clouds, in notebooks, and inside third-party products.

These are inferences, not published commitments. The test will be whether new integrations are designed as open extensions and whether the community retains a real voice in the roadmap. The Foundation has announced a technical advisory board with community participation, which could help turn that openness into a design practice.

The interesting question

The acquisition is not only about whether Amazon can benefit from DuckDB. The more interesting question is whether a cloud company can fund an infrastructure component that remains neutral, small, and pleasant to use—even when that component makes it easier to work outside its own cloud.

For now, the official answer is reassuring: DuckDB remains free, the MIT license stays in place, and the Foundation retains stewardship. The future will depend on less visible decisions: what gets prioritized, what stays out of the core, how extensions are accepted, how portability is protected, and who gets a voice when community interests and AWS interests diverge.

DuckDB became popular because it shortened the distance between a question and an answer about data. Its next chapter will be interesting if it can also shorten the distance between local and cloud analytics without forcing us to choose only one of them.

AWS on josiete.com