DuckDB on josiete.com

DBeaver: the data engineer's Swiss Army knife

Thu, 24 Sep 2026 00:00:00 +0000

Some tools solve one specific task. Others eventually become the whole toolbox. DBeaver belongs to the second category: it is a cross-platform database client for connecting to, querying, exploring, editing, and exporting data without changing applications. For a data engineer, it feels a lot like a graphical Swiss Army knife.

What is DBeaver?

DBeaver is a desktop application for working with databases and other data sources. It supports JDBC connections, schema and table browsing, SQL with autocomplete, execution-plan inspection, record editing, and data export.

Its main advantage is bringing together tasks that would otherwise be split between a console, an IDE, a spreadsheet, and several vendor tools. The metadata browser helps us understand a database quickly; the SQL editor lets us test a query; and the results view lets us filter, sort, copy, or export information.

DBeaver Community is open source, while the Enterprise edition adds features for teams and particular data sources. The company behind the product is DBeaver Corp, based in the United States. The project grew around the open-source community, and the company maintains both the Community edition and commercial variants.

One client, many databases

DBeaver is not a database engine: it is the client that connects to databases. That distinction matters. It does not replace PostgreSQL, MySQL, SQL Server, Oracle, MariaDB, SQLite, or Snowflake; it provides a common experience for working with them.

Depending on the driver and edition, we can connect to relational databases, analytical warehouses, cloud services, and sources exposing a JDBC driver. We can also configure SSH tunnels, manage credentials, save connections, and open several sessions in parallel. The interface does not remove engine-specific differences, but it reduces the mental cost of switching between them.

It is available for Linux, Windows, and macOS. On Linux it is useful for people moving between servers, containers, and a development laptop; on Windows it fits teams working with SQL Server, PostgreSQL, cloud services, and corporate tooling. The same idea and much of the workflow travel with us between operating systems.

Parquet: when a table does not live in a database

Parquet is a columnar storage format, not a database. It stores data by column and keeps metadata that allows readers to fetch only the necessary parts. That is why it is common in data lakes and analytical pipelines: it is compact, compresses well, and works with many engines.

DBeaver can explore Parquet files directly through compatible engines and connections instead of forcing us to load them into a traditional database first. This is useful for inspecting a dataset received from another team, checking column types, finding nulls, or validating a partition before building a pipeline.

Two operations should be distinguished. Viewing a Parquet file tells us what it contains; querying it with SQL requires an engine that can read it. That is where DuckDB comes in: DBeaver provides the interface and DuckDB provides the analytical engine.

DBeaver and DuckDB: a practical pairing

DuckDB is an embedded analytical database that can query Parquet, CSV, and JSON without starting a server. DBeaver can connect to DuckDB and become its visual environment: we write SQL in the editor, inspect plans and results in tables, and browse objects through a familiar interface.

A typical query can be as simple as:

SELECT customer_id, sum(amount) AS total
FROM 'data/sales/*.parquet'
GROUP BY customer_id
ORDER BY total DESC
LIMIT 20;

The file remains Parquet and DuckDB reads it directly; there is no need to import it into a server database before starting analysis. This combination is particularly comfortable for exploration, data-quality checks, and prototypes: DBeaver makes investigation visual and DuckDB provides fast, portable SQL.

It can also create a local DuckDB database, join several files, materialize intermediate results, and export them again. The workflow lands in a very useful space between a spreadsheet and a full data platform.

Why call it a Swiss Army knife?

The metaphor does not mean DBeaver does everything better than specialized tools. It means it covers many needs with surprisingly little friction:

explore schemas, indexes, views, and permissions;
write and save SQL queries;
compare results and inspect execution plans;
edit individual records carefully;
import and export CSV, JSON, and other formats;
work with multiple connections and tabs;
document queries and share configurations with the team;
connect to classic engines, cloud services, and embedded databases.

For someone learning SQL, the data view provides immediate feedback. For an experienced user, the value is having context, queries, and results in one place. For a data engineer, the mix of heterogeneous connections, Parquet, and DuckDB greatly shortens the path from “I was given these files” to “I know what they contain.”

Things to keep in mind

Convenience does not remove responsibility. A production connection should be protected with suitable credentials, least-privilege permissions, and care when running UPDATE, DELETE, or migrations. A graphical interface can also hide SQL-dialect differences, query costs, and access policies.

DBeaver is not an orchestrator or a data-governance system. It does not replace a versioned pipeline, automated tests, change control, or observability. Its role is more focused and, for that reason, valuable: it is a flexible workstation for understanding and operating on data.

A small tool for varied problems

The best way to understand DBeaver is as a common work layer. One day it can be a PostgreSQL client; the next, the window through which we inspect a Parquet file with DuckDB; later, a browser for a local SQLite database or a console for an analytical warehouse.

That versatility explains its popularity. DBeaver does not try to decide where data should live: it helps us see it, ask questions of it, and move between systems. And when the logo’s beaver appears next to a Parquet file and a DuckDB query, the joke is more than visual: it captures a modern, pragmatic way of working with data.

DuckDB: the analytical duck that just landed at AWS

Wed, 23 Sep 2026 00:00:00 +0000

Some tools appear in data pipelines because a large company imposes them. Others spread because they solve an everyday problem with an elegance that is hard to ignore. DuckDB belongs to the second category. It is a small, fast, embeddable analytical database that can query local or remote files with SQL. In recent years it has become a common part of Data Engineering, Data Science, and Analytics Engineering workflows.

It is now at the center of important news: Amazon has brought DuckLabs into AWS, the Amsterdam company built around the team developing DuckDB. The deal was announced in August 2026 and DuckLabs confirmed that it closed on August 31. The important point for users is that Amazon did not buy the open-source project itself: DuckDB remains under the independent DuckDB Foundation and under the MIT license.

What is DuckDB?

DuckDB is a relational database management system designed for analytical workloads—OLAP. Unlike a traditional server database, it runs inside the application process: it can be imported as a library from Python, R, Go, Java, Rust, Node.js, or C++, or used through its CLI.

SELECT country, sum(amount) AS total
FROM 'sales/*.parquet'
GROUP BY country
ORDER BY total DESC;

There is no need to start a cluster or create an intermediate table first. DuckDB can query CSV, Parquet, JSON, and other formats directly, including data accessible through HTTP(S) or S3. Its vectorized engine processes batches of values per operation and is optimized for scanning, filtering, aggregating, and joining large datasets.

The comparison that explains it best is SQLite: both are simple, embedded, and portable, but SQLite is primarily designed for transactional workloads while DuckDB is designed for analytics. It is not a universal replacement for PostgreSQL, an event queue, or a distributed warehouse. Its strength is occupying a specific niche and doing it exceptionally well.

Who maintains it?

DuckDB started in the database architecture group at CWI Amsterdam, with Hannes Mühleisen and Mark Raasveldt among its creators. Professional development was organized in DuckDB Labs, later renamed DuckLabs.

The key governance piece is the DuckDB Foundation, a Dutch non-profit foundation that owns the intellectual property and trademarks of the Duck Stack. Its board includes DuckDB’s creators and Peter Boncz, a database-systems researcher. DuckLabs provides the engineering team; the Foundation protects the project and its open license.

Why is it open source?

DuckDB is distributed under the MIT license, a permissive license that allows people to use, study, modify, and redistribute the software, including inside commercial products. DuckDB’s own documentation explains that the project began in a Dutch public research environment and that its authors see making the results freely available as a responsibility to society.

That freedom is not only a moral or academic matter. For a tool placed in the middle of a pipeline, being able to inspect the engine, compile it for different architectures, and add extensions reduces lock-in risk. It also lets universities, companies, tool vendors, and individual users work from the same foundation.

Why has it become so popular in Data Engineering?

DuckDB’s popularity is not explained by one feature, but by how several of them fit together:

No infrastructure to get started: it installs as a library and requires no server administration.
SQL over files: a directory of Parquet or CSV files becomes queryable without first loading it into a central system.
Analytical speed: columnar, vectorized execution fits aggregations, joins, and full scans.
Portability: it works on a laptop, in a container, in a function, in a browser through WebAssembly, or on a large server.
A strong Python and R fit: it connects DataFrames and SQL without unnecessary data movement.
Interoperability: Parquet, JSON, HTTP(S), S3, extensions, and lakehouse formats are part of its ecosystem.
Reproducibility: a database file or SQL script can travel with a project and run in CI, locally, or in a notebook.

This changes the order of some traditional workflows. Instead of uploading every dataset to a warehouse before exploring it, an engineer can run an initial transformation locally over lake objects, validate a hypothesis, and materialize only what deserves to become a permanent process. DuckDB does not eliminate the warehouse; it makes the work before, beside, or between warehouse queries cheaper and faster.

Amazon’s acquisition: what actually happened

The terminology matters. AWS acquired DuckLabs, not the DuckDB project as if it were a closed application. According to Amazon, Hannes Mühleisen and Mark Raasveldt will continue leading the team and the project’s technical direction. According to DuckLabs’ announcement, the team remains together in Amsterdam, DuckDB, DuckLake, and Quack remain under the Foundation, and the software remains under the MIT license.

AWS’s interest is easy to understand: DuckDB is an effective layer for working close to data, especially in S3, and can connect files, notebooks, applications, lakehouses, and analytical services. The combination gives AWS access to specialized talent and gives the project much greater resources, infrastructure, and commercial reach.

Will DuckDB’s design change?

It is too early to state which concrete design decisions will change. Public announcements say that the license, governance, and immediate roadmap do not change. We can still outline reasonable hypotheses:

Deeper cloud ecosystem integration. We may see more work around S3, lakehouse formats, catalogs, permissions, and AWS services. The challenge will be improving that integration without turning DuckDB into a privileged client for one provider.
More investment in remote operation. DuckDB began as an embedded engine, but its ecosystem is already exploring protocols and distributed scenarios. AWS may accelerate features that move from local exploration to remote data without losing the simplicity of the model.
More pressure on extensions and compatibility. The Foundation and DuckLabs have indicated that they want to grow the community and extension system. That can bring more connectors and formats, but it also requires stable APIs, security, and a coherent user experience.
Vendor neutrality as a strategic design decision. Paradoxically, for AWS to gain long-term value, DuckDB must remain useful outside AWS. The adoption that made it valuable depends on working locally, in other clouds, in notebooks, and inside third-party products.

These are inferences, not published commitments. The test will be whether new integrations are designed as open extensions and whether the community retains a real voice in the roadmap. The Foundation has announced a technical advisory board with community participation, which could help turn that openness into a design practice.

The interesting question

The acquisition is not only about whether Amazon can benefit from DuckDB. The more interesting question is whether a cloud company can fund an infrastructure component that remains neutral, small, and pleasant to use—even when that component makes it easier to work outside its own cloud.

For now, the official answer is reassuring: DuckDB remains free, the MIT license stays in place, and the Foundation retains stewardship. The future will depend on less visible decisions: what gets prioritized, what stays out of the core, how extensions are accepted, how portability is protected, and who gets a voice when community interests and AWS interests diverge.

DuckDB became popular because it shortened the distance between a question and an answer about data. Its next chapter will be interesting if it can also shorten the distance between local and cloud analytics without forcing us to choose only one of them.