josiete.com

Databases

CDC: keep data in sync without copying everything again

A transactional database sends small blocks of changes to an analytical system

Imagine a shop with ten million orders in its database. The analytics team needs to query today’s sales, so it copies those orders into an analytical warehouse every night. If only ten thousand changed today, transferring all ten million again is slow and expensive. Change Data Capture (CDC) identifies the changes so the pipeline can send only what is needed.

When bytes masquerade as text: Unicode, Python, and encoding errors

Streams of green characters on a black background, with the words decode and encode highlighted in the center

A file contains José, but after passing through a pipeline it becomes José. Another preserves accents, yet searching for Jose also returns José. In a third, emojis turn into ?.

All three look like text problems, but they call for different investigations: how we interpret bytes, how we compare strings, and what information was lost along the way. Switching to UTF-8 without locating the failure can simply store already damaged text correctly.

DBeaver: the data engineer's Swiss Army knife

A beaver working with Parquet data and DuckDB in DBeaver

Some tools solve one specific task. Others eventually become the whole toolbox. DBeaver belongs to the second category: it is a cross-platform database client for connecting to, querying, exploring, editing, and exporting data without changing applications. For a data engineer, it feels a lot like a graphical Swiss Army knife.

What is DBeaver?

DBeaver is a desktop application for working with databases and other data sources. It supports JDBC connections, schema and table browsing, SQL with autocomplete, execution-plan inspection, record editing, and data export.