josiete.com

Data Quality

RAPTOR: a tree of summaries for querying large document collections

Conceptual illustration of documents arranged in groups and summary cards on a worktable

AI-generated conceptual cover. It evokes a document collection and its summaries; it does not represent the method’s architecture or experimental results.

Lately, I had been wondering how much code was being generated with AI worldwide and how many programmers still wrote everything by hand. Investigating those questions would mean looking in many places: surveys, studies of GitHub repositories, Stack Overflow posts, and company reports. Each source might contribute part of the story, but we would need to check what it actually measures and which conclusions we can draw from it.

Schema Drift in ETL: when data changes and your code does not notice

September timeline: data moves from Schema A to B and C before the ETL changes from v1.0 to v1.1 and v1.2; orange bands mark periods whose compatibility must be checked

Every morning, an ETL receives a CSV, converts ages to integers, and loads customers. Yesterday it worked with this file:

id,name,age
1,John,25
2,Mary,30

Today, the provider adds a country:

id,name,age,country
1,John,25,USA
2,Mary,30,UK

A few days later, another surprise arrives:

id,name,age,country
1,John,25,USA
2,Mary,unknown,UK

An additional column may be harmless if we select fields by name and allow extra columns. If we unpack each row into three variables, it may break the load. unknown may cause an age conversion error, silently become null, or lead a subsequent inference to treat the entire column as text.

When bytes masquerade as text: Unicode, Python, and encoding errors

Streams of green characters on a black background, with the words decode and encode highlighted in the center

A file contains José, but after passing through a pipeline it becomes José. Another preserves accents, yet searching for Jose also returns José. In a third, emojis turn into ?.

All three look like text problems, but they call for different investigations: how we interpret bytes, how we compare strings, and what information was lost along the way. Switching to UTF-8 without locating the failure can simply store already damaged text correctly.