A CSV file arrives. Before loading it into a database, you want to see its columns, check for missing values, and find the records you need. Opening a spreadsheet works, but becomes inconvenient when you have to repeat the same operation across several files. Writing a program for every check is not always worthwhile either.
csvkit brings together command line tools for converting and processing tabular data. It is written in Python and handles many of these tasks with short commands you can save and run again.
Fragments of papyrus PHerc. 1521, held by the British Library. Photographer unknown, British Library; source page, public domain.
The Herculaneum scrolls look like lumps of charcoal. Yet Greek writing remains inside those dark cylinders. The eruption of Vesuvius in AD 79 buried the town and the Villa of the Papyri, where an exceptional ancient library was found. The heat carbonized the scrolls: that transformation helped them survive, but also made them brittle and difficult to open.
A photo application is a useful place to separate concepts that are often blended together. Imagine that Alice signs in, browses a gallery, and changes the title of a photograph she uploaded. She can see one of Bob’s photographs, but cannot edit it. What did each step establish?
The sign-in established Alice’s identity. The decision to edit each photograph applied rules to an action and a specific resource. OAuth 2.0, OpenID Connect (OIDC), and SAML solve different parts of that problem. Keeping their boundaries clear avoids two common mistakes: treating a token as unlimited authorization, and treating an identity assertion as a credential for every API.
A file contains José, but after passing through a pipeline it becomes José. Another preserves accents, yet searching for Jose also returns José. In a third, emojis turn into ?.
All three look like text problems, but they call for different investigations: how we interpret bytes, how we compare strings, and what information was lost along the way. Switching to UTF-8 without locating the failure can simply store already damaged text correctly.
Imagine receiving sensor readings every five minutes. Each message contains a time, a sensor ID, a metric name, and a value. Storing messages as they arrive is convenient: if a new metric appears tomorrow, it simply adds more rows. But a report consuming the data wants something else: one row per time and sensor, with temperature, humidity, and pressure in separate columns.
