AI-generated conceptual cover. It evokes a document collection and its summaries; it does not represent the method’s architecture or experimental results.

Lately, I had been wondering how much code was being generated with AI worldwide and how many programmers still wrote everything by hand. Investigating those questions would mean looking in many places: surveys, studies of GitHub repositories, Stack Overflow posts, and company reports. Each source might contribute part of the story, but we would need to check what it actually measures and which conclusions we can draw from it.

Another question soon followed: how do we work with thousands of documents and unstructured information to arrive at a coherent explanation? To make the problem concrete, imagine collecting a hundred documents from those sources. The whole collection exceeds the context window of the LLM we plan to use. Reading each document separately also leaves some questions unanswered.

The first idea that came to mind was a funnel: split the material, produce partial summaries, and combine them through several stages until we have a synthesis that fits in the context window. That seems reasonable, although the synthesis no longer contains all the original information. Before deciding how to build it, we should consider what we want to ask. Let’s return to AI adoption.

“What population did this survey study?” calls for a specific detail. “What methodological differences explain why these reports give different results?” requires connecting definitions, samples, and procedures scattered across sources.

A trap soon emerges: the percentage of developers who use AI is not the percentage of code generated by AI. Someone might ask an assistant to explain a function without accepting a single line. Someone else might generate substantial code and discard it before integration. Even “generated code” can mean suggestions displayed, suggestions accepted, code committed to Git, or code retained after review.

We need to select information without erasing those distinctions.

Following that thread, I found two papers that help explain the problem. In Recursively Summarizing Books with Human Feedback, by Jeff Wu and colleagues (2021), the goal is to summarize entire books: split the text into passages, summarize them, and recursively combine those summaries into a book-level synthesis. It is a concrete application of that funnel idea.

The human feedback component belongs to training: people write example summaries and compare model outputs; those signals help adjust the model’s behavior. The research also aims to make it easier to oversee a task too long to evaluate in one go. It does not mean that a person reviews every summary when the system is subsequently used. Sections 2.2 and 2.3 of the paper explain the decomposition and training.

The next step that interested me appears in RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval, by Parth Sarthi and colleagues (2024). It retains the original passages and adds summaries of related groups so that both can be retrieved according to the question. The focus shifts from producing a final synthesis to querying content at different scales. What do we gain from that structure, and what might we lose while building it?

Before the tree: collecting documents is not understanding them

A crawler visits addresses and obtains documents. It can help collect pages, but does not by itself determine whether a survey represents developers in general, whether two news stories repeat the same press release, or what a metric means.

The hundred documents are a conceptual example. They could come from one crawler, several crawlers, a folder, or manual downloads. Imagining a hundred crawlers does not recommend one per document or a distributed architecture. RAPTOR concerns content organization and retrieval after acquisition, rather than web acquisition or reliability assessment.

These responsibilities matter: a system can retrieve a text containing an incorrect interpretation perfectly well. An index does not turn a claim into evidence.

What passage-based RAG does

RAG stands for Retrieval-Augmented Generation. In a simple setup, we first split documents into manageable passages. We then compute an embedding for each: a list of numbers representing features of the text learned by a model. Related texts tend to be close according to a similarity measure, allowing searches beyond literal word matches.

We store those vectors and their connections to the text in an index. When a question arrives, we compute its representation, retrieve related passages, and give the LLM the question and that context. The model writes an answer from the selection. Vector similarity signals relevance; it does not measure truth or prove that two figures are comparable.

This is a starting point, not a definition of every RAG system. There is lexical search such as BM25, hybrid lexical and vector search, metadata filtering, candidate reranking, and multi-step queries. Passage retrieval can answer cross-source questions if it assembles the necessary pieces.

The difficulty arises when the budget allows only a few passages and each supplies part of the explanation. Searching for “AI adoption” might return similar headlines while missing the methodology section. Larger passages provide more local context, but consume more space and may mix topics.

A larger context window expands what we can include. It does not select reliable sources, remove duplicates, or automatically resolve contradictions. Even if every document fit, we would still need to distinguish their populations and denominators.

Building summaries from the bottom up

RAPTOR stands for Recursive Abstractive Processing for Tree-Organized Retrieval. “Abstractive” means that a summary can reformulate and synthesize information, rather than merely copying sentences. Parth Sarthi and colleagues propose adding synthesized texts to the material available for retrieval. Section 3 of their 2024 paper describes a bottom-up construction:

1. Split the material. Original passages become leaves, the most detailed level.
2. Compute their embeddings. These representations allow content comparisons.
3. Group related passages. Clustering finds groups in those representations without requiring texts to be consecutive.
4. Summarize each group with an LLM. The summary becomes a new node connected to the nodes that produced it.
5. Compute embeddings for the summaries. They can now be grouped and retrieved too.
6. Repeat the cycle. Further levels summarize previous ones until construction stops.

Higher levels provide a more condensed view; lower levels retain more detail. A higher node is not necessarily more accurate and does not necessarily preserve every qualification. It is another representation of the material.

How it decides which texts to group

Vector representations can have many dimensions. To facilitate clustering, RAPTOR uses UMAP, a technique that reduces dimensionality while attempting to preserve neighborhood relationships. Think of a more compact representation for clustering, rather than UMAP summarizing the text. This reduction transforms the geometry and does not guarantee preservation of every important relationship.

It then applies Gaussian mixture models, or GMMs. In practical terms, it fits several probabilistic components to the points and estimates each point’s membership in them. The number of components is selected by comparing candidates using BIC, a criterion balancing fit and complexity. This does not discover an objectively correct number of topics.

The procedure combines global clustering with local clustering inside those groups. UMAP’s neighbor count helps control the scale. If a group is too large to summarize within the input limit, it is clustered again. These are index-building operations, performed before the user’s question arrives.

Membership is soft: a passage can belong to several groups. In the clustering code examined, components are assigned when their probabilities exceed a threshold. A passage about a survey’s methodology and results can contribute to two summaries.

“Tree” is therefore the method’s name, but we should not imagine a classification with exactly one parent per node. Relationships across levels can share descendants, as in a directed acyclic graph. A single root is not required either: the implementation can retain several nodes at the highest level.

Teaching diagram, not an algorithm run. Arrows lead from input material to the summary it produces. F2’s two connections show a possible overlap; embeddings, UMAP, and other groups are omitted. Identifiers match the synthetic example below. Open the figure to enlarge it.

The paper and repository are different references

The paper’s experimental configuration uses passages of roughly 100 tokens, SBERT embeddings, and gpt-3.5-turbo for summaries. These are choices for those experiments, not universal RAPTOR requirements.

The official repository revision examined, commit f50d917, already includes other options. Its default classes use LiteLLM, text-embedding-ada-002 for embeddings, and gpt-4o-mini for summaries. The builder has a configurable maximum number of levels and also stops when few nodes remain relative to the reduction dimension. Construction can stop before reaching a single synthesis.

These details appear in tree building, clustering termination, embeddings, and summarization. Running today’s repository defaults does not automatically reproduce the paper.

Querying the tree does not require following one branch

The paper describes two strategies. Both compare the question with node representations; they differ in the candidates they consider.

Tree traversal starts at the highest-level nodes and selects the k closest to the question. It gathers their children and again selects the best k from that pool. It repeats to the chosen depth and concatenates the selected texts across levels. This need not select a single branch: k can exceed one and descendants can be shared. However, a poor initial selection can exclude useful details before they are compared with the query.

Collapsed tree considers nodes from every level together: leaves and summaries compete in the same search. It ranks by similarity and selects context within a token budget. “Collapsing” does not mean deleting originals or summarizing everything again; it means searching the set of nodes without following parent–child links.

Simplified conceptual comparison. Traversal pools candidates from several parents; collapsed tree can select R1, F2, and F3 together. The selection shown is illustrative, not a measured result or a prediction for a particular question. Not all overlaps are shown.

The authors choose collapsed tree for their main results, with up to 2,000 context tokens; UnifiedQA receives 400 because of its context window. Baselines receive the same context budget in those comparisons. These limits concern retrieved text: an actual call must also reserve space for instructions, the question, and the answer. See the querying section and Figure 3.

The code adds an operational detail. In retrieve_information_collapse_tree, it considers at most top_k ranked candidates and stops when the next would exceed max_tokens; it does not skip that node to look for smaller ones. The traversal branch accumulates selections by level without applying that same token limit. An integration must check the total budget instead of assuming both paths enforce it identically.

Keeping leaves makes them retrievable. It does not guarantee that a particular query selects them or that the LLM respects the retrieved material. Even an answer based on an accurate summary may omit a qualification that never reached its context.

A synthetic example: people, suggestions, and integrated code

The documents, numbers, and summaries below are invented to explain the mechanism. They do not describe real AI adoption, do not come from existing surveys, and are not outputs from running RAPTOR.

A possible grouping joins F1, F2, and F4 around the survey and its coverage. Another joins F2 and F3 around measurement differences. F2 belongs to both: its definition of “use” matters both to understanding the survey and to comparing it with the internal report.

We manually write two teaching summaries:

• R1, from F1, F2, and F4: A measures self-reported use in a voluntary sample, including code explanations; C repeats A and provides no independent observations.
• R2, from F2 and F3: self-reported use and suggestion acceptance have different units and populations; neither measures the share of ultimately integrated code attributable to AI.

A higher synthesis S could preserve that incompatibility. It could also degrade it into “AI adoption is high and it generates some of the code,” introducing a judgment and a conclusion the synthetic documents do not support.

For “What population did A study?”, F2 holds the decisive detail. For “Why can’t A and B be compared directly?”, R2 may orient the answer, but checking F2 and F3 would be useful. That last step is an evidence policy we would propose, not a guaranteed collapsed-tree step.

The justified answer in this example would be: A counts people reporting any use during a month; B counts suggestions accepted by a team during a week. Neither the 60% of participants nor the 20% of suggestions lets us calculate the percentage of integrated code generated by AI. We should not average or multiply those percentages, or count C as a second survey.

A game of telephone can move up the levels

Repeated summarization resembles a game of telephone: each stage receives a transformed version of the message. The LLM does more than remove words; it chooses what to preserve and how to connect it. If a qualification disappears at one level, the next may have no way to reconstruct it.

In our imagined investigation, errors could take concrete forms:

• Omitting conditions: turning “voluntary participants from a forum” into “developers” in general.
• Confusing association with causation: turning “assistant users reported higher productivity” into “the assistant increased productivity.”
• Mixing metrics: replacing “accepted suggestions” with “integrated code.”
• Erasing disagreements: keeping the majority pattern while dropping a study finding a different effect or no detectable effect.
• Manufacturing consensus: treating ten pages repeating A as ten independent measurements.
• Adding unsupported content: completing a plausible explanation with information absent from the documents.

Errors can be correlated. Many calls to the same model on similar material do not constitute independent observations: the same tendency to simplify denominators may recur in several groups and then in their synthesis. There is no reason to assume errors will cancel out.

Appendix E of the paper studies hallucinations in 150 nodes sampled across 40 stories. Six contained some hallucination: 4% of that sample. The authors report no observed propagation to parent nodes and no discernible effect on their question-answering tasks. This is an observation limited to their analysis, not a universal error rate or a guarantee that recursive summaries cannot propagate errors. Nor does it exhaustively measure all the methodological omissions that concern us here.

For an editorial system, we would propose additional measures: provenance and version records, duplicate detection, retained evidence passages, claim-to-source associations, and human review when a synthesis changes a conclusion’s scope. RAPTOR keeps links between nodes; this alone does not demonstrate traceability for every claim or that a citation supports it. A summary must remain open to challenge against its source material.

What the authors demonstrated and how far that evidence goes

The evaluation in Section 4 focuses on questions about long texts:

They compare retrieval using BM25, which relies on terms; DPR, a vector retriever trained for passages; and SBERT. They also compare configurations with and without the RAPTOR structure while keeping the retriever fixed and using UnifiedQA-3B as the reader, the model answering questions.

Two rows from Table 2 help put the effect in perspective without mixing configurations:

These are absolute changes on the stated scales. They do not mean RAPTOR improves any system by 1.7%, or that F1 directly measures the faithfulness of every claim.

The paper’s most striking result appears in Table 7: RAPTOR with GPT-4 reaches 82.6% accuracy on the QuALITY test set, compared with 62.3% for CoLISA with DeBERTaV3-large, a difference of 20.3 percentage points. That comparison also changes the reader and the complete system. It cannot attribute all 20.3 points solely to the tree. The earlier controlled comparisons are more informative about the structure’s contribution.

The authors add qualitative examples and ablations: tests that change one system component to study its contribution. Appendix I includes cases where using every level is not the best option for an individual story. The hierarchy’s aggregate usefulness does not imply uniform improvement on every query.

There is a substantial gap between these tasks and our web collection. Much of the evaluation asks about an identified book, passage, or paper; it does not assess the truth of a hundred heterogeneous sources, their independence, or reconciliation of incompatible surveys. Answer metrics also do not replace documentary evidence review. Appendix H details adaptations to the NarrativeQA evaluation script, another reason to preserve the protocol when reproducing results.

The hierarchy also has a cost. It requires additional embeddings, dimensionality reduction and clustering, summarization calls, and storage for new nodes. Appendix A describes an approximately linear trend in time and token use within the tested range. This does not guarantee complexity or cost for every collection, model, or configuration. A stable collection queried repeatedly might amortize construction; frequently changing documents could demand substantial update work. We would need to measure that tradeoff in our own case.

Another way to organize summaries: GraphRAG

Microsoft GraphRAG builds a different structure: it extracts entities and relationships, organizes the graph into communities, and generates reports about them. Its global search uses community reports from a level of the hierarchy, generates partial answers, and combines selected ones through a map-reduce process.

Both prepare summaries for broad questions, but clustering text embeddings and constructing communities in an entity graph are different choices. GraphRAG global search is also different from ranking all RAPTOR nodes by similarity. We have not compared their performance: these references situate the design rather than establish a winner.

Looking at it as a data pipeline

From a data engineering perspective, the tree is a collection of transformations and derived products. It deserves the same questions as an ETL: what came in, what changed, which version processed it, and which outputs depend on it.

This table proposes instrumentation, not guarantees supplied by RAPTOR. The Node in the examined code stores text, an index, children, and embeddings; it does not itself contain that entire documentary record.

We would also distinguish format cleaning from content rewriting. Removing a repeated menu attempts to isolate the document body, although a parser might accidentally remove a footnote. Reformulating a methodology to make it “clearer” changes the semantic representation and may introduce interpretation. Keeping the original and recording both transformations with different validation criteria is useful.

What happens when a document changes? We would propose marking its passages and dependent summaries as stale, following every affected link, including shared ones. But regenerating only previous ancestors may be insufficient: new embeddings can change groups and membership. We would need to choose between batch rebuilding and an explicitly evaluated incremental strategy.

The add_documents function in the examined revision rebuilds the tree from the supplied text and warns when overwriting an existing one; its name should not make us assume incremental updates with document provenance.

How do we trace a claim to its source? We would propose linking it to specific passages and checking that they support its scope, population, and metric. A parent–child path identifies dependencies; it does not prove that every summary sentence follows from them. For our opening question, verification must reach the definition of “use” and the code actually counted, rather than stopping at a headline.

A small experiment worth considering

We have not run a retrieval comparison for this article. What follows is an experimental proposal without measured results. Earlier summaries and illustrated selections were written to explain the mechanism.

An initial test could use a synthetic collection of 12 short documents with incompatible metrics, explicit limitations, a contradiction, and identified republications. Before building the index, we would fix specific and cross-source questions, the evidence each requires, and when the correct answer is “these data do not allow that calculation.”

We would compare a passage index with one adding summaries through the official RAPTOR implementation pinned to a commit. Both would use the same documents, initial passages, embeddings, reader, and context budget. We would record models, prompts, parameters, seeds, and spending limits before any run; selecting default settings would not suffice.

With such a small collection, we would need to check that the builder actually produces additional levels. If it stops early because of the node count, we would document that and justify configuration adjustments. Manually forcing groups can help teaching, but that variant would be a didactic implementation rather than a reproduction of the paper.

We would preserve documents, questions, criteria, configuration, nodes, selections, and answers for repetition and review. A comparison on these questions alone would not measure real-world AI adoption or prove general superiority. No credentials or paid services have been used to run a RAG experiment.

When it is worth considering

RAPTOR offers a useful idea when questions move between detail and synthesis: make both kinds of text retrievable. It may be relevant where several passages contribute parts of an explanation and queries repeat often enough to justify index preparation.

In exchange, it adds a generative step before answering and new dependencies to maintain. If most queries seek exact facts in small documents, it makes sense to first check how much simpler retrieval can solve. When the task compares studies, definitions and provenance remain decisive with any structure.

In the investigation that opened this article, a good answer might conclude that the sources do not allow us to calculate what percentage of code was AI-generated. Finding and preserving that limitation would be as valuable as retrieving a number.

References

• Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., and Manning, C. D. (2024). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. Version v1 was read in full, including Appendices A–I; versioned PDF.
• Official implementation, latentsp/raptor-rag, revision f50d91739c6844d36d48c10970d531d961bc75eb. Consulted October 16, 2026; technical files linked near the claims they support.
• Wu, J., and colleagues (2021). Recursively Summarizing Books with Human Feedback. The abstract, introduction, and Sections 2.1–2.3 of version v2 were consulted for task decomposition and training with human feedback.
• Microsoft. GraphRAG documentation and Global Search. Consulted October 16, 2026.

Article credits

The editorial direction and initial question come from José Pérez’s brief. Source research, writing, translation, and diagrams were prepared with AI assistance through OpenAI Codex. The cover was generated with the built-in image tool; diagrams are code-authored SVGs, not generated scientific images.

Human review of the article is complete. The proposed RAG comparison has not been run, and no experimental results of our own are presented.