In this article, you will learn how to build a text clustering pipeline by combining large language model embeddings with HDBSCAN, a density-based clustering algorithm, to automatically discover topics in unlabeled text data. Topics we will cover include: How to generate text embeddings for raw documents using a pre-trained sentence-transformers model. How to reduce the dimensionality of those embeddings with UMAP to prepare them for clustering. How to apply HDBSCAN to automatically discover topic clusters and visualize the results. Clustering Unstructured Text with LLM Embeddings and HDBSCAN Introduction The current era of Generative AI seems to primarily focus on chat interfaces and prompts, but the range of applications of large language models, or LLMs for short, is not limited to just that. Indeed, one of their most powerful downstream abilities consists of turning raw, messy, unstructured text into semantically rich mathematical representations called embeddings. Once that’s done, we can use these text representations for a variety of machine learning use cases, with clustering being no exception. In particular, embeddings can be combined with advanced, density-based clustering techniques like HDBSCAN, allowing as a result for the discovery of hidden topics, patterns, or categories in your collection of text documents: all without the need for prior labeling. This article shows how to construct a text-based clustering pipeline from scratch. We will use a freely available dataset containing text instances, as well as an open-source LLM that has been trained for generating embeddings — i.e. a so-called embedding model. The icing on the cake: we’ll use free and handy, modern Python libraries providing implementations of clustering algorithms like HDBSCAN. Step-by-Step Walkthrough First, let’s start by installing the key Python libraries we will need: Sentence transformers, to load a pre-trained LLM for embedding generation from Hugging Face — you’ll need a Hugging Face API key, also called an access token, to be able to load the model. Umap-learn, to apply an algorithm to reduce the dimensionality of embeddings. Likewise, if you are working on a local IDE instead of a cloud notebook environment and don’t have scikit-learn and pandas, you may need to install them too. !pip install sentence-transformers umap-learn !pip install sentence–transformers umap–learn Now we start the coding part by getting some fresh data. The fetch_20newsgroups function, which fetches a dataset containing texts from categorized news articles, will do. Note that even though the dataset contains labels, we will omit them, as we are pretending not to know this information for the sake of clustering these data instances into groups based on similarity. Also, we sample down the dataset to 150 instances, which will be representative enough for our example. import pandas as pd from sklearn.datasets import fetch_20newsgroups # Fetching a highly targeted subset of data (~150-200 docs) categories = [‘sci.space’, ‘sci.med’, ‘rec.autos’] newsgroups = fetch_20newsgroups(subset=”train”, categories=categories, remove=(‘headers’, ‘footers’, ‘quotes’)) # Sampling down into a representative, illustrative subset df = pd.DataFrame({‘text’: newsgroups.data, ‘true_label’: newsgroups.target}) df = df[df[‘text’].str.strip().str.len() > 100].sample(150, random_state=42).reset_index(drop=True) print(f”Loaded {len(df)} text documents.”) print(“\nSample document:”) print(df[‘text’].iloc[0][:150] + “…”) import pandas as pd from sklearn.datasets import fetch_20newsgroups   # Fetching a highly targeted subset of data (~150-200 docs) categories = [‘sci.space’, ‘sci.med’, ‘rec.autos’] newsgroups = fetch_20newsgroups(subset=‘train’, categories=categories, remove=(‘headers’, ‘footers’, ‘quotes’))   # Sampling down into a representative, illustrative subset df = pd.DataFrame({‘text’: newsgroups.data, ‘true_label’: newsgroups.target}) df = df[df[‘text’].str.strip().str.len() > 100].sample(150, random_state=42).reset_index(drop=True)   print(f“Loaded {len(df)} text documents.”) print(“\nSample document:”) print(df[‘text’].iloc[0][:150] + “…”) Output: Loaded 150 text documents. Sample document: Okay Mr. Dyer, we’re properly impressed with your philosophical skills and ability to insult people. You’re a wonderful speaker and an adept politic… Loaded 150 text documents.   Sample document:   Okay Mr. Dyer, we‘re properly impressed with your philosophical skills and ability to insult people. You’re a wonderful speaker and an adept politic… The next step is to obtain the embeddings from raw texts. To do this, we load all-MiniLM-L6-v2 from Hugging Face’s sentence-transformers library. This is a lightweight yet effective model to obtain embeddings quickly. from sentence_transformers import SentenceTransformer # Loading the free, open-source model model = SentenceTransformer(‘all-MiniLM-L6-v2’) # Encoding text documents into dense vector embeddings print(“Generating embeddings…”) embeddings = model.encode(df[‘text’].tolist(), show_progress_bar=True) print(f”Embedding matrix shape: {embeddings.shape}”) from sentence_transformers import SentenceTransformer   # Loading the free, open-source model model = SentenceTransformer(‘all-MiniLM-L6-v2’)   # Encoding text documents into dense vector embeddings print(“Generating embeddings…”) embeddings = model.encode(df[‘text’].tolist(), show_progress_bar=True)   print(f“Embedding matrix shape: {embeddings.shape}”) Since the embedding dimension is originally too high for clustering purposes, we now apply a dimensionality reduction technique by using the UMAP algorithm from the namesake library installed earlier: import umap # Reducing embedding dimensions to 5, to retain enough density information for clustering reducer = umap.UMAP(n_neighbors=15, n_components=5, min_dist=0.0, random_state=42) reduced_embeddings = reducer.fit_transform(embeddings) print(f”Reduced matrix shape: {reduced_embeddings.shape}”) import umap   # Reducing embedding dimensions to 5, to retain enough density information for clustering reducer = umap.UMAP(n_neighbors=15, n_components=5, min_dist=0.0, random_state=42) reduced_embeddings = reducer.fit_transform(embeddings)   print(f“Reduced matrix shape: {reduced_embeddings.shape}”) Now our numerical embedding vectors associated with news articles consist of five dimensions (attributes) only. Let’s see if this compact representation is meaningful enough to obtain insightful clustering by applying the HDBSCAN algorithm, which is a density-based clustering approach: from sklearn.cluster import HDBSCAN # Initializing HDBSCAN # min_cluster_size=8: we specified that each cluster must have at least 8 documents clusterer = HDBSCAN(min_cluster_size=8, min_samples=3, store_centers=”centroid”) df[‘cluster’] = clusterer.fit_predict(reduced_embeddings) # Counting instances per cluster cluster_counts = df[‘cluster’].value_counts() print(“\nCluster Distribution:”) print(cluster_counts) from sklearn.cluster import HDBSCAN   # Initializing HDBSCAN # min_cluster_size=8: we specified that each cluster must have at least 8 documents clusterer = HDBSCAN(min_cluster_size=8, min_samples=3, store_centers=‘centroid’) df[‘cluster’] = clusterer.fit_predict(reduced_embeddings)   # Counting instances per cluster cluster_counts = df[‘cluster’].value_counts() print(“\nCluster Distribution:”) print(cluster_counts) Important: the clustering results are partly influenced by the hyperparameter settings we defined for HDBSCAN. I recommend you try out other configurations for the minimum cluster size and other hyperparameters to explore how this affects results. Result: Cluster Distribution: cluster 0 101 1 49 Name: count, dtype: int64 Cluster Distribution: cluster 0    101 1     49 Name: count, dtype: int64 It looks like HDBSCAN detected two

In this article, you will learn why a large context window is not the same thing as agent memory, and how techniques like retrieval, compression, and summarization fit together in an agent’s cognitive stack. Topics we will cover include: Why a context window behaves like a stateless scratchpad rather than persistent memory. How retrieval-augmented generation, compression, and summarization each play a distinct role in managing what enters that scratchpad. How agents can achieve genuine memory persistence by acting as a database administrator rather than as the database itself. Introduction Context windows are a key aspect of modern AI models, particularly language models, whereby these models can attend to and utilize a limited amount of input and prior conversation — typically measured as a number of tokens — at once when producing a response. When an AI lab releases a model with a 2-million token context window, it is no surprise some developers instinctively think like this: “Let’s shove the whole codebase into the prompt! Memory issues sorted!” However, there is a caveat. Deeming a huge context window as “memory” is, in architectural terms, similar to buying a 25-foot-wide office desk because you are reluctant to acquire a filing cabinet. Sure, you can have all your documents laid in front of you, but as soon as the working session ends, the entire desk’s documents are wiped out (by cleaning staff!). To clarify this distinction and demystify other related concepts, this article offers a conceptual breakdown of multiple layers in AI agents’ cognitive stack. We will use several, mostly office-related metaphors to facilitate a better understanding of these concepts. Context Window A context window in an AI model, particularly agent-based ones with underlying language models, is like a desk surface or a stateless scratchpad. It is important to note that models are inherently fully stateless. No matter what, every API call to a model starts at “step zero”. When passing an agent a conversation history spanning over 200K tokens (large context window), it isn’t remembering what happened at a previous step in time. Instead, it is quickly re-reading “its universe” from scratch in a matter of milliseconds. In the long-run, relying on this strategy in agent-based environments may introduce several dangerous (if not fatal) traps: AI models act like a lazy student, who pays close attention to the initial and final parts of a massive prompt (text), but utterly glosses over ideas and facts buried deep in the middle parts. There is a snowballing effect: as the conversation grows, the agent must re-send and re-read the entire history at every single step, including the earliest, often irrelevant turns. In terms of latency, there is a “brain freeze” effect, so that against a huge wall of text, the model will take some time until starting to generate the very first word in its response. To make this concrete, consider what a single API call actually looks like under the hood. Because the model holds no memory between calls, every prior turn must be resent in full just to ask one new question: model.generate( messages=[ {“role”: “user”, “content”: “Step 1: Let’s call this variable `session_id`.”}, {“role”: “assistant”, “content”: “Got it, I’ll use `session_id` going forward.”}, # … every intervening turn must be resent, every single time … {“role”: “user”, “content”: “Step 47: What variable name did we agree on back in step 1?”} ] ) model.generate(     messages=[         {“role”: “user”, “content”: “Step 1: Let’s call this variable `session_id`.”},         {“role”: “assistant”, “content”: “Got it, I’ll use `session_id` going forward.”},         # … every intervening turn must be resent, every single time …         {“role”: “user”, “content”: “Step 47: What variable name did we agree on back in step 1?”}     ] ) Step 47 alone forces the entire desk — all 46 prior turns — back onto the table, just to answer a question about step 1. That is the snowballing effect described above, made concrete. Retrieval Retrieval-augmented generation (RAG) systems are like a big bookshelf across the office room, that helps fetch static, existing data relevant to the current step in a “Just-In-Time” fashion. RAG systems pull the top-K relevant document chunks into the scratchpad (the context window) as the user asks a certain question: the retrieved documents are, of course, the ones determined as most semantically relevant to the user’s question or prompt. When agents are in the loop, things are not that easy, however, as vector similarity (the type of similarity measure and data representation used in RAG systems) is not necessarily equivalent to semantic truth in certain cases. For example, suppose a user tells their scheduling agent to move a meeting to Friday, and later says “cancel Thursday, Alice is sick.” A vector search engine may retrieve both statements from a document base, even though they contradict each other. The agent and its associated language model must be able to act as accountants capable of determining which statement better reflects the current reality. A naive RAG pipeline simply concatenates whatever it retrieves and leaves the model to guess which instruction still holds. A more reliable pattern resolves the conflict before generation ever happens, for example by favoring the most recently recorded statement: retrieved_chunks = [ {“text”: “Move meeting to Friday”, “timestamp”: “2025-01-10T09:00:00”}, {“text”: “Cancel Thursday, Alice is sick”, “timestamp”: “2025-01-12T14:30:00”} ] # Reconcile contradictory chunks before they ever reach the prompt latest_relevant = max(retrieved_chunks, key=lambda chunk: chunk[“timestamp”]) retrieved_chunks = [     {“text”: “Move meeting to Friday”, “timestamp”: “2025-01-10T09:00:00”},     {“text”: “Cancel Thursday, Alice is sick”, “timestamp”: “2025-01-12T14:30:00”} ]   # Reconcile contradictory chunks before they ever reach the prompt latest_relevant = max(retrieved_chunks, key=lambda chunk: chunk[“timestamp”]) That one line of reconciliation logic is the difference between an agent that confidently restates a stale instruction, and one that correctly knows the meeting was cancelled. Compression This is an easy one to understand if you are familiar with compressing into ZIP files. In the context of agents and language models, this entails some algorithmic token reduction: keeping the key underlying data intact, while its physical footprint inside a

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/infinix-note-60-pro-pininfarina-edition-launched-in-india-supercar-design-6500mah-battery-90w-charging-check-price-3057501.html” on this server. Reference #18.eff43717.1782317985.21ba8d61 https://errors.edgesuite.net/18.eff43717.1782317985.21ba8d61

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/your-resume-may-never-reach-a-human-recruiter-ai-filters-read-it-first-heres-how-to-get-shortlisted-3057155.html” on this server. Reference #18.c4f43717.1782299289.3179b7b2 https://errors.edgesuite.net/18.c4f43717.1782299289.3179b7b2

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/telegram-restored-on-play-store-in-india-after-week-long-block-3057165.html” on this server. Reference #18.7ffdd417.1782295462.2143d7e9 https://errors.edgesuite.net/18.7ffdd417.1782295462.2143d7e9

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/whatsapp-may-now-flag-unknown-numbers-before-chat-begins-report-here-s-what-it-means-for-your-safety-3057430.html” on this server. Reference #18.eff43717.1782291317.1fbecc15 https://errors.edgesuite.net/18.eff43717.1782291317.1fbecc15

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/hack-of-the-day-how-to-read-whatsapp-messages-secretly-without-opening-the-chat-3057379.html” on this server. Reference #18.c4f43717.1782287194.303e5f74 https://errors.edgesuite.net/18.c4f43717.1782287194.303e5f74

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/instagram-reels-vs-posts-which-gets-more-reach-and-engagement-in-2026-heres-what-data-shows-3057398.html” on this server. Reference #18.c4f43717.1782283025.2fcba242 https://errors.edgesuite.net/18.c4f43717.1782283025.2fcba242

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/meta-ai-smart-glasses-launched-with-fresh-designs-8-hour-battery-check-price-and-upgraded-features-3057386.html” on this server. Reference #18.eff43717.1782278673.1f039dd5 https://errors.edgesuite.net/18.eff43717.1782278673.1f039dd5

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/microsoft-ceo-satya-nadella-warns-against-ai-concentration-3057194.html” on this server. Reference #18.54fdd417.1782273965.3072a2c4 https://errors.edgesuite.net/18.54fdd417.1782273965.3072a2c4