In this article, you will learn how a vector database works under the hood by building one from scratch in ten incremental steps using Python and NumPy. Topics we will cover include: How documents are encoded into fixed-size vectors and searched by meaning rather than by keyword. How to add metadata filtering, input validation, and persistence to a minimal vector database. How brute-force cosine similarity scales with corpus size, and when to consider approximate indexing. Introducing Vector Databases A vector database answers questions by meaning rather than by keyword. It operates by turning every document into a vector of numbers and then finding the numbers that point in a similar direction to your query (which has also been turned into a vector of numbers). This tutorial will demonstrate how to build a working vector database of your very own, through ten steps that each demonstrate one atomic idea. To follow along, create an empty script and name it something clever like tutorial.py. Append each step’s code to the script as you go and re-run it after you make sense of the commentary. The resulting output should make sense at that point. Nothing here needs a GPU or an API key; one small model downloads on the first run, and everything after that is plain NumPy. Step 1: Setup You need three files from this repository in your working directory: vector_db.py is the actual database which, yes, is already built for you… but the real magic is the understanding of the code and the interaction with it using the code herein. The good news is, once you go through this tutorial and understand the code, recreating the vector database on your own is nearly trivial. corpus.py contains 25 simulated sample documents and their topic tags. test.py is the test suite, only here to make you feel safe and secure that the vector database works properly as implemented, which you can verify by running at any point with python test.py. Install the two dependencies: pip install numpy sentence-transformers pip install numpy sentence–transformers Now start your tutorial.py file with the imports and two small display helpers. show() prints a list of search results as score, topic, document (relied upon later). header() just labels each section so the growing script’s output stays readable. import time from pathlib import Path import numpy as np from corpus import DOCS, META from vector_db import VectorDB WIDTH = 64 def header(title): print(f”\n{title}\n{‘─’ * len(title)}”) def show(results): if not results: print(” (no matches)”) for hit in results: text = hit.text if len(hit.text) <= WIDTH else hit.text[: WIDTH – 1] + “…” print(f” {hit.score:+.3f} [{hit.meta[‘topic’]:<7}] {text}”) print() 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 import time from pathlib import Path import numpy as np from corpus import DOCS, META from vector_db import VectorDB WIDTH = 64 def header(title): print(f“\n{title}\n{‘─’ * len(title)}”) def show(results): if not results: print(” (no matches)”) for hit in results: text = hit.text if len(hit.text) <= WIDTH else hit.text[: WIDTH – 1] + “…” print(f” {hit.score:+.3f} [{hit.meta[‘topic’]:<7}] {text}”) print() Running the script now produces no output. This is what we want; nothing has been called yet. Step 2: Building the Index Creating a VectorDB loads the embedding model, and add() encodes every document into a vector and stores it. header(“2. Building the index”) t0 = time.perf_counter() db = VectorDB() load_seconds = time.perf_counter() – t0 t0 = time.perf_counter() db.add(DOCS, META) encode_seconds = time.perf_counter() – t0 print(f” {db!r}”) print(f” model load: {load_seconds:5.2f}s”) print(f” encoding: {encode_seconds:5.2f}s for {len(db)} documents “ f”({encode_seconds / len(db) * 1000:.0f} ms each)”) print(f” index size: {db.vectors.nbytes / 1024:5.1f} KiB “ f”{db.vectors.shape} of {db.vectors.dtype}”) header(“2. Building the index”) t0 = time.perf_counter() db = VectorDB() load_seconds = time.perf_counter() – t0 t0 = time.perf_counter() db.add(DOCS, META) encode_seconds = time.perf_counter() – t0 print(f” {db!r}”) print(f” model load: {load_seconds:5.2f}s”) print(f” encoding: {encode_seconds:5.2f}s for {len(db)} documents “ f“({encode_seconds / len(db) * 1000:.0f} ms each)”) print(f” index size: {db.vectors.nbytes / 1024:5.1f} KiB “ f“{db.vectors.shape} of {db.vectors.dtype}”) Output: 2. Building the index ───────────────────── VectorDB(25 docs, dim=384, model=”sentence-transformers/all-MiniLM-L6-v2″) model load: 1.64s encoding: 0.14s for 25 documents (6 ms each) index size: 37.5 KiB (25, 384) of float32 2. Building the index ───────────────────── VectorDB(25 docs, dim=384, model=‘sentence-transformers/all-MiniLM-L6-v2’) model load: 1.64s encoding: 0.14s for 25 documents (6 ms each) index size: 37.5 KiB (25, 384) of float32 Note that the index size does not depend on how long the documents are. Every document, whether a six-word sentence or a six-page essay, becomes the same 384 numbers at 4 bytes each: 1,536 bytes, flat. That is fixed, and is what makes a vector index predictable to size and cheap to scan. Step 3: A First Search header(“3. A first search”) query = “what keeps a cell supplied with energy?” print(f’ query: “{query}”\n’) show(db.search(query, k=3)) header(“3. A first search”) query = “what keeps a cell supplied with energy?” print(f‘ query: “{query}”\n’) show(db.search(query, k=3)) Output: 3. A first search ───────────────── query: “what keeps a cell supplied with energy?” +0.589 [bio ] The mitochondria is the powerhouse of the cell. +0.440 [bio ] During aerobic respiration, mitochondria produce ATP through th… +0.329 [comics ] Thor’s mitochondria-rich muscle fibres make him a biological po… 3. A first search ───────────────── query: “what keeps a cell supplied with energy?” +0.589 [bio ] The mitochondria is the powerhouse of the cell. +0.440 [bio ] During aerobic respiration, mitochondria produce ATP through th… +0.329 [comics ] Thor‘s mitochondria–rich muscle fibres make him a biological po… The top hit shares exactly one word with the query (“cell”) and the runner-up shares none at all. A keyword index would have ranked these very differently, if it found them at all. Step 4: Searching Without Sharing a Single Word header(“4. Searching without sharing a single word”) for query in (“why does my loaf taste sour”, “superheroes”): print(f’ query: “{query}”\n’) show(db.search(query, k=3)) header(“4. Searching without sharing a single word”) for query in (“why does my loaf taste sour”, “superheroes”): print(f‘ query: “{query}”\n’) show(db.search(query, k=3)) Output: 4.
Agent or Workflow? A Practical Test for Knowing When You Actually Need an AI Agent
In this article, you will learn the key differences between AI workflows and agents, and how to decide which approach is right for your use case before writing a single line of code. Topics we will cover include: What distinguishes a workflow from an agent, using concrete examples of each. A practical single test to determine whether your application genuinely requires an agent. A five-point checklist to guide your decision before building. “Agent” has become one of the most overused words in AI. A chatbot with three tools gets called an agent. A fixed document-processing pipeline gets called an agent. A scheduled automation gets called an agent. Sometimes, a genuinely autonomous system that plans, acts, observes results, and changes its strategy is also called an agent. Due to all this hype, people often rely on agents even when their application does not actually need one. So, before going further, let’s briefly understand what an agent is and what a workflow is. What Is a Workflow? A workflow, also called a pipeline or chain, is a system where the control flow is fixed at design time. The developer decides the sequence of steps, branches, stop conditions, and other logic beforehand. You may still use an LLM for one or more steps, which makes it a hybrid system, but the overall path is predetermined. For example, if you have to process customer refunds, your workflow might look like this: You can see that there are decisions here. There are LLMs and tools as well. But it is still fundamentally a workflow because the possible paths are designed in advance. You could draw the state diagram before receiving the customer request. What Is an Agent? An agent is a system where the LLM itself decides what to do next at runtime. It receives a goal, has access to tools, and decides which tool to call, in what order, and when to stop. It can backtrack, loop, or gather more information depending on what it discovers. In other words, the control flow lives with the model. Let’s say you have a production outage and want to answer this question: Figure out why checkout failures increased in the last 30 minutes and produce a likely root cause. You may give the system tools for querying logs and metrics, searching error traces, and reading incident documents. But you cannot reliably know beforehand what the correct sequence of actions should be. For one incident, it might do: Check error rate → inspect recent deployment → inspect stack traces → identify failing database call → verify database latency Check error rate → inspect recent deployment → inspect stack traces → identify failing database call → verify database latency For another incident, it might do: Check error rate → segment failures by region → inspect CDN status → inspect DNS errors → identify regional provider outage Check error rate → segment failures by region → inspect CDN status → inspect DNS errors → identify regional provider outage Here, the observation after each action determines what the system does next. That is what makes it agentic. The Single Practical Test Ask yourself one question before writing any code: Can you draw a complete flowchart of the task before the LLM ever runs? If yes, and every major step and branch can be listed with reasonable confidence, build a workflow. If the next step depends on what the system discovers during execution — such as new data, unexpected tool results, or intermediate findings — you probably need an agent. This single test can eliminate many unnecessary agents. If a competent engineer can describe the process on a whiteboard using a reasonable number of conditional steps, the extra flexibility of an agent is usually not worth the added complexity. One common mistake is assuming: Workflow = simpleAgent = sophisticated That is not true. A workflow can contain multiple LLM calls, retrieval, tool calls, retry logic, human approvals, and complicated business rules. At the same time, a very simple system can still be agentic if the model itself decides what happens next. So, before making a decision, go through the checklist below. A Simple Checklist Before You Build 1. Can I list the major steps and branches before runtime? Yes → Workflow For example, if you want to extract information from a contract and save it to a database, the overall steps are already known. Read the contract, extract the fields, validate them, and save them. You may use an LLM for extraction, but you do not need an agent to decide what happens next. 2. Is the input variability low enough that a decision tree remains maintainable? Yes → Workflow If the inputs are open-ended and unpredictable, an agent may make more sense. For example: Help me solve this unusual customer issue. It may be difficult to create a fixed workflow for every possible issue. In this case, letting an agent decide dynamically what information to gather and what action to take can be useful. 3. Is the application sensitive to volume, cost, and latency? High volume, tight budget, or low-latency requirements → Workflow Agents normally require more reasoning and tool calls, which means more tokens, more API calls, and more latency. If the task is less frequent and valuable enough to justify exploring multiple possibilities — such as complex research or investigation — an agent may make sense. For high-volume FAQs or routine tasks, stick to workflows. 4. Do I need identical execution paths for audit or compliance? Strict audit or compliance requirements → Workflow For example: Verify identity → check credit → apply policy → approve/reject Verify identity → check credit → apply policy → approve/reject If every application must go through the same documented checks, use a workflow. If different investigation paths are acceptable as long as the final result is correct, an agent may be suitable. 5. Have I already tried a workflow with LLM judgment? This is usually the best place to start. For example, in
Access Denied
Access Denied You don’t have permission to access “http://zeenews.india.com/technology/meta-unveils-new-vr-glasses-with-built-in-ai-calls-it-the-most-advanced-device-yet-3073012.html” on this server. Reference #18.c4f43717.1790234050.b5a36597 https://errors.edgesuite.net/18.c4f43717.1790234050.b5a36597
Access Denied
Access Denied You don’t have permission to access “http://zeenews.india.com/world/too-powerful-too-intimidating-tech-giants-call-for-un-to-enforce-global-ai-safeguards-3072994.html” on this server. Reference #18.eff43717.1790224289.14a4faf3 https://errors.edgesuite.net/18.eff43717.1790224289.14a4faf3
Combining LLM Embeddings with Tabular Features in a Unified Scikit-learn Pipeline
In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with structured tabular features for classification tasks. Topics we will cover include: How to generate text embeddings using Hugging Face’s sentence-transformers library and wrap them in a custom scikit-learn transformer class. How to use a ColumnTransformer to run parallel preprocessing branches for text, numeric, and categorical features simultaneously. How to assemble and evaluate a complete, deployment-ready classification pipeline on a mixed dataset combining real text data with synthetic tabular features. Introduction Real-world tasks like ticket triage or customer churn prediction are typically addressed by building classification models. Yet, in an increasingly data-pervaded era, the data used to construct these models and perform inference on them rarely comes in a single flavor. We are often faced with a mix of tabular, structured data of numeric and qualitative nature, as well as unstructured data like text — for instance, ticket descriptions or customer messages. Feeding these data types together into machine learning models requires effective and unified pipelines that accommodate the latest data nuances and techniques to handle them. This article shows you how to build a clean, deployment-ready solution that encapsulates embeddings generated by open-source LLMs (language models) into a unified scikit-learn pipeline, bringing together text representations and tabular features of distinct types — all based on the use of a ColumnTransformer. To illustrate its use, we will consider a classification scenario for detecting spammer users in a customer base. Prerequisites Instead of resorting to a paid API like OpenAI’s or Google Gemini’s, or a massive open-source LLM like LLaMA 3, we will use a more lightweight, CPU-friendly solution to generate embeddings from a collection of texts: Hugging Face’s sentence-transformers. Depending on your running environment, all you may need is to install the following libraries and dependencies: !pip install -q sentence-transformers scikit-learn pandas numpy !pip install –q sentence–transformers scikit–learn pandas numpy Remove the ! if you are working in your own Python IDE rather than a cloud notebook environment like Google Colab. Step-by-Step Guide Here’s what our intended, unified scikit-learn pipeline architecture looks like: But first, we need a mixed dataset that looks reasonably realistic. For this, we adopt a hybrid approach: we pull a real dataset available on GitHub — the well-known SMS Spam Collection dataset containing users’ text messages labeled as spam or not — and augment it with synthetic tabular data features. Put together, the data will serve us to set up a customer churn/triage scenario. The code excerpt required for data generation is a bit large, but there are plenty of comments to help you understand every decision behind the synthetic data creation process: import pandas as pd import numpy as np # 1. Loading base text dataset from GitHub url = “https://raw.githubusercontent.com/justmarkham/pycon-2016-tutorial/master/data/sms.tsv” df = pd.read_csv(url, sep=’\t’, header=None, names=[‘label’, ‘message’]) # 2. Encoding original target variable first (0 for normal/ham, 1 for spam) df[‘target’] = df[‘label’].map({‘ham’: 0, ‘spam’: 1}) # 3. Synthesising meaningful tabular features WITH realistic overlap (noise) # Without noise and some degree of overlap, the classifier we will build would # easily achieve perfection: something not quite realistic in practice. np.random.seed(42) # Account Age: Normal users can be brand new, and spammers sometimes use older hacked accounts df[‘account_age_days’] = np.where( df[‘target’] == 1, np.random.randint(1, 365, df.shape[0]), # Spam: 1 to 365 days np.random.randint(1, 1500, df.shape[0]) # Ham: 1 to 1500 days (Massive overlap) ) # Premium Status: Adding a bit more noise here df[‘is_premium’] = np.where( df[‘target’] == 1, np.random.choice([‘no’, ‘yes’], df.shape[0], p=[0.95, 0.05]), # Spam: 95% free np.random.choice([‘no’, ‘yes’], df.shape[0], p=[0.80, 0.20]) # Ham: 80% free, 20% premium ) # Priority Score: Overlapping distributions so the model can’t rely on this feature alone to classify customers df[‘priority_score’] = np.where( df[‘target’] == 1, np.random.uniform(0.4, 1.0, df.shape[0]), # Spam: 0.4 to 1.0 np.random.uniform(0.0, 0.7, df.shape[0]) # Ham: 0.0 to 0.7 (Overlap between 0.4 and 0.7) ) # Viewing a sample of the logically cohesive mixed data df.head(3) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 import pandas as pd import numpy as np # 1. Loading base text dataset from GitHub url = “https://raw.githubusercontent.com/justmarkham/pycon-2016-tutorial/master/data/sms.tsv” df = pd.read_csv(url, sep=‘\t’, header=None, names=[‘label’, ‘message’]) # 2. Encoding original target variable first (0 for normal/ham, 1 for spam) df[‘target’] = df[‘label’].map({‘ham’: 0, ‘spam’: 1}) # 3. Synthesising meaningful tabular features WITH realistic overlap (noise) # Without noise and some degree of overlap, the classifier we will build would # easily achieve perfection: something not quite realistic in practice. np.random.seed(42) # Account Age: Normal users can be brand new, and spammers sometimes use older hacked accounts df[‘account_age_days’] = np.where( df[‘target’] == 1, np.random.randint(1, 365, df.shape[0]), # Spam: 1 to 365 days np.random.randint(1, 1500, df.shape[0]) # Ham: 1 to 1500 days (Massive overlap) ) # Premium Status: Adding a bit more noise here df[‘is_premium’] = np.where( df[‘target’] == 1, np.random.choice([‘no’, ‘yes’], df.shape[0], p=[0.95, 0.05]), # Spam: 95% free np.random.choice([‘no’, ‘yes’], df.shape[0], p=[0.80, 0.20]) # Ham: 80% free, 20% premium ) # Priority Score: Overlapping distributions so the model can’t rely on this feature alone to classify customers df[‘priority_score’] = np.where( df[‘target’] == 1, np.random.uniform(0.4, 1.0, df.shape[0]), # Spam: 0.4 to 1.0 np.random.uniform(0.0, 0.7, df.shape[0]) # Ham: 0.0 to 0.7 (Overlap between 0.4 and 0.7) ) # Viewing a sample of the logically cohesive mixed data df.head(3) Example output: The next step is key, as this is where we create the custom text transformer — see the leftmost branch in the previous diagram. In scikit-learn, this is done by creating a custom class that inherits from TransformerMixin and BaseEstimator. The requirement is to define fit() and transform() methods, just like any pre-existing data transformation class in the library (e.g. standard scalers and one-hot encoders). from
3 Ways to Enhance Your AI Model’s Interpretability
In this article, you will learn three concrete techniques for making machine learning model predictions interpretable, covering both global and local explanations across tree-based and neural network architectures. Topics we will cover include: Why traditional feature importance scores fall short as a complete interpretability solution, and when they mislead. How SHAP, LIME, and Integrated Gradients each work, and what makes each one suited to different deployment constraints. How to apply all three techniques to the same customer churn example so their explanations can be directly compared. A model that predicts accurately and a model whose reasoning you can actually explain are two different achievements, and only one of them is optional anymore. A churn model that flags a loyal, five-year customer as high-risk isn’t just an interesting edge case if nobody on the team can say why; it’s a decision nobody can defend, to a manager, to the customer, or increasingly, to a regulator. The EU AI Act’s Article 13 now requires high-risk AI systems to provide sufficient transparency for deployers to actually interpret their outputs, which has moved interpretability from a nice-to-have research topic to a genuine deployment requirement for a growing share of real systems. This article covers three concrete, current techniques for getting real answers out of a model that would otherwise stay a black box. One example runs through the whole piece: a customer churn prediction model, first a gradient-boosted tree, later a small neural network trained on the same data, so every technique is explaining the same underlying problem rather than jumping between disconnected toy examples. What Model Interpretability Actually Means Model interpretability is the degree to which a human can understand why a model produced a specific output, not just that it produced one. That definition splits cleanly into two questions that get conflated constantly, and untangling them now saves confusion in every section after this one. Global interpretability asks how the model behaves overall: across the whole dataset, which features matter most, and in which direction. Local interpretability asks something narrower and, for most real decisions, more important: why did the model make this prediction, for this customer, right now? A model can be reasonably interpretable globally — “tenure and contract length matter most on average” — while still being a total mystery locally, since knowing what matters on average tells you nothing about why one specific loyal customer just got flagged as a churn risk. The Traditional Method, and Why It Doesn’t Scale Ask most data scientists how to explain a tree-based model and the first answer is usually the same: pull the built-in .feature_importances_ attribute that ships with practically every scikit-learn ensemble model, or read the coefficients straight off a linear model. It’s fast, it requires no extra library, and it gives you a ranked list in one line of code. importances = pd.Series(model.feature_importances_, index=FEATURES).sort_values(ascending=False) importances = pd.Series(model.feature_importances_, index=FEATURES).sort_values(ascending=False) Run against the churn model, this returns tenure at the top, followed by monthly charge, support tickets, contract type, and late payments. That’s a real answer, and it’s also where the traditional method’s real limits start showing up. It’s global-only by construction; it can tell you tenure matters most across the whole customer base, but it says nothing at all about why one specific customer — someone with five years of tenure who should look safe — just got flagged as high-risk. It can also be measurably biased toward high-cardinality features, inflating the apparent importance of a variable simply because it has more possible split points, not because it’s genuinely more predictive. And it only exists at all for models that happen to expose that attribute; the moment you’re working with something that doesn’t ship a built-in importance score — a neural network, an ensemble of mixed model types, a black-box API you’re calling — this method has nothing to offer. That gap — no per-prediction explanation, a bias baked into how the score is computed, and no coverage outside a narrow set of model types — is exactly what the three techniques below exist to close. Prerequisites Python 3.11+ pip install shap lime scikit-learn pandas numpy torch captum pip install shap lime scikit–learn pandas numpy torch captum Every code snippet in the three sections below imports from one shared file, churn_data.py, which builds the synthetic churn dataset and trains the gradient-boosted tree model used in Ways 1 and 2. Save this first, before running anything else: # churn_data.py import numpy as np import pandas as pd from sklearn.ensemble import GradientBoostingClassifier from sklearn.model_selection import train_test_split rng = np.random.default_rng(42) n = 2000 tenure_months = rng.integers(1, 72, n) monthly_charge = rng.normal(70, 25, n).clip(15, 200) support_tickets = rng.poisson(1.5, n) contract_is_monthly = rng.integers(0, 2, n) # 1 = month-to-month, 0 = annual+ late_payments = rng.poisson(0.8, n) # True churn logic: short tenure, month-to-month contracts, and lots of # support tickets all push churn probability up; long tenure pulls it down logit = ( -1.5 – 0.04 * tenure_months + 0.015 * monthly_charge + 0.35 * support_tickets + 1.1 * contract_is_monthly + 0.25 * late_payments ) prob_churn = 1 / (1 + np.exp(-logit)) churned = (rng.uniform(0, 1, n) < prob_churn).astype(int) df = pd.DataFrame({ “tenure_months”: tenure_months, “monthly_charge”: monthly_charge, “support_tickets”: support_tickets, “contract_is_monthly”: contract_is_monthly, “late_payments”: late_payments, “churned”: churned, }) FEATURES = [“tenure_months”, “monthly_charge”, “support_tickets”, “contract_is_monthly”, “late_payments”] X = df[FEATURES] y = df[“churned”] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) model = GradientBoostingClassifier(random_state=42) model.fit(X_train, y_train) if __name__ == “__main__”: print(f”Train accuracy: {model.score(X_train, y_train):.3f}”) print(f”Test accuracy: {model.score(X_test, y_test):.3f}”) print(f”Churn rate in data: {y.mean():.1%}”) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 # churn_data.py import numpy as np import pandas as pd from sklearn.ensemble import GradientBoostingClassifier from sklearn.model_selection import train_test_split rng = np.random.default_rng(42) n = 2000 tenure_months = rng.integers(1, 72,
Access Denied
Access Denied You don’t have permission to access “http://zeenews.india.com/technology/samsung-india-profit-falls-36-to-rs-7228-crore-in-fy26-3072872.html” on this server. Reference #18.54fdd417.1790104547.20c5106 https://errors.edgesuite.net/18.54fdd417.1790104547.20c5106
Access Denied
Access Denied You don’t have permission to access “http://zeenews.india.com/technology/trai-mandates-shorter-validity-voice-sms-stvs-to-ensure-choices-for-low-income-consumers-3072862.html” on this server. Reference #18.eff43717.1790091750.6e8af8f https://errors.edgesuite.net/18.eff43717.1790091750.6e8af8f
The Roadmap to Mastering LLM Inference Optimization
The Roadmap to Mastering LLM Inference Optimization – MachineLearningMastery.com The Roadmap to Mastering LLM Inference Optimization The Roadmap to Mastering LLM Inference Optimization – MachineLearningMastery.com The Roadmap to Mastering LLM Inference Optimization
Access Denied
Access Denied You don’t have permission to access “http://zeenews.india.com/technology/google-fined-rs-4300-crore-million-over-location-data-what-went-wrong-3072834.html” on this server. Reference #18.eff43717.1790075971.44c151b https://errors.edgesuite.net/18.eff43717.1790075971.44c151b
