In this article, you will learn how pgvector compares to dedicated vector databases in terms of real, measured performance, and how to decide which approach suits your production workload. Topics we will cover include: What pgvector and dedicated vector databases are, and how they differ architecturally. How to set up and run a hands-on benchmark comparing pgvector and Qdrant on your own machine. What production-scale benchmark data actually shows about throughput, latency, and the tradeoffs between the two approaches. A team adds pgvector to their existing Postgres database on a Tuesday afternoon, embeddings and relational data sitting in the same table, one query doing both the filter and the similarity search. Six months later, a vendor pitch lands in their inbox insisting they need to migrate to a “real” vector database before they hit a wall. No benchmark accompanies the claim in either direction, just confidence. That gap — strong opinions with no numbers behind them — is exactly what this article exists to close. What pgvector Actually Is pgvector is an open-source Postgres extension, currently at version 0.8.6 and supporting Postgres 13 and later, according to its official GitHub repository. It adds a native vector column type to Postgres, along with exact and approximate nearest-neighbor search, directly inside the database you’re likely already running. What that means in practice: a vector embedding lives in the same table as the record it describes, in the same row if you want, queried with the same SQL you already know. There’s no separate service to stand up, no second system to keep in sync, no network hop between your relational data and your vector data. pgvector supports single-precision, half-precision, binary, and sparse vector types, and five distance functions — L2, inner product, cosine, L1, and Hamming — covering essentially every distance metric a real embedding model uses. What a Dedicated Vector Database Is A dedicated vector database is a separate, purpose-built service, run as its own process with its own storage engine, designed from the ground up around vector search rather than having it added onto a general-purpose database. It’s a genuinely different architectural choice, not just a different product, since it means a new system to deploy, monitor, back up, and keep synchronized with whatever holds your actual source of truth. Here’s a plain rundown of the five most commonly reached for: Database Best for Pinecone A fully managed, zero-setup cloud database, best for low-ops enterprise scale Qdrant A high-performance Rust-based database, best for fast queries with complex JSON payload filtering Weaviate An open-source, AI-native system, best for built-in vectorization and hybrid BM25 search Milvus A distributed, GPU-accelerated database, best for massive, billion-scale enterprise workloads ChromaDB A lightweight, embedded database, best for local prototyping and Python development Prerequisites Everything in this tutorial runs entirely on your own machine — no cloud account required. You’ll need Docker and Docker Compose installed, Python 3.10 or later, roughly 8GB of free RAM, and basic comfort reading SQL and Python. The hands-on benchmark below is sized to run comfortably on a laptop. Setting Up pgvector Locally Start with a Docker Compose file that runs Postgres with pgvector already built in: # docker-compose.yml services: postgres: image: pgvector/pgvector:pg17 ports: – “5432:5432″ environment: – POSTGRES_PASSWORD=benchmark – POSTGRES_DB=vectorbench # docker-compose.yml services:   postgres:     image: pgvector/pgvector:pg17     ports:       – “5432:5432”     environment:       – POSTGRES_PASSWORD=benchmark       – POSTGRES_DB=vectorbench Run docker compose up -d, then connect and set up the extension and table: import psycopg2 conn = psycopg2.connect( host=”localhost”, port=5432, dbname=”vectorbench”, user=”postgres”, password=”benchmark”, ) conn.autocommit = True cur = conn.cursor() # Enables the vector type and index access methods for this database cur.execute(“CREATE EXTENSION IF NOT EXISTS vector;”) # 1536 dimensions matches OpenAI’s text-embedding-3-small, a realistic # stand-in for a genuine production embedding model cur.execute(“”” CREATE TABLE IF NOT EXISTS items ( id bigserial PRIMARY KEY, embedding vector(1536) ); “””) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 import psycopg2   conn = psycopg2.connect(     host=“localhost”, port=5432, dbname=“vectorbench”,     user=“postgres”, password=“benchmark”, ) conn.autocommit = True cur = conn.cursor()   # Enables the vector type and index access methods for this database cur.execute(“CREATE EXTENSION IF NOT EXISTS vector;”)   # 1536 dimensions matches OpenAI’s text-embedding-3-small, a realistic # stand-in for a genuine production embedding model cur.execute(“”“     CREATE TABLE IF NOT EXISTS items (         id bigserial PRIMARY KEY,         embedding vector(1536)     ); ““”) pgvector/pgvector:pg17 is the official Docker image that bundles the extension with Postgres 17, so there’s no separate compile step. CREATE EXTENSION IF NOT EXISTS vector is the one-time step that registers the vector type and its index methods in this specific database, per the pgvector documentation. Setting Up Qdrant Locally For the dedicated-database side of the comparison, run Qdrant with a single Docker command: docker run -d –name qdrant-bench -p 6333:6333 -p 6334:6334 qdrant/qdrant docker run –d —name qdrant–bench –p 6333:6333 –p 6334:6334 qdrant/qdrant Then connect from Python and create a matching collection: from qdrant_client import QdrantClient from qdrant_client.models import Distance, VectorParams qdrant = QdrantClient(url=”http://localhost:6333″) qdrant.create_collection( collection_name=”items”, vectors_config=VectorParams(size=1536, distance=Distance.COSINE), ) from qdrant_client import QdrantClient from qdrant_client.models import Distance, VectorParams   qdrant = QdrantClient(url=“http://localhost:6333”)   qdrant.create_collection(     collection_name=“items”,     vectors_config=VectorParams(size=1536, distance=Distance.COSINE), ) Qdrant is worth picking specifically for this comparison because it’s the subject of the most detailed, methodologically transparent public benchmark against pgvector available, which this article draws on directly in a later section. VectorParams(size=1536, distance=Distance.COSINE) mirrors the same dimensionality and distance metric used in the pgvector table above, which matters — a fair comparison needs both systems measuring the same thing. Generating and Loading a Real Test Dataset With both systems running, generate a realistic set of synthetic vectors and load them into each: import numpy as np NUM_VECTORS = 100_000 DIM = 1536 rng = np.random.default_rng(seed=42) vectors = rng.normal(size=(NUM_VECTORS, DIM)).astype(np.float32) # Real embedding models produce normalized vectors, so normalize # these to length 1 for a realistic cosine-distance comparison vectors /= np.linalg.norm(vectors, axis=1, keepdims=True) import numpy as np   NUM_VECTORS = 100_000 DIM = 1536   rng

In this article, you will learn how to benchmark a deterministic 3-Tiered Graph-RAG system against a standard vector RAG pipeline on fact-dense queries, and what the results reveal about matching prompt complexity to model capacity. Topics we will cover include: How to build a synthetic, fact-dense sports dataset that intentionally introduces noisy, contradictory text into a vector database while keeping ground truth in a graph store. How to implement and compare two retrieval functions — one for standard vector RAG and one for the 3-Tiered Graph-RAG — using ChromaDB and a local Flan-T5 model. Why a small-capacity local LLM can underperform with complex, conflict-resolution prompts, and what this means for production RAG system design. Introduction In a previous article, a deterministic, 3-Tiered Graph-RAG system architecture was proposed to address “lossiness” in dense vector databases when dealing with atomic facts. Because semantic proximity vector search and retrieval rely on similarity, classical RAG pipelines often confuse numeric data, historical stats, or entity relationships when several similar entities coexist in a latent space. The proposed architecture tackled this limitation through a layered approach consisting of: A lightweight QuadStore knowledge graph based on a SPOC (Subject-Predicate-Object-Context) schema. A standard ChromaDB vector database capable of managing fuzzy, long-tail contexts. In production-oriented engineering, however, hard numbers are critical. Accordingly, we now illustrate a benchmark between standard Vector RAG and our 3-Tiered Graph-RAG to measure how much reduction in hallucinations can be expected when querying over a fact-dense dataset. Prerequisites and Setup To run the example benchmark in this article, you’ll need to install these libraries if you don’t have them in your IDE — if you try the example out in a Google Colab notebook, install them all, as follows: !pip install -q chromadb transformers !pip install –q chromadb transformers Next, we import all the modules and functions we will need and set up a local LLM, in particular, the google/flan-t5-base model available with Hugging Face’s transformers library: import chromadb from transformers import AutoModelForSeq2SeqLM, AutoTokenizer import random print(“Initializing local, free-tier LLM (flan-t5-base)…”) tokenizer = AutoTokenizer.from_pretrained(“google/flan-t5-base”) model = AutoModelForSeq2SeqLM.from_pretrained(“google/flan-t5-base”) def llm(prompt): “””Wrapper to generate text directly from the local model””” inputs = tokenizer(prompt, return_tensors=”pt”) outputs = model.generate(**inputs, max_new_tokens=15) return tokenizer.decode(outputs[0], skip_special_tokens=True) import chromadb from transformers import AutoModelForSeq2SeqLM, AutoTokenizer import random   print(“Initializing local, free-tier LLM (flan-t5-base)…”) tokenizer = AutoTokenizer.from_pretrained(“google/flan-t5-base”) model = AutoModelForSeq2SeqLM.from_pretrained(“google/flan-t5-base”)   def llm(prompt):     “”“Wrapper to generate text directly from the local model”“”     inputs = tokenizer(prompt, return_tensors=“pt”)     outputs = model.generate(**inputs, max_new_tokens=15)     return tokenizer.decode(outputs[0], skip_special_tokens=True) Next, we build a simple QuadStore, following the same principles and guidelines as the one shown in the previous article: class SimpleQuadStore: def __init__(self): self.facts = set() def add(self, subject, predicate, obj, context): self.facts.add((subject, predicate, str(obj), context)) def query(self, subject): return [f for f in self.facts if f[0] == subject] qs = SimpleQuadStore() class SimpleQuadStore:     def __init__(self):         self.facts = set()              def add(self, subject, predicate, obj, context):         self.facts.add((subject, predicate, str(obj), context))              def query(self, subject):         return [f for f in self.facts if f[0] == subject]   qs = SimpleQuadStore() Likewise, we need to initialize a Vector DB. We do this by instantiating a local ChromaDB collection that will handle unstructured chunks of text. chroma_client = chromadb.Client() collection = chroma_client.create_collection(name=”sports_stats”) chroma_client = chromadb.Client() collection = chroma_client.create_collection(name=“sports_stats”) Next, we get some fresh data — and for this example, it couldn’t be fresher: we synthetically create it from scratch. The following code generates a synthetic dataset with performance information for 50 basketball players. It is designed to inject accurate truths into the graph database while intentionally feeding messy, contradictory paragraphs into the vector DB, just as often happens in the real-world texts that RAG systems are built upon. benchmark_queries = [] print(“Populating databases with synthetic sports data…”) for i in range(50): player = f”Player_{i}” real_ppg = str(random.randint(15, 30)) fake_half = str(random.randint(2, 10)) fake_career = str(random.randint(11, 14)) # Graph DB receives the absolute truth qs.add(player, “season_ppg”, real_ppg, “Sports_DB”) # Vector DB receives messy unstructured text with multiple numbers text = ( f”During the recent game, {player} had a terrible first half, scoring only {fake_half} points. “ f”Historically, his career average sat around {fake_career}. “ f”However, his official season average PPG is currently {real_ppg}.” ) collection.add(documents=[text], ids=[f”doc_{i}”]) benchmark_queries.append({ “question”: f”What is the official season average PPG for {player}?”, “entity”: player, “true_answer”: real_ppg }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 benchmark_queries = [] print(“Populating databases with synthetic sports data…”)   for i in range(50):     player = f“Player_{i}”     real_ppg = str(random.randint(15, 30))     fake_half = str(random.randint(2, 10))     fake_career = str(random.randint(11, 14))       # Graph DB receives the absolute truth     qs.add(player, “season_ppg”, real_ppg, “Sports_DB”)       # Vector DB receives messy unstructured text with multiple numbers     text = (         f“During the recent game, {player} had a terrible first half, scoring only {fake_half} points. “         f“Historically, his career average sat around {fake_career}. “         f“However, his official season average PPG is currently {real_ppg}.”     )     collection.add(documents=[text], ids=[f“doc_{i}”])       benchmark_queries.append({         “question”: f“What is the official season average PPG for {player}?”,         “entity”: player,         “true_answer”: real_ppg     }) We are almost there. Before running the benchmarking pipeline, the two missing pieces are the retrieval functions — one for each system. The standard RAG uses a simple context prompt, while the 3-Tiered Graph-RAG employs a strict rule-based prompt that forces the LLM to give first priority to graph facts over vector fallbacks. def standard_vector_rag(q): # Simulating real-world context pollution by pulling top 3 similar chunks results = collection.query(query_texts=[q], n_results=3) context = ” “.join(results[“documents”][0]) # Joining the 3 paragraphs together prompt = f”Context: {context}\nQuestion: {q}\nAnswer strictly with the exact number:” return llm(prompt) def deterministic_3_tier_rag(q, entity): graph_res = qs.query(entity) p1_context = f”{entity} season average PPG is {graph_res[0][2]}” if graph_res else “None” p3_context = collection.query(query_texts=[q], n_results=1)[“documents”][0][0] prompt = f”””Context 1 (Absolute Truth): {p1_context} Context 2 (Fallback Text): {p3_context} Question: {q} Answer strictly using Context 1 with the exact number:””” return llm(prompt) 1 2 3 4 5 6

In this article, you will learn what the major AI agent benchmarks measure, how they differ from traditional model benchmarks, and how to use them as directional signals rather than performance guarantees. Topics we will cover include: How agent benchmarks like SWE-Bench, GAIA, WebArena, and OSWorld differ from one another and what each one actually tests. What leaderboard scores consistently leave out, including cost, latency, safety, and error recovery. How to combine public benchmarks with private evaluations to make more reliable decisions about agent deployment. Introduction Every AI provider claims their agent is state of the art. Leaderboards update weekly. Marketing pages glow with impressive percentages. But for anyone who has actually deployed an AI agent in production, one question matters more than any score: does it reliably do the job you hired it for? This guide cuts through the leaderboard noise. You’ll learn what the major AI agent benchmarks actually measure, why each one captures only a slice of real-world capability, and how to use these evaluations as directional signals rather than performance guarantees. Agent Benchmarks vs. Model Benchmarks Before examining specific evaluations, it helps to understand what separates an agent benchmark from a standard model benchmark. Traditional benchmarks like MMLU or HumanEval test a model’s ability to generate a correct response in a single turn. You ask a question, the model answers, and a judge scores the output. These tests are useful for measuring raw language understanding and reasoning, but they say nothing about whether a system can navigate the real world. Agent benchmarks introduce a categorically different challenge. They drop a model into an environment where it must take sequences of actions over multiple steps, use external tools, adapt when something goes wrong, and ultimately achieve a goal that can’t be reached in one response. The model might browse websites, write and execute code, click through interfaces, or manipulate files. Failure isn’t just a wrong answer — it might be a stuck loop, a broken workflow, or a task abandoned halfway through. The contrast is stark: a traditional benchmark is a written exam. An agent benchmark is a live work simulation where the candidate must navigate real tools, real friction, and real surprises. With that distinction in mind, let’s walk through the major evaluations and what each one actually tests. The Major Benchmarks, Explained SWE-Bench and SWE-Bench Verified SWE-bench is the closest thing to a gold standard for software engineering agents. It presents the agent with a real GitHub issue drawn from popular Python repositories, including projects like Django and scikit-learn, and asks the agent to produce a code patch that resolves the problem. Success is measured by whether the patch passes the repository’s existing unit tests. SWE-bench Verified is a curated subset where human engineers have confirmed that each task is genuinely solvable and that the unit tests accurately reflect whether the fix worked. This addresses a real weakness of the original benchmark: some tasks were ambiguous or had flawed test suites that could pass with incorrect solutions. The catch is worth understanding. Passing a unit test suite is not the same as writing good code. An agent can produce a patch that technically passes every test while simultaneously introducing a memory leak, creating a security vulnerability, or solving only a narrow edge case while breaking three others. SWE-bench measures whether an agent can satisfy a specific, automated criterion — it doesn’t measure code quality, maintainability, or safety. With that limitation in mind, let’s look at a benchmark targeting a very different capability. GAIA GAIA, which stands for General AI Assistants, targets the ability to answer complex questions that require reasoning across multiple tools and modalities. A GAIA task might ask an agent to look up a document, cross-reference it with information from the web, perform a calculation, and synthesize a final answer. What makes GAIA distinctive is that its questions are designed to be trivially easy for a human with internet access but genuinely difficult for current AI systems. The benchmark tests whether an agent can coordinate web browsing, file handling, and multi-step deduction in the right sequence without losing track of the original goal. The limitation is that GAIA tasks, by design, tend to have single correct answers. Real assistant workflows are messier: goals shift, instructions are ambiguous, and the right answer often depends on context that an agent can’t retrieve from a search engine. From reasoning and retrieval, we can move to a benchmark that focuses squarely on something most agents encounter daily: the web browser. WebArena and BrowserGym WebArena tests whether an agent can navigate realistic web applications to complete multi-step tasks. Rather than using live websites, it spins up sandboxed versions of e-commerce platforms, content management systems, forums, and other common web environments. The agent must interact with these interfaces the way a human would: clicking buttons, filling out forms, navigating menus, and interpreting page content. BrowserGym extends this into a more modular framework, allowing researchers to plug in different web environments and evaluation criteria. Together, these benchmarks are the best current tools for evaluating browser-based agents. The contamination problem looms large here. Many of WebArena’s original tasks have been in circulation long enough that they may appear in the training data of newer models. A high score could reflect genuine capability or pattern-matching against memorized task structures. Researchers and vendors rarely disclose how thoroughly they’ve checked for this. If WebArena represents the browser, the final benchmark takes things a step further, handing the agent an entire computer. OSWorld OSWorld takes the most expansive view of agent capability. It places an agent inside a sandboxed desktop operating system and tasks it with completing real computer-use workflows: manipulating files, navigating GUI applications, adjusting system settings, and coordinating across multiple programs. This benchmark is the most ambitious in scope because it mirrors how a human assistant would actually operate a computer. It’s also the most fragile. If the simulated operating system crashes, the display renders incorrectly, or an application behaves unexpectedly, the

In this article, you will learn how to build a lightweight, end-to-end RAG pipeline using scikit-learn for vector retrieval and open-source models for embeddings and text generation, without the use of paid APIs or heavyweight frameworks. Topics we will cover include: How to download a document dataset, chunk it into semantic pieces, and generate embeddings using a local sentence-transformer model. How to build a custom retriever class using scikit-learn’s NearestNeighbors with cosine distance as a stand-in for a vector database. How to wire the retriever together with a lightweight open-source LLM to produce a fully functional, in-memory RAG pipeline. Introduction Modern retrieval augmented generation (RAG systems) are often thought of as inseparable from an expansive ecosystem of frameworks, including typically expensive, managed vector databases to store document collections for retrieval. At the core, however, RAG systems rely on a fairly simple notion: calculating distances between semantic vectors representing texts. It’s not strictly necessary to navigate the intricacies of a dedicated AI infrastructure when working with a small-to-medium-sized corpus, prototyping a new proof-of-concept, or deploying to environments under cost constraints. Instead, we can use scikit-learn procedures together with open-source LLMs to build transparent, fast RAG pipelines that remain fully functional. This article shows how. Step-by-Step Guide The solution provided below is completely free to use, relying on sentence-transformers to build local embeddings from a text dataset and a lightweight pre-trained LLM from Hugging Face for the generation stage. If you need to brush up on basic RAG concepts, check out this article first. While the pipeline can be implemented and run in a local IDE, for demonstration purposes I recommend running the entire pipeline on Google Colab, using its free T4 GPU for more efficient execution. If you follow this approach, start by installing and importing the required libraries and dependencies: !pip install -q sentence-transformers transformers import os import glob import urllib.request import zipfile import numpy as np import pandas as pd from sklearn.neighbors import NearestNeighbors from sentence_transformers import SentenceTransformer from transformers import pipeline print(“Libraries loaded successfully!”) !pip install –q sentence–transformers transformers   import os import glob import urllib.request import zipfile import numpy as np import pandas as pd from sklearn.neighbors import NearestNeighbors from sentence_transformers import SentenceTransformer from transformers import pipeline   print(“Libraries loaded successfully!”) I made available a small dataset containing nine text files — brief descriptions of Asian countries — in a public GitHub repository. You can download it, extract the files, and save the texts in a simple list as follows: # Downloading and extracting the toy dataset, containing 9 documents url = “https://github.com/gakudo-ai/open-datasets/blob/main/asia_documents.zip?raw=true” zip_path = “asia_documents.zip” extract_dir = “asia_documents” urllib.request.urlretrieve(url, zip_path) with zipfile.ZipFile(zip_path, ‘r’) as zip_ref: zip_ref.extractall(extract_dir) # Reading all extracted text files documents = [] for filepath in glob.glob(f”{extract_dir}/**/*.txt”, recursive=True): with open(filepath, ‘r’, encoding=’utf-8′) as file: documents.append(file.read()) print(f”Loaded {len(documents)} documents.”) # If the zip contained a CSV instead, you’d use pd.read_csv() here. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Downloading and extracting the toy dataset, containing 9 documents url = “https://github.com/gakudo-ai/open-datasets/blob/main/asia_documents.zip?raw=true” zip_path = “asia_documents.zip” extract_dir = “asia_documents”   urllib.request.urlretrieve(url, zip_path) with zipfile.ZipFile(zip_path, ‘r’) as zip_ref:     zip_ref.extractall(extract_dir)   # Reading all extracted text files documents = [] for filepath in glob.glob(f“{extract_dir}/**/*.txt”, recursive=True):     with open(filepath, ‘r’, encoding=‘utf-8’) as file:         documents.append(file.read())   print(f“Loaded {len(documents)} documents.”) # If the zip contained a CSV instead, you’d use pd.read_csv() here. The dataset we just loaded will constitute our miniature document base for RAG. We need to put it in the right format — that is, turn the texts into embedding vectors. Before doing that, it is common practice in RAG to chunk documents into smaller, semantic pieces that the embedding model — responsible for mapping pieces of text into numerical embeddings — can process more effectively. The texts in our dataset don’t really require chunking because they are quite short, but we are simulating this step to illustrate best practices in real-world RAG pipelines. We define a native chunking function, using smaller-than-usual values for chunk size and overlap (both measured in words) so that we can observe some actual effect on our small documents: def chunk_text(text, chunk_size=25, overlap=5): “””Splits text into chunks natively, without external frameworks.””” words = text.split() chunks = [] for i in range(0, len(words), chunk_size – overlap): chunk = ” “.join(words[i:i + chunk_size]) chunks.append(chunk) return chunks # Processing all documents into a flat list of chunks all_chunks = [] for doc in documents: all_chunks.extend(chunk_text(doc)) print(f”Created {len(all_chunks)} chunks from the original documents.”) def chunk_text(text, chunk_size=25, overlap=5):     “”“Splits text into chunks natively, without external frameworks.”“”     words = text.split()     chunks = []     for i in range(0, len(words), chunk_size – overlap):         chunk = ” “.join(words[i:i + chunk_size])         chunks.append(chunk)     return chunks   # Processing all documents into a flat list of chunks all_chunks = [] for doc in documents:     all_chunks.extend(chunk_text(doc))   print(f“Created {len(all_chunks)} chunks from the original documents.”) Output: Created 31 chunks from the original documents. Created 31 chunks from the original documents. In a real-world application, you will likely use significantly larger values for both arguments in the chunking function. Next, we generate the embeddings using a popular lightweight model for this task: the ‘all-MiniLM-L6-v2′ sentence-transformer: # Initializing the embedding model (runs locally on Colab’s free CPU/GPU) embedding_model = SentenceTransformer(‘all-MiniLM-L6-v2’) # Generating embeddings for all text chunks print(“Generating embeddings… (this is fast on Colab!)”) chunk_embeddings = embedding_model.encode(all_chunks, show_progress_bar=True) # Initializing the embedding model (runs locally on Colab’s free CPU/GPU) embedding_model = SentenceTransformer(‘all-MiniLM-L6-v2’)   # Generating embeddings for all text chunks print(“Generating embeddings… (this is fast on Colab!)”) chunk_embeddings = embedding_model.encode(all_chunks, show_progress_bar=True) Once we have a collection of embeddings, it’s time to define one of the key components of any RAG pipeline: the retriever. Since we are doing this the scikit-learn way, we will wrap sklearn.neighbors.NearestNeighbors — a simple yet effective distance calculation algorithm — into a clean, reusable class that mimics the functionality of a vector database, all based on cosine distance, which is ideal for embedding vector representations. class SklearnRetriever: def __init__(self,

In this article, you will learn how to fine-tune Llama 3 8B for custom tool calling using Unsloth and QLoRA, so that the model reliably outputs structured JSON payloads for a specific API schema. Topics we will cover include: Why prompt engineering alone is insufficient for reliable tool calling, and how fine-tuning with QLoRA addresses the problem at the weight level. How to build and format a tool-calling dataset using Llama 3’s chat template, and what makes a high-quality training example. How to train, test, and save a fine-tuned tool-calling adapter that runs on a free Google Colab T4 GPU in under 10 minutes. How to Fine-Tune Llama 3 for Custom Tool Calling with Unsloth in PythonImage by Author Introduction Llama 3 is a capable generalist, but that’s exactly the problem when you need an agent that reliably outputs structured JSON for a specific API. Prompt engineering can nudge a model toward a format, but it can’t guarantee it, and a single malformed response breaks an entire agentic pipeline. Fine-tuning changes the model’s behavior at the weight level. As covered in The Machine Learning Practitioner’s Guide to Fine-Tuning Language Models, fine-tuning earns its cost when you need deep specialization: a custom tone, strict output formatting, or entirely new capabilities. Tool calling checks all three boxes. In this tutorial, you’ll use Unsloth and QLoRA to fine-tune Llama 3 8B for custom tool calling: the ability to interpret a user query and return a perfectly formatted JSON payload matching your API schema. The entire workflow runs on a free Google Colab T4 GPU. Why Tool Calling Needs Fine-Tuning Before writing any code, it’s worth understanding why prompt engineering alone falls short for tool calling. When you ask a base model to return {“name”: “get_weather”, “arguments”: {“location”: “Tokyo”}}, it will often comply in testing, but under pressure from complex queries or long conversations, it drifts. It adds prose, swaps quote styles, or omits required fields. Fine-tuning addresses this because you’re teaching the model a new behavioral pattern, not just a new piece of knowledge. The distinction matters: retrieval-augmented generation (RAG) works when you need external facts; fine-tuning works when you need consistent behavior. Strict JSON output for a custom API schema is a behavior problem. QLoRA makes this practical. By freezing the base model’s weights and injecting small trainable matrices into the attention layers, you update roughly 1% of parameters while keeping memory low enough for a free Colab T4. As detailed in How to Fine-Tune a Local Mistral or Llama 3 Model on Your Own Dataset, this approach achieves results comparable to full fine-tuning at a fraction of the compute cost. Setting Up the Environment and Installing Dependencies With the “why” covered, let’s set up the tools. Open a new Google Colab notebook, switch the runtime to a T4 GPU (Runtime → Change runtime type → T4 GPU), then run the installation block. !pip install “unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git” !pip install –no-deps xformers trl peft accelerate bitsandbytes !pip install “unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git” !pip install —no–deps xformers trl peft accelerate bitsandbytes Once installation finishes, confirm your environment is ready: from unsloth import FastLanguageModel import torch print(f”CUDA available: {torch.cuda.is_available()}”) print(f”GPU: {torch.cuda.get_device_name(0)}”) from unsloth import FastLanguageModel import torch   print(f“CUDA available: {torch.cuda.is_available()}”) print(f“GPU: {torch.cuda.get_device_name(0)}”) You should see your T4 GPU confirmed. Unsloth handles CUDA configuration automatically, so no additional setup is required before loading the model. Loading the Model and Configuring LoRA Now load Llama 3 8B using Unsloth’s pre-quantized model. This version is already stored in 4-bit format, which means no gated Hugging Face token is required and the download is roughly four times faster than pulling the original weights. model, tokenizer = FastLanguageModel.from_pretrained( model_name=”unsloth/llama-3-8b-Instruct-bnb-4bit”, max_seq_length=1024, dtype=None, load_in_4bit=True, ) model, tokenizer = FastLanguageModel.from_pretrained(     model_name=“unsloth/llama-3-8b-Instruct-bnb-4bit”,     max_seq_length=1024,     dtype=None,     load_in_4bit=True, ) With the base model and tokenizer loaded, attach the LoRA adapters. This is where the QLoRA approach becomes concrete: instead of updating all the model’s weights, you’re injecting small, trainable matrices into specific layers. The r parameter (rank) controls how many parameters you train, and lower values reduce overfitting risk when working with small datasets. Note that Unsloth requires lora_dropout=0 to keep its optimized training kernels active; setting a non-zero value forces a fallback to slower code paths. model = FastLanguageModel.get_peft_model( model, r=8, target_modules=[“q_proj”, “k_proj”, “v_proj”, “o_proj”, “gate_proj”, “up_proj”, “down_proj”], lora_alpha=16, lora_dropout=0, bias=”none”, use_gradient_checkpointing=”unsloth”, random_state=42, ) print(“LoRA adapters attached — training ~1% of parameters”) model = FastLanguageModel.get_peft_model(     model,     r=8,     target_modules=[“q_proj”, “k_proj”, “v_proj”, “o_proj”,                     “gate_proj”, “up_proj”, “down_proj”],     lora_alpha=16,     lora_dropout=0,     bias=“none”,     use_gradient_checkpointing=“unsloth”,     random_state=42, ) print(“LoRA adapters attached — training ~1% of parameters”) A rank of 8 works well for a focused behavioral task like tool calling. Higher ranks give the model more expressive capacity but increase the risk of overfitting on small datasets. Set lora_alpha to twice r as a reliable starting point. At this point, the model is loaded, quantized, and wired with LoRA adapters, but it still doesn’t know anything about your tools. That’s where the dataset comes in. Building the Tool-Calling Dataset This is the most consequential step in the workflow. The model and LoRA configuration give you the machinery, but the dataset teaches the model what to do with it. Getting the data structure right here determines whether your fine-tuned model produces clean JSON or hallucinated half-answers. Tool-calling data has three required components in each example: a system prompt that defines the available tools and their JSON schemas, a user query in natural language, and the exact JSON output the model must produce. The code below builds a small sample dataset demonstrating two tools: get_weather and fetch_stock_price. The model loads first because the tokenizer is needed to apply Llama 3’s chat template correctly in the next step. tool_calling_data = [ { “system”: “””You have access to these tools: get_weather(location: str) -> dict fetch_stock_price(ticker: str) -> dict Respond ONLY with a valid JSON tool call. No other text.”””, “user”: “What is the weather like in Tokyo right now?”, “output”: ‘{“name”: “get_weather”, “arguments”: {“location”: “Tokyo”}}’ }, { “system”: “””You have access to these tools: get_weather(location:

In this article, you will learn how to choose the right chunking strategy for a retrieval-augmented generation system by working through a structured decision tree tied to the properties of your documents and retrieval task. Topics we will cover include: What each major chunking strategy assumes about your documents, and when those assumptions hold. A six-question decision tree that maps concrete document and retrieval conditions to specific chunking strategies. The failure signals that indicate a chosen strategy is not working, and the fixes that tend to resolve them. Introduction When you are building a retrieval-augmented generation system, you often work with documents that are larger than what an embedding model or LLM context window can accommodate, so you need to split them into smaller chunks. Chunk size and overlap may seem like straightforward configuration choices, but the way you set those boundaries directly affects retrieval. Poor boundaries can cause retrieval to miss the passage that answers a query, or return a chunk that is technically relevant but lacks the surrounding context needed to make it useful. A chunking strategy defines how you decide where to split a document, and that choice can have a major effect on retrieval quality. When the boundaries preserve the right context, embeddings, vector search, and generation have a stronger basis to work with. When the boundaries break important context, reranking and prompt tuning cannot fully recover what was lost during splitting. No single chunking method works well for every document. The right choice, therefore, depends on the structure of the document and the retrieval task. This article covers: What each major chunking strategy assumes about your documents A decision tree that maps every one of those strategies to a concrete condition worth checking How the strategies land on real document types, with tradeoffs laid out side by side The failure signals that tell you a chosen strategy isn’t working We start with why this choice is important. What Each Chunking Strategy Assumes Each chunking strategy is based on a specific assumption about the document. Choosing a strategy that matches the document’s structure and needs helps improve retrieval quality while avoiding unnecessary complexity: Strategy What It Assumes Fixed-size chunking Boundary placement doesn’t matter much, since overlap can patch whatever gets cut mid-thought Recursive chunking Paragraph and sentence boundaries are a safe enough proxy for meaning when no stronger structure is available Document-based chunking The document’s own structure — headers, sections, function boundaries — lines up with its semantic structure Semantic chunking Topic shifts are detectable sentence by sentence through embedding distance Late chunking A chunk’s meaning depends on the whole document, not just its own text Adaptive chunking A single chunk size can’t serve sections that vary sharply in information density Hierarchical chunking Retrieval needs to answer both broad and specific questions from the same corpus LLM-based chunking Only a model reading the text can find logical breakpoints that structure and embeddings miss Agentic chunking The collection is varied enough that no single rule handles all of it well When the strategy fits the document, its added complexity can be worthwhile. When it does not, it adds overhead without improving retrieval. For example, semantic chunking is unnecessary for a short FAQ entry, while fixed-size chunking can split related clauses in a contract. These mismatches can lead to retrieval problems that originate in preprocessing rather than generation. Working Through a Decision Tree to Choose a Chunking Strategy The tree has six branching questions, each narrowing the strategy space based on a concrete property of the document or the retrieval task. Every question below ends with a direct answer: which strategy to use if the condition holds, and where to go next if it doesn’t. Following them in order helps you choose a strategy based on the document and retrieval task rather than defaulting to a familiar option. Question 1: Does the Document Need Chunking at All? Short, self-contained content — such as FAQ entries, product descriptions, or single config snippets — often fits comfortably inside one embedding call and covers exactly one topic. If the document sits well within the model’s context window and doesn’t mix subjects, embed it whole and skip chunking. Splitting content that is already coherent only adds retrieval overhead. If the document is too long, or covers more than one topic, move to Question 2. Question 2: Does the Document Have a Clear Internal Structure? Some documents already contain natural boundaries that reflect how their content is organized. Markdown headings, HTML elements, PDF sections and tables, and code functions or classes can all provide meaningful chunk boundaries. If the document has a reliable structure that maps to logical content units, use document-based chunking. The document is parsed according to its format, keeping sections, paragraphs, tables, functions, or other structural elements together. If the structure is inconsistent, incomplete, or doesn’t provide useful boundaries, move to Question 3. Question 3: Do Chunks Need Context From Elsewhere in the Document? Some documents carry dependencies that span the whole text. Take, as an example, a legal contract that defines a term in section one and uses it in section eight, or a technical spec whose terminology only makes sense next to an earlier paragraph. If retrieval needs to preserve that document-wide relationship, use late chunking. Embed the full document with a long-context model first, then derive chunk-level embeddings from that result, so each chunk still carries a trace of the whole document. If chunks can stand on their own without losing meaning, move to Question 4. Question 4: Does Information Density Vary, or Do You Need Multiple Levels of Granularity? This question covers two different conditions, each pointing to a different strategy: If information density varies across sections, use adaptive chunking to dynamically adjust parameters such as chunk size and overlap to match the content. Dense, information-rich sections can use smaller, more granular chunks, while broader or less dense sections can use larger chunks. If the same corpus needs to support both broad questions (such as