In this article, you will learn how to build an end-to-end sentiment analysis pipeline using Scikit-LLM and open-source large language models served through the Groq API. Topics we will cover include: How Scikit-LLM bridges classical scikit-learn pipelines with modern large language model API calls. How to set up Scikit-LLM with a Groq backend and prepare the IMDB Movie Reviews dataset for inference. How to build, run, and evaluate a zero-shot sentiment classification pipeline using scikit-learn-compatible syntax. Building an End-to-End Sentiment Analysis Pipeline with Scikit-LLM Introduction Traditional machine learning pipelines for predictive tasks like text classification usually rely on extracting structured, numerical features from raw text — for instance, TF-IDF frequencies or token embeddings — to feed into classical models such as logistic regression, ensembles, or support vector machines. With the rise of large language models (LLMs), the rules of the game have somewhat changed: it is now possible to leverage zero-shot or few-shot reasoning on existing, pre-trained models for language tasks as part of a machine learning framework. Scikit-LLM is a Python library that addresses this: it bridges the gap between classical machine learning and modern LLM API calls. In this article, we will use Scikit-LLM alongside Groq backend models to build an end-to-end pipeline for sentiment analysis (a domain-specific form of text classification), achieving reasonably fast inference results with open-source models. From preprocessing to inference, we will use a large, realistically-sized dataset — the IMDB movie reviews dataset. Prerequisites, Setup, and Obtaining the Dataset To make the code shown in this tutorial work, you’ll need to have installed the Scikit-LLM library: Once installed, the first step is to set it up and configure API credentials. In other words, we will need to “connect” Scikit-LLM to an endpoint — namely an LLM API repository like Groq. Make sure you register on Groq and generate an API key here: you’ll need to copy and paste it in the code below: from skllm.config import SKLLMConfig # 1. Pointing to a Groq’s compatible endpoint SKLLMConfig.set_gpt_url(“https://api.groq.com/openai/v1”) # 2. Set your free Groq API key # Get yours at https://console.groq.com/keys SKLLMConfig.set_openai_key(“YOUR-API-KEY-GOES-HERE”) from skllm.config import SKLLMConfig   # 1. Pointing to a Groq’s compatible endpoint SKLLMConfig.set_gpt_url(“https://api.groq.com/openai/v1”)   # 2. Set your free Groq API key # Get yours at https://console.groq.com/keys SKLLMConfig.set_openai_key(“YOUR-API-KEY-GOES-HERE”) Scikit-LLM uses an endpoint function, set_gpt_url, that is compatible with OpenAI by default; we have routed it to make internal requests to a custom Groq URL: https://api.groq.com/openai/v1. The next stage of the process is importing the IMDB Movie Reviews dataset — which has about 50K instances — and preparing it for the sentiment analysis pipeline we will build. Instances consist of a text review labeled with a sentiment, which can be positive or negative (this is a binary classification problem, solvable with models like logistic regression, for instance). For convenience, we read the dataset from a publicly available GitHub repository version in CSV format: import pandas as pd from sklearn.model_selection import train_test_split # Fetching a large, realistic-sized dataset (IMDB Movie Reviews – 50,000 rows) # We will read the data from a public raw CSV for convenience url = “https://raw.githubusercontent.com/Ankit152/IMDB-sentiment-analysis/master/IMDB-Dataset.csv” print(“Downloading dataset…”) df = pd.read_csv(url) print(f”Total dataset size: {df.shape[0]} rows”) # In a realistic LLM pipeline using a free-tier API, sending 50,000 requests # will likely trigger quota limits. Thus, we will use 500 rows for demonstrating our pipeline execution. # Feel free to use more data if you have paid API access. df_sampled = df.sample(n=500, random_state=42) # The IMDB dataset contains HTML tags and formatting noise: that’s perfect for testing our cleaner X = df_sampled[“review”] y = df_sampled[“sentiment”] # Labels are ‘positive’ or ‘negative’ # Splitting into training (for initializing zero-shot labels) and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 import pandas as pd from sklearn.model_selection import train_test_split   # Fetching a large, realistic-sized dataset (IMDB Movie Reviews – 50,000 rows) # We will read the data from a public raw CSV for convenience url = “https://raw.githubusercontent.com/Ankit152/IMDB-sentiment-analysis/master/IMDB-Dataset.csv” print(“Downloading dataset…”) df = pd.read_csv(url)   print(f“Total dataset size: {df.shape[0]} rows”)   # In a realistic LLM pipeline using a free-tier API, sending 50,000 requests # will likely trigger quota limits. Thus, we will use 500 rows for demonstrating our pipeline execution. # Feel free to use more data if you have paid API access. df_sampled = df.sample(n=500, random_state=42)   # The IMDB dataset contains HTML tags and formatting noise: that’s perfect for testing our cleaner X = df_sampled[“review”] y = df_sampled[“sentiment”] # Labels are ‘positive’ or ‘negative’   # Splitting into training (for initializing zero-shot labels) and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) Note that we fetched 500 rows only for demonstration purposes, as otherwise inference may take long without sufficient computing resources. You can freely change this sample size, n=500, to adapt it to your own needs. Building the Sentiment Analysis Pipeline Here comes the most interesting part of the process! A data science pipeline boils down to a series of preprocessing, cleaning, and data preparation steps followed by model setup or training, inference, and evaluation. For a predictive, text-based scenario like ours, preprocessing typically entails cleaning and normalizing the text. Scikit-learn provides an elegant class, FunctionTransformer, to define and encapsulate preprocessing steps based on a custom function: from sklearn.preprocessing import FunctionTransformer def clean_text_data(texts): “””Cleans raw text inputs by removing HTML tags and stripping whitespace.””” series = pd.Series(texts).astype(str) # Remove HTML tags like <br /> cleaned = series.str.replace(r'<[^>]+>’, ‘ ‘, regex=True) # Remove extra spaces cleaned = cleaned.str.strip().str.replace(r’\s+’, ‘ ‘, regex=True) return cleaned.tolist() # Wrapping the cleaning function to enable its use inside a Pipeline object text_cleaner = FunctionTransformer(clean_text_data) from sklearn.preprocessing import FunctionTransformer   def clean_text_data(texts):     “”“Cleans raw text inputs by removing HTML tags and stripping whitespace.”“”     series = pd.Series(texts).astype(str)     # Remove HTML tags like <br />     cleaned = series.str.replace(r‘<[^>]+>’, ‘ ‘, regex=True)