Qwen3 Multimodal Embeddings: Vision-Language RAG Explained with Code

Sam Witteveen

Introduction to Qwen3 Multimodal Embeddings [0:00]

• Qwen3-VL-Embedding and Qwen3-VL-Reranker models released by Qwen. [0:04]
  • These models are multimodal, capable of processing text, images, and videos. [0:12]
  • They are engineered for multimodal information retrieval and cross-modal understanding. [0:15]
• Video's purpose: Explain multimodal embeddings, detail Qwen3-VL models and re-rankers, and provide a practical code example. [0:20]

Embeddings Refresher [0:47]

• What are Embeddings? [0:55]
  • Numerical representation of meaning. [0:55]
  • Input content (text, audio, etc.) is processed by an embedding model (neural network) to generate a numerical output vector representing the content. [0:59]
• Applications of Embeddings [1:15]
  • Semantic search.
  • Recommendation systems.
  • Clustering and anomaly detection.
  • Comparing numerical vectors is easier than comparing raw content (e.g., two paragraphs of text). [1:24]
• Traditional vs. Multimodal Embeddings [1:38]
  • Early Text Embeddings: NLP models convert text into text vectors (e.g., "The cat sat on the mat"). [1:44]
  • Creative Image Embeddings: CNN models convert images into image vectors (e.g., "Photo of a cat"). [1:52]
  • Separate Vector Spaces: Traditionally, text and image vectors existed in "different universes" or separate vector spaces, making direct comparison difficult. [1:59]
• The "Revolutionary Thing" with Multimodal Embeddings [2:07]
  • A single multimodal embedding model can process various modalities (text, image, video) and generate vector representations in a shared semantic space. [2:13]
  • This means text about a cat, a photo of a cat, or a video of a cat will all have semantically similar vectors, allowing for unified understanding. [2:28]
  • Multimodal Joint Embedding Space: Text and image inputs are processed by respective encoders (e.g., Transformer for text, CNN/Vision Transformer for image) and mapped into a shared, high-dimensional vector space. [2:46]
  • Items with high semantic similarity (e.g., "a picture of a cat" text and a "cat photo" image) are close in this space, forming clusters (e.g., "Dog Cluster," "Car Cluster," "Landscape Cluster"). [2:57]
• Benefits for RAG Systems [3:13]
  • Traditional RAG systems often discard images, charts, diagrams, and screenshots from PDFs, processing only the extracted text. [3:17]
  • Multimodal embeddings allow RAG systems to create embeddings for text, charts, diagrams, and screenshots, all within the same shared vector space. [3:25]
  • This enables similarity matching across diverse content types. [3:41]
• Multimodal RAG Framework [3:44]
  • Text queries and image queries are processed by a powerful multimodal model (e.g., CLIP, SigLIP) to create a shared embedding space. [3:45]
  • These embeddings are stored in a vector database containing multimodal data. [3:46]
  • Queries retrieve relevant context from the database based on semantic similarity. [3:47]
  • Contrastive Pre-training (CLIP Example): Images and text are encoded separately. The model learns to bring similar image-text pairs closer in the embedding space and push dissimilar pairs apart. [3:50]
  • Models like SigLIP take this concept further. [4:17]

Qwen3-VL-Embeddings and Rerankers Overview [4:40]

• Embedding Model vs. Reranking Model [4:49]
  • Embedding Model: Excellent at "recall" – finding a broad set of items similar to the query. [4:50]
    • These are often trained as bi-encoder models, where two inputs (e.g., query and document) are independently encoded, and their similarity is compared (e.g., using cosine similarity). [5:00]
  • Reranking Model: Focuses on "precision" – fine-grained scoring to determine the best match among the recalled candidates. [5:25]
    • The reranker takes the query and a set of candidate documents (from the embedding model) and re-evaluates their relevance, often with a more complex interaction between query and document. [5:39]
• Why use both? [5:56]
  • The embedding model is fast for initial retrieval but might only achieve ~85% precision. [6:00]
  • Running the reranker on the entire corpus is too slow. [6:07]
  • Combining them: The embedding model quickly narrows down a large corpus to relevant candidates, and then the reranker provides high-precision scoring on this smaller subset. [6:12]
• Key Features of Qwen3-VL models [5:57]
  • Multimodal Versatility: Seamlessly handles various inputs, including text, images, screenshots, and video within a unified framework. Supports tasks like image-text retrieval, visual question answering (VQA), and multimodal content clustering. [5:57]
  • Unified Representation Learning (Embedding): Leverages Qwen3-VL architecture to generate semantically rich vectors for visual and textual information in a shared space, enabling efficient similarity computation and retrieval across different modalities. [6:01]
  • High-Precision Reranking (Reranker): The Qwen3-VL-Reranker complements the embedding model. It takes a (query, document) pair (which can contain mixed modalities) and outputs a precise relevance score. The embedding model performs efficient initial recall, while the reranker refines results in a subsequent re-ranking stage, boosting retrieval accuracy. [6:06]
  • Exceptional Practicality: Inherits Qwen3-VL's multilingual capabilities, supporting over 30 languages, making it ideal for global applications. Offers flexible vector dimensions and customizable instructions. [7:48]
  • Sequence Length: Both models have a 32K context window, allowing for large document processing. [8:04]
  • Matryoshka Representation Learning (MRL): Supported by the embedding models. This allows using lower dimensions (e.g., 64, 128, 256, 512, 1024, 2048) of the embeddings for faster search operations, without needing to take the full embedding dimension for all searches. [8:17]

MMEB Leaderboard Performance [8:55]

• Qwen3-VL-Embedding-8B model is ranked #1 on the Massive Multimodal Embedding Benchmark (MMEB) leaderboard. [9:03]
• Qwen3-VL-Embedding-2B model is ranked #5, showing strong performance despite its smaller size, even outperforming some 7B models. [9:10]

Use Cases for Multimodal RAG [9:47]

• Visual Document Search: Embedding entire documents (including charts, diagrams, screenshots) as images allows searching for visual information that traditional OCR might miss. [9:32]
• E-commerce Product Search: Uploading an image of a product (e.g., a red dress) and then applying text filters (e.g., "green dress") to find similar items in different colors. [9:47]
• Video Surveillance/Content Search: Indexing video frames as embeddings enables querying long surveillance footage with text (e.g., "two people at ATM") or images to quickly locate specific events or objects. [10:17]

Colab Demo: Multimodal RAG Implementation [10:48]

• Setup & Installation [10:50]
  • Environment setup on a free Colab GPU (Tesla T4) for demonstration. [10:52]
  • Required packages installed (huggingface_hub, pillow, requests, transformers, qwen-vl-utils, torch). [11:10]
• Qwen3-VL Embedder Implementation [11:15]
  • Demonstrates loading the Qwen3-VL-Embedding-2B model and performing basic text encoding. [11:20]
  • The embedding dimension is 2048. [11:41]
• Image-Text Retrieval Demo [11:57]
  • Loads a collection of demo images (beach dog, cat laptop, mountain, city night, food pizza). [11:58]
  • Generates embeddings for these images. [12:11]
  • Defines text queries (e.g., "A woman playing with her dog on a beach at sunset", "A fluffy cat relaxing"). [12:28]
  • Computes similarity scores between text queries and image embeddings. [12:30]
  • Displays ranked image matches for each query, showing successful cross-modal retrieval. [12:31]
  • Visualizes the Query-Image Similarity Matrix, highlighting strong semantic alignment between queries and relevant images. [13:24]
• Building and Querying a Mini Multimodal RAG System [13:49]
  • Implements a MultimodalRAG class that supports adding text documents, images with optional captions, and mixed text+image documents. [13:50]
  • Initializes the RAG system and adds several diverse documents (e.g., machine learning text, golden retriever text, pizza text, Mount Everest text) and images with captions. [14:09]
  • Builds the embedding index for all documents. [14:27]
  • Performs text queries (e.g., "Tell me about dogs and pets", "Italian cuisine and traditional recipes", "Tall mountains and hiking destinations"). [14:32]
  • Retrieves a mix of text, and image+text results based on the query. [14:50]
• Advanced Features: Image-to-Image Search [15:31]
  • Demonstrates querying the RAG system using an image as input (e.g., a golden retriever photo) to find similar documents (text or images). [15:31]
  • Successfully retrieves the text description of a golden retriever and the image of a woman with a dog on the beach. [15:56]
• Advanced Features: Embedding Dimension Reduction (MRL) [16:07]
  • Shows how the reduce_embedding_dim function can be used to take subsets of the full embedding dimensions (e.g., 64, 128, 256, etc.). [16:07]
  • Compares similarity scores across different reduced dimensions, indicating that reduced dimensions can still yield comparable results while potentially speeding up search operations. [16:26]
  • Notes that larger dimensions (e.g., 2048d) are not always significantly better than smaller, quantized dimensions (e.g., 512d) in terms of semantic similarity. [17:06]
• Quantized Models (GGGUF) [18:28]
  • Mentions that quantized versions (e.g., GGGUF format) of Qwen3-VL models are available, allowing them to be run locally on personal computers, suitable for building local RAG systems. [18:28]