VMVLM: Vision-Modulated Vision-Language Models for Improved Instruction Following

VMVLM (Vision-Modulated Vision-Language Models) is a novel architecture that enhances multimodal instruction-following capabilities through dual-stream visual feature integration. By introducing complementary visual pathways that combine traditional query-based visual encoding with direct intermediate visual feature injection, VMVLM achieves superior performance in vision-language understanding tasks. This approach leverages a frozen Vision Transformer (ViT) encoder, a Q-Former for learned visual queries, and a direct vision projection module that feeds intermediate visual representations directly into the Large Language Model (LLM). Experimental results demonstrate significant improvements in visual question answering, image captioning, and multimodal instruction following across multiple benchmarks.

2.1 Overall Design

VMVLM consists of three main components:

1. Vision Encoder: A frozen EVA-CLIP Vision Transformer that extracts hierarchical visual features
2. Q-Former Module: A BERT-based transformer that learns visual queries for cross-modal alignment
3. Vision Modulation Layer: A novel projection module that directly injects intermediate visual features into the LLM

2.2 Model Flow

  graph TD
    A[Input Image] --> B[Vision Encoder
EVA-CLIP ViT]
    B --> C[Final Layer Features]
    B --> D["Intermediate Layer Features"]
    C --> E[Q-Former
Learned Query Tokens]
    D --> F[Vision Projection]
    G[Text Input] --> E
    E --> H[Q-Former Output
Input for LLM]
    F --> I[Direct Visual Features
Visual Features for LLM]
    H --> J["Large Language Model
Vicuna-7B / Flan-T5-XXL"]
    I --> J
    K[Text Embeddings] --> J
    J --> L[Generated Response]
    

2.3 Vision Modulation Mechanism

The core innovation of VMVLM lies in its dual-pathway visual processing:

• Utilizes learnable query tokens (32 tokens) processed through Q-Former
• Provides compressed, task-relevant visual information
• Maintains compatibility with existing BLIP-2 architectures

• Extracts intermediate features from the second-to-last ViT layer
• Projects features directly into LLM embedding space
• Preserves fine-grained visual details often lost in query-based approaches

4.1 Loss Function Design

The model employs carefully designed loss masking to ensure optimal training dynamics and prevent interference between different input modalities. This sophisticated masking strategy implements selective gradient computation that distinguishes between instructional content and target responses, while properly handling the dual visual pathways. The loss masking mechanism operates at the token level, applying different treatment strategies based on token type and origin within the input sequence. This approach ensures that the model learns to generate appropriate responses without being penalized for the visual or instructional components, leading to more stable training and improved instruction-following capabilities.

• No loss applied to visual tokens (both Q-Former and direct features)
• No loss applied to instruction text (input)
• Loss applied only to target response tokens
• Padding tokens excluded from loss calculation

4.2 Multi-stage Training

              
# Key components trained:
# - Q-Former parameters
# - LLM projection layers  
# - Vision projection module

# Frozen: ViT encoder, LLM backbone
              
            

              
# Datasets: VQA, Image Captioning, Instruction Following
# Full model fine-tuning with task-specific prompts
# Supports both generative and discriminative tasks
              
            

4.3 Inference Pipeline

  sequenceDiagram
    participant U as User
    participant M as VMVLM
    participant V as Vision Encoder
    participant Q as Q-Former
    participant P as Vision Projection
    participant L as LLM
    U ->> M: Image + Question
    M ->> V: Extract visual features
    V ->> Q: Final layer features
    V ->> P: Intermediate layer features
    Q ->> L: Compressed visual queries
    P ->> L: Direct visual features
    M ->> L: Text embeddings
    L ->> M: Generated response
    M ->> U: Final answer
    

6.1 Parallel Evolution

Unlike other single-pathway approaches, VMVLM maintains two complementary streams of visual information, enabling both high-level semantic understanding and fine-grained detail preservation.

6.2 Intermediate Feature Utilization

By using intermediate ViT layers rather than only final representations, the model captures multi-scale visual features that prove crucial for complex reasoning tasks.

6.3 Flexible Architecture Support

The vision modulation approach is architecture-agnostic and can be applied to both autoregressive (Vicuna) and encoder-decoder (FlanT5) language models.