Overview
Vision-language models are usually used as a black box: text in, similarity score out. I built four pipelines that open that box from different sides and run over the same inputs, because the informative failures tend to sit between the parts, not inside any one of them.
The four pipelines
- Image-text grounding: zero-shot matching of images to prompts with OpenCLIP ViT-B/32.
- Attention visualization: custom hooks extract multi-head attention from the Vision Transformer and overlay it as heatmaps, showing which image patches carried a match.
- Image generation: Stable Diffusion v1.5 through
diffusers, with configurable conditioning, probes the same semantics from the generative side. - Embedding analysis: cosine similarity matrices plus t-SNE and UMAP projections show how well image and text embeddings line up.
Why the combination matters
Grounding scores tell you that a match happened. Attention maps tell you where. The embedding analysis tells you whether the space is separable at all, or whether a good score was luck. Running all four over the same inputs is what makes a VLM result interpretable instead of just reported.
Scope
This is an analysis tool for VLM and VLA research, not a benchmark entry. Its outputs are heatmaps, similarity matrices and projections to be read and compared, so the page reports no single accuracy score.