Method developed within my independent dissertation research, Seeing HAI
Lead author: Liu Liu · Collaborative paper with researchers at MIT and the Hasso Plattner Institute
MINGLE is a dissertation method and a coauthored research output. It is also used within the separate Sidewalk Ballet lab collaboration.
People can usually recognize who is interacting with whom in a street scene, but this remains difficult for computer vision. A social group is not a physical object with clear edges. It is a relational pattern among two or more people who may share attention, conversation, or movement. Standard object detectors identify bounded entities, but the target here is defined by social context.
MINGLE (Multi-person Interaction-aware Neural Group Localization and Extraction) combines person detection, depth-aware vision-language reasoning, and spatial aggregation. The paper was published in the AAAI 2026 AI for Social Impact Track and introduces annotations and metadata for 100,000 urban street images.
The Problem
William H. Whyte, Jan Gehl, and Allan Jacobs established the importance of observing where and how people gather in public space. Their methods required trained fieldworkers at selected locations, producing detailed observations that were difficult to repeat across a city.
Street-view imagery makes it possible to revisit this observational tradition at a larger scale. Detecting social interactions in these images can support comparisons between human co-presence, accessibility, street design, and land use.
The core difficulty is semantic complexity. A social group has no fixed shape, color, or texture. It is defined by relationships such as proximity, body orientation, shared gaze, and co-movement.
Off-the-shelf detectors return bounding boxes around individuals, not coalitions. Zero-shot prompting of large vision-language models (VLMs) like GPT-4o yields F1 scores near zero on this task. A new approach is needed.
Figure 1. Comparison of three approaches on the same New York street scene. MINGLE (top) correctly identifies and localizes socially interacting groups with tight bounding boxes. Traditional object detection (middle) has no category for “social group.” Zero-shot VLM prompting (bottom) produces noisy, unreliable results.
When queried with “groups of people interacting,” the tested open-vocabulary detector returns one bounding box around one person rather than the relationship among people.
Figure 2. Open vocabulary detection given the query “groups of people interacting” (right) detects only one individual, failing to capture any social group structure.
The MINGLE Pipeline
MINGLE solves the problem in three stages, each addressing a distinct challenge:
Stage 1: Person detection
An ATSS-Swin-L-DyHead detector first identifies all individual people in the scene with high-confidence bounding boxes. This separates the problem of finding people from the problem of reasoning about their relationships.
Stage 2: Pairwise affiliation classification
For every pair of detected individuals, MINGLE constructs a prompt containing:
- RGB crops of each person’s bounding box
- A depth-map crop of the scene, generated by a monocular depth estimator
- Numeric depth values and absolute depth differences inserted directly into the text prompt
This multimodal prompt is fed to a fine-tuned Qwen2-VL / Qwen2.5-VL model, which classifies each pair as interacting, not interacting, or uncertain. The use of explicit depth cues is a key innovation: distance in the image plane is an unreliable proxy for physical proximity (due to perspective), so depth information lets the model reason about actual spatial separation between individuals.
Computational cost is managed by filtering out pairs whose image-plane distance or depth difference exceeds configurable thresholds, dramatically reducing the number of VLM calls without meaningful loss in accuracy.
Stage 3: Group aggregation
A greedy clustering algorithm merges pairwise classifications into group regions. A set of two or more individuals with mutual “interacting” links forms a social group, and a bounding box is computed from the extreme coordinates of its members. The resulting social group region represents collective structure rather than a single body.
Figure 3. The full MINGLE pipeline (top row) and its output format (bottom row). Stage 1 detects individuals; Stage 2 classifies each person pair using depth-aware VLM prompts; Stage 3 aggregates pairs into group bounding boxes. The bottom row shows how group detections and a Social Activity Index are computed for a given street image.
Dataset
A central contribution of MINGLE is its 100K-image annotated dataset, the first large-scale benchmark for social group detection in urban street-view imagery. Images were sourced from four providers to ensure geographic and visual diversity:
| Source | Share |
|---|---|
| Apple Look Around | 60% |
| Google Street View | 15% |
| Bing Streetside | 15% |
| Mapillary | 10% |
The dataset contains 588,430 human instances and 72,537 annotated social group regions, combining manual human labels with pipeline-validated pseudo-labels. A separate pairwise fine-tuning set of 79,265 annotated person pairs was collected to train the VLM classifier (76K training / 2K test), with human labels of “Yes,” “No,” or “Not Sure” for each pair.
Figure 4. Dataset composition and statistics. Left: source distribution across four street-view providers. Right: histograms of people per image, groups per image, and group size.
Results
MINGLE substantially outperforms all baselines:
Pairwise classification (Table 2 in the paper):
| Model | F1 | Precision | Recall |
|---|---|---|---|
| GPT-4o (zero-shot) | 0.00 | — | — |
| Qwen2-VL (zero-shot) | 0.53 | — | — |
| MINGLE (fine-tuned, 76K) | 0.70 | 0.73 | 0.66 |
Full pipeline: social group region detection (Table 3):
| Method | mIoU | F1 | Precision | Recall |
|---|---|---|---|---|
| Baseline VLMs | ~0.00–0.02 | — | — | — |
| MINGLE | 0.64 | 0.60 | 0.75 | 0.61 |
The tested baseline VLMs produce near-zero region-detection scores. In these experiments, combining specialized person detection with depth-grounded pairwise reasoning substantially improves performance.
The depth and distance filtering ablation (Table 4) shows that aggressive filtering reduces VLM calls substantially with minimal impact on accuracy, making the pipeline practical for city-scale deployment.
Publication
Liu, L., Kudaeva, A., Cipriano, M., Al Ghannam, F., Tan, F., de Melo, G., & Sevtsuk, A. (2026). MINGLE: VLMs for Semantically Complex Region Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 40(45). https://doi.org/10.1609/aaai.v40i45.41239
arXiv preprint: arxiv.org/abs/2509.13484