Seeing HAI

Measuring how people stay, gather, and use sidewalks

Independent Research
Urban AI
Computer Vision
Pedestrian & Public Life
Lead Researcher

Liu Liu

Published

Aug, 2026

Seeing HAI measures how people use sidewalks: who is moving or staying, who is together, and what they are doing. My MIT dissertation uses street-level images and vision-language models to extend close observation of public life across streets and cities. It asks what pedestrian counts miss when equally busy sidewalks support different activities and forms of social presence. The measures describe what is visible at the moment an image was captured.

Same count, different public life

Each pair below has nearly the same number of people, but shows very different forms of sidewalk social life.

Pair 1, left: N = 9. People on their own, mainly passing through.
N = 9. People on their own, mainly passing through.
Pair 1, right: N = 9. Groups and people standing or seated.
N = 9. Groups and people standing or seated.
Pair 2, left: N = 9. People on their own, mainly passing through.
N = 9. People on their own, mainly passing through.
Pair 2, right: N = 10. A group of four, a pair, and four people alone.
N = 10. A group of four, a pair, and four people alone.
Pair 3, left: N = 9. People on their own, mainly passing through.
N = 9. People on their own, mainly passing through.
Pair 3, right: N = 10. Several pairs; most people are stationary.
N = 10. Several pairs; most people are stationary.

These examples are from Figure 8 of A Vision-Language Framework for Measuring Social Life on Sidewalks (M10; under review).

What people do on sidewalks

These activities appear in street-level images; many would be missed by a pedestrian count or an object detector alone.

Street vending; example from the original activity collage, with faces blurred.
Street vending
Talking beside strollers; example from the original activity collage, with faces blurred.
Talking beside strollers
Children's activities; example from the original activity collage, with faces blurred.
Children's activities
Dining and drinking together; example from the original activity collage, with faces blurred.
Dining and drinking together
Walking dogs; example from the original activity collage, with faces blurred.
Walking dogs
Talking in a group; example from the original activity collage, with faces blurred.
Talking in a group
Talking on the sidewalk; example from the original activity collage, with faces blurred.
Talking on the sidewalk
Sitting and watching; example from the original activity collage, with faces blurred.
Sitting and watching
Protest rally; example from the original activity collage, with faces blurred.
Protest rally
Fishing; example from the original activity collage, with faces blurred.
Fishing
Hugging; example from the original activity collage, with faces blurred.
Hugging
Interaction with the street-view vehicle; example from the original activity collage, with faces blurred.
Interaction with the street-view vehicle
Political expression; example from the original activity collage, with faces blurred.
Political expression
Fixing a tire; example from the original activity collage, with faces blurred.
Fixing a tire
Seated with belongings; example from the original activity collage, with faces blurred.
Seated with belongings

Faces are blurred in the source images, and no individual is identified.

These examples define the measurement problem: finding people is only the first step. To study sidewalk life, we also need to describe how people are together, what they are doing, and where and when they were observed.

How observations become measures

The method connects human observation with image-based measurement. Human judgments about whether pairs of people are together provide training examples for MINGLE, published at AAAI 2026. My M10 framework extends that approach by combining social-group detection with per-person coding, then turning their outputs into interpretable measures. The paper, coauthored with Andres Sevtsuk, is under review at Computers, Environment and Urban Systems.

Five-stage workflow showing street panoramas reprojected into sideviews, parallel group and per-person attribute detection, validity filtering and cross-branch refinement, indicator construction, and New York City maps.

The complete five-stage framework: timestamped imagery, parallel group and activity detection, cross-branch refinement and fusion, rule-based indicators, and city-scale analysis. The provisional sex fields shown in the diagram are excluded from primary analyses. Click to enlarge.

Recognize social groups

A social group is a relationship among people, so proximity alone is insufficient. Occlusion, depth differences, and the lack of motion in a still image make it difficult to distinguish people together from people who happen to be close. The framework detects people with SAM3, estimates depth with PatchFusion, and uses a fine-tuned Qwen2.5-VL-3B model to judge pairs before assembling them into groups. The preprint reports F1 = 80.64% for this pairwise model, using an expanded 76K training corpus.

Describe visible states

Alongside grouping, Qwen3-VL-30B-A3B-Instruct records ten fields per person: validity, posture, locomotion, object, object relation, support surface, zone type, uniform presence, apparent age group, and model-perceived sex. Separating these fields lets a person be recorded as walking, holding a phone, and occupying a particular part of the sidewalk, rather than forcing the observation into a single activity label. The preprint reports 88.5% mean field accuracy on 999 held-out samples.

The vocabulary draws on Whyte and Mehta’s observational research. It distinguishes visible states from interpretations of intent or social meaning. Apparent age and model-perceived sex are exploratory estimates; sex is excluded from primary analyses because provider face blurring makes it unreliable.

A grid of street-image examples organized by eight attribute fields, illustrating seated and moving people, objects such as strollers and bicycles, and sidewalk contexts.

Examples from the attribute vocabulary: posture, locomotion, objects and object relations, support surfaces, spatial zones, uniforms, and apparent age. Labels describe model readings of visible evidence; apparent age is an exploratory estimate.

Connect groups and activities

The two branches inform one another. A validity filter removes detections in advertisements, inside vehicles, or outside the target pedestrian space before groups are rebuilt. Group-level consistency helps refine locomotion labels; group IDs then join to the per-person fields. Rules applied to these records produce activity, spatial-context, and accessibility-related indicators, with location and capture time retained for every observation.

Build interpretable measures

Grouping and dwelling summarize two aspects of the scenes shown above. Each person receives a grouping weight \(\alpha\): 1 when alone, 2 in a pair, and 3 in a group of three or more. A dwelling weight \(\beta\) is 3 for a stationary, seated person who is not riding; 2 for a stationary, non-seated person; and 1 otherwise.

Index Definition What it summarizes
Weighted Grouping Index (WGI) \(\frac{1}{N}\sum_{i=1}^{N}\alpha_{g(i)}\) Grouping composition
Weighted Dwelling Index (WDI) \(\frac{1}{N}\sum_{i=1}^{N}\beta_{s(i)}\) Observable dwelling posture
Social Dwelling Index (SDI) \(\frac{1}{N}\sum_{i=1}^{N}\alpha_{g(i)}\beta_{s(i)}\) Grouping and dwelling combined for each person

SDI averages the per-person products of the two weights, which is why scenes with similar headcounts can differ. These ordinal measures describe visible configurations, rather than the duration of a visit, someone’s reasons for being there, or the quality of a social experience. The method paper reports sensitivity checks for alternative weights.

Connect observations to streets

These measures become useful for urban research when the visual evidence can be connected to a location and an observation time. The spatial workflow retains that connection from the source panorama to a particular side of a street or an intersection.

Keep capture time explicit

Imagery with capture timestamps is organized into four observation windows: weekday and weekend, each at 12:00–14:00 and 17:00–19:00. This keeps the observed activity tied to when it occurred. The broader collection includes Apple Look Around, Bing Streetside, and Mapillary; the national analysis below uses the Apple and Bing subset.

From panorama to sidewalk-facing views

Each panorama is reprojected into left and right sidewalk-facing views, with a 150° field of view and a 1600 × 800 pixel frame. The original presentation figure links the two photographs to the capture point, road direction, and opposite viewing directions on the street network, preserving which side of the street each image observes.

The original panorama, left and right sidewalk photographs, and a street map with red arrows connecting each photograph to its viewing direction on opposite sides of the road.
From street view to side view: the red arrows connect the two photographs to their sidewalk-facing directions on the road network. Click to view the full presentation figure.

Keep intersections distinct

At intersections, a wide view retains corners and crossings together. These records remain separate from mid-block views. For spatial analysis, sidewalk-facing observations are assigned to side-specific block frontages on the TIGER/Line road network, while intersection records retain their junction association. View and frontage identifiers, capture time, and a road-segment crosswalk support linkage to land use, transportation, and demographic data.

A panoramic intersection photograph above a map of the surrounding street network with blue intersection-location markers.
An intersection panorama and the corresponding collection of intersection locations on the street network. Click to view the full presentation figure.

What counts miss in New York City

Applying the framework to 102,514 New York City sideviews makes the distinction in the opening examples visible across the city: pedestrian count and SDI have a weak image-level correlation (r = 0.168). Mapping their standardized difference shows where relatively high counts coincide with less grouping and dwelling, and where the reverse occurs.

New York City map comparing standardized pedestrian count with the Social Dwelling Index; blue indicates relatively higher counts and red relatively higher SDI.

Standardized differences between pedestrian count and SDI in New York City. Blue areas have relatively higher counts than SDI; red areas have relatively higher SDI than counts.

This motivates the dissertation’s ongoing Findings study: which street environments are associated with grouping and dwelling, alongside foot traffic? Preliminary block-frontage models examine sociodemographic context, street design, and land use. Factors associated with higher counts are not consistently associated with more grouping or dwelling. This work remains in progress; the associations are descriptive and sensitive to imagery coverage, model uncertainty, and spatial dependence.

From one city to 496

The next step asks whether these forms of observed sidewalk life can be studied across cities. Stay Together: Street-Level Social Indicators for Sidewalk Life Across 496 U.S. Cities, coauthored with Andres Sevtsuk, develops that national research resource. The manuscript is in preparation.

The assembled panel contains 7,139,672 timestamped views from Apple Look Around and Bing Streetside across 496 U.S. cities, captured during 2011–2025. Its 5,692,332 sidewalk-facing views are organized into 3,725,328 block frontages. A separate file retains 1,447,340 intersection views as individual observations. Shared identifiers and a crosswalk to TIGER/Line road segments allow researchers to link observations to other spatial data.

The original presentation coverage maps: United States at left, followed by New York City, Chicago, San Francisco, and Boston, with imagery locations shown in yellow.
From one city to a national corpus: imagery coverage at national and city scales. Click to view the full presentation figure.

The presentation maps document collection coverage, while the counts above describe the data paper’s analytical subset. Mapillary imagery is excluded from that analysis and the planned release of indices: its uncontrolled capture geometry prevented almost all of those images from being assigned to block frontages.

Compare like with like

The national analysis examines grouping (WGI) and visible posture and locomotion (WDI) separately. SDI is included in the planned release but not used in this paper’s empirical analysis. Comparisons must also account for crowd size: a view containing one person necessarily has WGI = 1, so pooling different person counts can reverse an apparent association with neighborhood density.

Holding the detected count at exactly two people, the share classified as one group falls from 38.0% in the lowest tract-density quintile to 26.8% in the highest. This descriptive pattern demonstrates the importance of the comparison being made; it does not establish that density causes less socializing.

Two-panel analysis showing mean WGI across density quintiles at fixed person counts, alongside negative correlations of density with WGI and WDI within each person-count and time-window combination.

Grouping and tract density, conditioning on the number of detected people and capture window. The pooled relationship differs from the within-count patterns.

Build a research resource with explicit limits

The national maps summarize grouping and dwelling where the imagery supports measurement. People were detected on 430,984 frontages (11.6%); grouping and dwelling are undefined on the rest. The median frontage has one view. Coverage is opportunistic, and the national sample has not received an independent annotation audit, so these records support research on observed configurations rather than validated rankings of places.

National maps of grouping and dwelling across measurable 2-kilometer grid cells, with detailed grouping maps for twelve selected cities.

National patterns in grouping and dwelling across 6,066 measurable 2-km grid cells in 472 cities, with grouping shown within twelve selected cities. The full dataset covers 496 cities.

The panel is assembled and in use. Derived tables, schemas, and documentation are being prepared for public release; source imagery will not be redistributed.

Inspect the evidence

The final connection is back to the images. The Great Streets prototype lets a user move from segment-level indicators in a 3D map to the source views and person-level labels. Comparing observations across sources and times makes it possible to trace a mapped value back to the visible evidence from which it was constructed.

Screen recording of the prototype using New York City data. The interface links segment-level indicators, 3D context, source imagery, and visual labels.

More project visualizations.