I used to❤️Twitter as a place to genuinely connect & learn from others. It's not that place anymore. I uninstalled Twitter in April'22 & have decreasingly used it. Other platforms haven't worked as a replacement; hoping Bluesky will be diff. Join me here:

StreetReaderAI leverages context-aware AI to solve a critical accessibility barrier, making immersive street-level imagery interpretable by screen readers. This work is the path to fully inclusive digital exploration.

Delighted that our work is being featured. This was an incredible team effort by a set of hugely talented, passionate Googlers: @shaunkane, @AlexFiannaca, Victor Tsaran, Nimer Jaber, & Phil Nelson. An accessible street view has enormous potential for blind travel planning and O&M

Interested in learning more? See: 📄UIST'25 preprint: https://t.co/wgztdIBEgr 📽️UIST'25 video: https://t.co/N3B9xabqrV 🚀Project page:

On a personal level, this project was like a sabbatical dream 🦄, I got to reunite with @shaunkane (we first worked together in 2006), collaborate with some incredible Googlers like Alex Fiannaca, Nimer Jaber, & Victor Tsaran, and even write (lots of) code in Google's monorepo 🧑🏽‍💻

Research challenges ahead: 🧠 Mental models of pedestrian navigation vs. street-level streetscape imagery ⚠️ Bias towards trusting AI output (even when wrong) 🎧 Interaction design difficulties in creating concise audio feedback 🌐 Improving spatial reasoning & multimodal AI

User feedback from our lab study with 11 blind users: 🚀 "This is a huge leap forward in navigation" ✨ "Incredible!" 😊 "Going to make a lot of blind people very happy" A key finding: users overwhelmingly preferred conversing with an AI Chat Agent vs. other AI modes.

I have had the incredible privilege to sabbatical at Google Research. What have I been up to? Attempting to make Street View accessible to all! 🌍✨ StreetViewAI is a new, accessible street view prototype using context-aware AI & voice interaction. https://t.co/VwpK2YKJe7

We're launching RampNet, an open-source AI that helps detect curb ramps with near-human accuracy. The most amazing part? The entire project was conceived of and led by high school student John O'Meara. 🧵 A thread on what we built and why it matters.

📢We've completely redesigned our API pages! It's never been easier to: ✔️ Download data in CSV, GeoJSON, or Shapefiles. ✔️ Access data programmatically through our revised API. ✔️ Build tools that champion accessibility & improve urban mobility. https://t.co/b7Ge1JSiYE

Today, we kick off hashtag#ASSETS2024 in beautiful St. John's, Newfoundland. Would love to meet up with you if you're also here!

Can your robot vacuum do this? Researchers in @UW @uwengineering #UWAllen’s @makeabilitylab adapted one to create MobiPrint, a 3D printer on wheels that maps a room and prints objects on location, on demand based on a user's needs. #UWinnovates #NSFfunded

Thrilled to be heading to #ASSETS24 to present “Engaging with Children’s Artwork in Mixed Visual-Ability Settings”, done with my wonderful collaborators and advisors @wobbrockjo and @jonfroehlich! I will be presenting on Mon, Oct 28 as a part of Session 1A: Creativity 🖼️🎨 (1/7)

Chu Li is a fantastic, multi-talented PhD student seeking an internship for Summer'25. She will level-up any group that she joins. See:

One step towards this future is Chu Li's (@Chimichurrichu) new VIS'24 tool AltGeoViz, which attempts to provide interactive high-level spatial analytic summaries verbally via screenreader I/O controls. https://t.co/GWwOXcYtqg

Drawing on recent prior work in accessible visualization (e.g., from @arvindsatya1, @FrankElavsky, @domoritz, @athersharif, @ohnobackspace, & others), we are working on making geovisual analytics accessible to blind and low vision users.

These interactive spatial analytic tools are critical to informing urban planners, advocacy groups, and policy makers; however, because of their intrinsically visual nature, they are not accessible to screen reader users.

For the last decade+, my group has developed new interactive tools to advance understanding of the accessibility of the physical world for people with disabilities—projects like AccessScore, AccessVis, and Sidewalk Equity.

On this last day of #UIST2024, PhD student @XiaSu09 will be presenting our joint work w/Chang Xiao and Eunyee Koh at Adobe Research on SonifyAR, which uses LLMs to generate contextual sound effects in AR environments. Talk is 2-3:15PM in Allegheny 2. https://t.co/NuJXkWehlS

And @jaewook_jae will demo EARLL, an embodied and context-aware language learning application for AR glasses. Imagine touching an object & seeing its word in multiple languages.