Provocation

“Indeed, it [the cell phone] has taken over much of what used to be defined as the responsibility of shelter in terms of sense of security, space, orientation, and representation.” [1] Both at home and in public, we are often absorbed by our devices, dissociated from physical reality and algorithmically siloed off to our assigned echo chambers. How can we feel safer and more comfortable in the world, so as to not require constant shielding and distraction from it? How can we consume digital content together and avoid polarization? What role can the built environment play in the pursuit of these goals?

The Project

The technology to display information at a variety of sites is readily available: projectors, TVs, monitors, speakers… but pairing them to a person’s unique digital world can be a cumbersome process. The challenge here is securely providing these displays and interfaces access to a person’s information in a fully consensual way.



I propose the development of a system for such device pairing and information sharing. It will be employed by a mobile application that allows users to create a profile, as they would at a social media platform, and selectively expose data to authorized peripherals. These peripherals will broadcast BLE beacon signals, alerting the application of their proximity and their capabilities, at which point the application can broker data exchange over cloud infrastructure. The peripherals are now able to activate context-aware experiences based on the people present.



On the other hand, these technologies are routinely exploited to surveil consumers. Most people are unaware of this phenomenon, due to a lack of transparency in privacy policies and the prevalence of advertising software that finds its way into popular app SDKs. Much like cookies track people through the web, BLE beacons can be used to uniquely and precisely track people's location indoors (where GPS usually fails). My proposal is to develop a process for identifying such beacons and which apps they are communicating with. Then I will conduct a survey of retail locations throughout Manhattan to build a dataset of spatial data collection practices.



The project will be presented as an interactive experience, where people will have previously downloaded the app and created a profile. The space will curate and display media experiences based on those present and the data they have shared. Data visualizations about my research in unethical use of this technology will be displayed throughout the room in juxtaposition. In this way, people will experience a frictionless, context-aware and embodied social media interaction, while learning about the dangers of these technologies and how to guard themselves against its misuse.

Computational Methods

  1. Spatial UX / Installation

    The entire space will be designed to afford collaborative interactions, rather than individual consumption of media on a phone. A combination of screens with classic UIs, as well as sensors and projectors will come together to enable gestural interaction with the system.

  2. Algorithmic governance

    Studying and crafting the interaction between algorithms and social dynamics will be a crucial method for creating the experience. In order for the right content to be surfaced as users approach Resonators, curation will be required. On resonators where interaction is possible, the rules of the system and its feedback mechanisms will define the resulting experience.

  3. Embedded Systems and Sensors

    In order to make the experience possible, I will make use of sensors and radio signals for devices to locate and communicate with each other. I will be using microcontrollers such as ESP32 or Raspberry Pi to give life to the Resonators.

Design Methods

I am inspired by research on spatial, tangible, and collaborative interfaces, and wish to push this inquiry further by applying ideas and practices developed for communal living to a newly imagined communal digital space. In communal living, people have varying preferences of where their boundary of privacy lies, such as being comfortable sharing a kitchen, but not a bathroom. This work is speculative and participatory in nature, as it aims to expand the boundary of digital privacy people are comfortable with through a co-lived experience. The effects of such an experience might be restorative and lead to strengthened social connection [2] and pave the way for people to more deeply engage with each other in public.

Proof of Concept

Audience

The project is aimed for the General public, but people will differ in their willingness to engage with others in public space. People have varying levels of comfort sharing media online as well. Designing the UX of the application, as well as the experiences themselves, will be a crucial point to iterate and perfect so that no personality types or groups are left out of the experience. Because the system is novel and has few precedents, it will be important to adapt as the system is tested, since predicting all possible outcomes would be impossible.



Data

  1. Ethnographies

    I will organize volunteer sessions to study how people interact in public space. In particular, aiming to answer the questions: How do people use their phones while in a public space? How much do they interfere with otherwise IRL socializing?

  2. Feedback from user sessions

    I will create prototypes as early as possible to start getting people to interact with them and provide feedback. This will be instrumental to the success of the project.

  3. BLE beacon usage in retail stores

    For my critical work, I will have to develop a data collection mechanism and conduct surveys throughout the city’s more prominent retail areas.

Materials/Sensors

  1. Bluetooth beacon
  2. Mobile phone (iPhone): bluetooth, wifi
  3. Camera
  4. Light
  5. Water
  6. Projection

References

  1. Colomina, Beatriz, and Mark Wigley. Are We Human? Notes on an Archaeology of Design. Zürich, Switzerland: Lars Müller Publishers, 2016.

    Provided inspiration to reimage the world breaking free from our phone as our shelter.

  2. Cheong, Jin Hyun, Zainab Molani, Sushmita Sadhukha, and Luke J. Chang. “Synchronized Affect in Shared Experiences Strengthens Social Connection.” Communications Biology 6, no. 1 (October 28, 2023): 1099. https://doi.org/10.1038/s42003-023-05461-2

    Provided some proof that communal consumption of media leads to better human relationships and bonding.

  3. Getting Started with iBeacon. (2014). Apple

    A reference document which allowed me to understand Apple’s iBeacon technology and how to work with it.

  4. Kwet, M. (2019, June 14). Opinion | In Stores, Secret Bluetooth Surveillance Tracks Your Every Move. The New York Times. https://www.nytimes.com/interactive/2019/06/14/opinion/bluetooth-wireless-tracking-privacy.html

    Showed me that people are looking into the use of this technology and its adverse effects, and what has been done about it up to this point.

  5. Kearns, M., & Roth, A. (2020). The ethical algorithm: The science of socially aware algorithm design. Oxford University Press.

    Thoughts on how to craft ethical algorithms and consider how they affect different people.

  6. Schimanski, C.-E. (2018, March 6). Location Awareness in IoT with Beacon Technology. Medium. https://medium.com/@cayeric/location-awareness-in-iot-with-beacon-technology-2803f48a4e8c

    Useful technical and implementation information.

  7. Pouw, W. T. J. L., Van Gog, T., & Paas, F. (2014). An Embedded and Embodied Cognition Review of Instructional Manipulatives. Educational Psychology Review, 26(1), 51–72. https://doi.org/10.1007/s10648-014-9255-5

    Review of what HCI work has been done, in particular with focus on cognition benefits of tangible interfaces.




Download Project Poster
View Semester's Progress Slides