Materialfolio
Junling Zhuang
Project Description
Kevin Lynch's 1960 book "The Image of the City" is considered a classic of urban image theory, but the scientific rigor of the theory of urban form is controversial. Marshall (2012) argues that Lynch's theory has not been significantly tested, and much subsequent research has simply taken Lynch's claims for granted. However, the emergence of computer vision-related deep learning algorithms is changing this situation. The Perceptions concept model proposed by Reid Ewing and Susan Handy (2009) established a connection between objective environmental elements and subjective urban experiences, laying the foundation for a machine-readable perceptual model.
Based on algorithms and using street view as a dataset, algorithm-based research on urban perception challenges the objectivity of priori-based urban image theory and has gradually become the mainstream of research. However, the emergence of digital technology poses both a threat and an opportunity to urban image research. Machines have become "agents" in the field of urban perception research, and Google Street View has become the darling of urban perception research.
Unfortunately, the uneven distribution of Google Street View, insufficient data for disadvantaged communities, lack of real-time data, and commercial interests have led to data set unfairness. This can result in potential uncertainty and unfairness in research based on these data sets in the long run. To address these issues, this project aims to establish a user-friendly and user-incentivized program for collecting data and visualizing it, supplementing the shortcomings of the Google Street View dataset by crowdsourcing street views. Additionally, by taking important architectural materials in urban perception as a clue, the project will provide real-time updates for more image modalities in urban image research.
Urban Peception Development
Kevin Lynch: Urban Image Theory
The Perceptions concept model (2009)
Machine as Perception Agent
Online social networking sites are increasingly location-based. Social media are increasingly location-based. Social media, led by MySpace, Facebook, Twitter, LinkedIn, and so on, have been described as one of the defining characteristics of Web 2.0 technologies. This offers new opportunities and perspective to study the relationship between the urban physical environments and location-based social media networks
The limitations of the Google Street View as Urban perception dataset
News from CNN
John D. Sutter is a columnist for CNN Opinion. There is a Lake Providence, which is an oxbow lake that separates a predominantly wealthy neighborhood from a mostly impoverished one. After the article was published, a reader pointed out to me that, at the time, it was impossible to search the poorer side of Lake Providence using Google Street View.
The worldwide distribution of Google Street View
Putting aside commercial and political reasons, North America has the highest coverage, followed by some areas in Asia and South America, while Africa has the lowest. Google Street View is not evenly distributed throughout the world. In some remote areas and developing countries, GIS data coverage and updates may be relatively low. Street view images are usually taken at specific times and locations, and some urban features may be difficult to capture in their entirety due to limitations such as acquisition angles and occlusions.
- Inadequate data coverage
In some remote areas and developing countries, GIS data coverage and updates may be relatively low. This is due to the relatively low level of infrastructure and information technology in these areas and the high cost of data collection and updating. In addition, the relatively low commercial value and user demand in these areas may lead to bias in resource allocation by data providers.
- Insufficient data for disadvantaged communities
Some disadvantaged communities (e.g., low-income, minorities, etc.) may be underrepresented in GIS. This is because the geographic location, living environment, and cultural characteristics of these communities may differ from those of the mainstream society, leading to difficulties in data collection and analysis. In addition, the political status and economic power of these communities are weak, which may affect their rights and interests in the GIS.
- Limitations of data time and location
Street view images are usually taken at specific times and locations, and these images may not fully reflect the overall characteristics of a city. In addition, the frequency of updating streetscape images may also affect the accuracy of the study results.
- Acquisition angle and occlusions limitation
Even in areas where street view data is abundant, some urban features may be difficult to capture in their entirety due to limitations such as acquisition angles, occlusions or lighting conditions.
The User Generated Dataset - Social Media
Within the field of urban perception research, user-generated content represented by social media is also a popular area of research. They have advantages such as real-time and diversity, but their data is subject to selection bias, copyright issues, and overuse problems.
Crowdsourcing Materialfolio
Aims
My vision is not to criticize Google Street View and social media data, but to try to create a crowdsourced urban awareness program with a timeline, privacy protection, clear user groups, and defined data usage by analyzing their pros and cons.
Urban Perception sample
When we talk about urban perception, street scenes and pictures provide a machine-readable perspective of the overall urban image, in which building materials play an important role. In different regions, cultures and historical backgrounds, people may have different preferences and uses of building materials. The visual characteristics of building materials such as texture, color, and gloss can bring different feelings and impressions. From this perspective, the shaping of urban perception by building materials is particularly important.
MIT Sensible City lab used urban images as the basis for their research. They analyzed the similarity of cities using predictive methods based on object recognition data. I believe that crowdsourced material data can provide an additional dimension to benefit this type of research approach.
Benefit from one more modal: Built environment Material
In the era where machines act as perceptual agents, how do we extract the characteristics of the real world and transform them into machine-readable digital data? This is also the reason why multimodal machine learning is prevalent. We can abstract each modality as a sensory organ of our body. Therefore, in computer vision, multimodal machine learning models such as image depth and object recognition have emerged.
Target User and Data Usage
- Environmental protection and sustainability
Environmental organizations and research institutes may be interested in the environmental impact and sustainability of building materials. By analyzing Street View data, they can assess the environmental performance of building materials and promote green building and sustainable urban development
- Urban planning and management
Urban planning departments and government agencies can analyze the building materials in the streetscape to better understand the architectural style, historical evolution and current condition of the city, so that they can develop more reasonable planning policies and conservation measures
- Construction industry
Construction companies, contractors and developers may be interested in the identification and analysis of construction materials to understand market dynamics and competitor construction methods to optimize their own project planning and cost control
- Historic building conservation
Cultural heritage conservation organizations and researchers can better understand the historical context and conservation value of buildings by identifying original materials of historic buildings. This will help develop effective conservation measures and restoration programs.
Application: Materialfolio
It is necessary to choose the size and type of the pictures, which guarantees the universality of the data. As time goes on, this application will establish a timeline to view pictures from different periods.Both anonymous upload and real name upload modes are designed to ensure the privacy needs of different users.
This application mainly sources data from the pictures taken by crowdsourcing users, as a supplement to Google Street View. This provides dimensions for personalization of the dataset. At the same time, when uploading pictures, users can choose between anonymous or real name modes.
The key technology to implement the crowdsourced building material map is real-time uploading and updating. I used ReactJS to build an interactive frontend map, and at the same time, a material segmentation machine learning model is running on the Flask API backend. This model can recognize 56 common materials, and can perform real-time recognition when users upload pictures.