2025

Crafticulate

Pear Jansomwong

Advisor
William Martin

👋🏼 Welcome to Crafticulate, an Archiving the Language of Making 👋🏼

Crafticulate is about preserving and documenting pottery gestures—those silent movements of the hand that embed generations of knowledge into clay.

It uses machine learning, hand tracking, and interactive storytelling to transform the ephemeral knowledge of artisans into a living, growing digital memory.

I've always been fascinated by how knowledge moves—not through textbooks or blueprints, but through hands. Growing up around craft communities in Thailand, I often watched artisans work: shaping clay, weaving fibers, carving wood. They rarely spoke instructions. Instead, their hands danced silently, passing knowledge through movement, intuition, and shared memory.

Yet as cities modernized and traditions formalized, I began to realize how vulnerable this bodily knowledge is. Unlike written language, craft gestures often go undocumented. They vanish quietly when an artisan retires, or when urbanization displaces a community. When we lose these gestures, we lose entire ways of seeing, feeling, and relating to the world.

This growing awareness shaped the core questions that drive Crafticulate:

Can we preserve the silent language of making?

Can technology help capture the living, breathing memory of craft beyond static documentation?

🧶 Why Craft Gestures Matter

Artisans tell stories through their hands. A slight pressure, a wrist turn, a coiling motion — these are more than just physical acts. They are repositories of tradition, intuition, and embodied skill.

But unlike spoken language, these gestures often go undocumented. When they disappear, we lose more than a technique—we lose culture, relationships to materials, and ways of inhabiting the world.

Craft gestures are dynamic, not static. Yet most documentation—photographs, videos, written guides—flattens them into static representations. They fail to capture the richness of movement: its timing, its evolution, its silent dialogue with material.

Traditional craft learning happens through physical observation and repetition, not verbal instruction.
If an artisan retires without an apprentice, the lineage of making can vanish quietly.

Other fields—like American Sign Language and dance notation—have developed ways to document movement computationally. Could we imagine similar systems for preserving and teaching craft gestures?

Crafticulate explores this idea: Preserving the language of craft through machine learning and interaction, not through frozen images.

🚕 The Journey Toward Crafticulate

Crafticulate didn't emerge all at once. It grew from two earlier research paths that shaped how I think about craft, space, and knowledge:

🌐 The Digital Tapestry: Uncovering Hidden Networks

My first investigation, "The Digital Tapestry," mapped the invisible ecosystems of craft communities in Bangkok. It started with a simple question:

What if craft communities existed right under our noses—yet remained invisible?

Through interactive maps, soundscapes, and image archives, I explored how informal trust networks, spatial memory, and shared material practices sustain these communities—despite being absent from formal records.

Preserving craft, I realized, isn't just about finding artisans or cataloging techniques. It's about capturing the fragile, often invisible systems that allow craft traditions to breathe and adapt.

Crafticulate

🧾 Nonlinear Interactive Instruction: Embodied Learning Systems

Building on that, I shifted direction in Colloquium II —starting not with space, but with instruction itself.

How do you learn a craft technique? You don't read about it. You watch. You mimic. You adjust by feel.

Through speculative interfaces and real-time gesture tracking using Procession (developed by Spatial Pixel), I explored whether embodied, nonlinear learning could be digitized: Could a digital system teach not through rigid steps, but through movement, experimentation, and adaptation?

How do you teach the slight press of fingers flattening clay?

How do you capture a thumb's subtle rotation into the curve of a bowl?

I found that craft learning is inherently nonlinear, cyclical, and deeply resilient—but also fragile when cut off from embodied practice.

🧶 Crafticulate: Weaving Systems Together

These two threads—mapping hidden systems and designing embodied learning—ultimately wove together into Crafticulate.

Crafticulate treats gestures as living memory: moldable, relational, adaptive.

It is a speculative tool, a digital playground for pottery gestures: each gesture a key, each interaction a moment of making, feeling, and learning.

At the heart of Crafticulate is the idea of moldability:

🟤 Clay is moldable.
📚 Knowledge is moldable.
💻 Archives should be moldable—not rigid museums, but living, breathing systems shaped through participation.

🎨 Crafticulate in Action

When you visit Crafticulate, you begin at the Gesture Gallery — a constellation of floating pottery gestures you can explore freely.

You can filter gestures by:

Process (e.g., forming, shaping)
Stage (e.g., centering, finishing)
Application (e.g., bowl-making, vase-making)
Hand Use (single-hand, double-hand)
Technique (e.g., coiling, pinching)
Cultural Context (comparisons across traditions)

🎟️ Following a Gesture Journey

Suppose you choose bowl-making. You're guided into a process timeline—almost like a recipe made of movements.

Each gesture flows into the next: kneading, pinching, pulling, lifting—each shaping the clay toward form.

🔍 Diving Deeper: Gesture Metadata Pages

When you click on a gesture—say, Ball Kneading—you open its deeper archive page, revealing:

A 3D model of the gesture, which you can rotate and animate.
A description of when and why this movement is used.
The tool or material involved (hands, ribs, wheels).
Voice reflections or short clips from artisans sharing personal insights.
Cultural overlays comparing similar gestures across global craft traditions.

👐🏻 Practicing the Gestures Yourself

When ready, you can turn on your webcam and practice a gesture live. A real-time hand tracking model maps your movements through 21 keypoints, offering visual feedback—like a digital mirror guiding your hands.

Because gestures live on a spectrum, the system doesn't enforce rigid categories. Small variations are part of learning, just as theey are in real craft.

🔍 Technical Development Journey

Step 1: Recording artisans performing pottery gestures using webcam capture and MediaPipe.
Step 2: Tracking 21 hand landmarks (X, Y, Z) and extracting movements into JSON format.

Step 3: Preprocessing data into CSV files, plotting landmarks in 3D space using Matplotlib.
Step 4: Building a gesture classification model using TensorFlow.

Model training reached ~100% accuracy in training and ~95% in validation—strong results but slight overfitting risks.

Depth perception posed a major challenge. Webcams struggle to capture subtle Z-axis shifts (e.g., pinching vs. pulling motions).

To improve accuracy, I expanded my dataset through multi-angle recordings and began live user playtests, revealing how users interpret and embody gestures differentlyy.

Future Directions

Crafticulate remains a living project. Future goals include:

Expanding the dataset with more diverse user contributions.
Refining gesture recognition across different hand types and movement speeds.
Building adaptive recognition for both one-hand and two-hand gestures.
Refining 3D visualization using p5.js or Three.js for richer feedback.

For me, Crafticulate is not just about storing craft—it's about reactivating it. It's about feeling it in your fingers, translating it across generations, and inviting others to mold it anew.

Craft is knowledge. Knowledge is movement. Movement is memory.

I hope you enjoyed Crafticulate as much as I enjoyed making it.

References & Acknowledgments

Tools & Technologies

Spatial Pixel - Real-time gesture tracking and visualization framework
MediaPipe - Hand tracking and pose estimation library
TensorFlow - Machine learning framework for gesture classification
Three.js - 3D visualization library for gesture modeling
p5.js - JavaScript library for creative coding
Figma - Collaborative design tool
Cursor - AI-powered coding assistant

Craft Communities & Inspiration

Pakkret Pottery Village, Thailand - Traditional pottery community and knowledge center
Clay Circle Workshop, Bangkok - Contemporary ceramics studio and learning space
Ban Bat Community, Bangkok - Bat making community
Fingerspelling - ASL movement documentation project by Hello Monday
Cybersubin - "Human-AI co-dancing," blending traditional Thai dance knowledge with virtual choreographic agents.

People

William Martin - Project Advisor
Seth Thompson - Faculty Support
Catherine Griffith - Faculty Support
Laura Kurgan - Faculty Support
Snoweria Zhang - Faculty Support
Violet Whitney - Faculty Support

Special Thanks

My family and friends for their support and feedback.
The Columbia GSAPP community for their insights and encouragement.