Under RISD Maharam STEAM Fellowship, I took on the role as an artist-in-residence and exhibit design intern at the Strong National Museum of Play, Rochester New York. After two months of researching, prototyping, and experimenting, I eventually developed an interactive product called SYNDBOX: a wireless, multi-user (and eventually programmable) sound controller that is made out of geometrical building blocks.

Here are some initial renders for the product and its logo:

Here are the three following themes that I pinpoint as necessarily relevant and framework-worthy, as well as the individual elements that I actively think about.

But first of all, why sound? Why not on-screen visualization or 3d simulations? True, eventually I am imagining these control points to be application agnostic, i.e. it behaves as an input device for a wide variety of application. However, despite its less than tangible appearance, sound occupies real physical space in a way that visualization could not. With multiple output channels, sound can be sculpted, and I reckoned that there is opportunity for it to be utilized as a component that parallels spatial computing. Additionally, the museum has not had the opportunity to explore sound and music on a more experimental approach, and after a few discussions we agreed that the outcome would be informative to both parties.

However, the default knobs and buttons of synthesizers are more often than not fixed to its hardware. The gesture associated with a button is either pressed or clicked, and that of knob is turned on a fixed plane. But what kind of musical expression would form if this button is instead rolled, thrown, shuffled, balanced, swung? When I arrived at this juncture I was wondering if I should just appropriate the smartphone which, with its multifarious sensors and image recognition abilities, is the exemplar of an interconnected control object/a wireless play object. And thus in thinking about its uniformly planar and rectangular shape, one could almost wish that there is a modular and universal cardboard holder for smartphones, similar to Nintendo Switch’s Labo! Imagine an application that allows a person to make a custom holder for a smartphone that they own, and interact with apps and softwares in a totally different way! (I’m poking at Google cardboard, but in a direction that is beyond goggle/optical application of VR/AR)

All that said, I ended up designing my own system instead. The reason for this was size as well as ease of programming. A smartphone could be bulky, and there were several proprietary limitation that had to be overcome. A DIY microcontroller like Arduino on the other hand, is more flexible and customizable. Eventually I ended up with modules that are half the size of my own smartphone, and I am certain that through mass manufacturing and further development, these could grow to be even way more compact. (On top of that I ended up learning about mesh network and inductive charging designs, which are both pretty neat!)

I chanced upon a number of important inspiration while I was on the go. Mike Streb, the head of the exhibit team, lent me his LittleBits Korg synthesizer pack, and it really got me thinking about modular control points. My supervisor mentioned Johann Sebastian Joust–a local multiplayer no screen video game which is facilitated by a simple handheld tool. The classic Froebel’s Gift became a starting and end point, and as such the concept of SYNDBOX arrived at a super simple approach: embed sensors (such as one that measures tilt, or a color detector) into each shapes, and let their spatial orientation and location determines ‘how far a knob is turned’ or ‘which button is pressed’.

After a series of sketches and experiments, I realized that geometrical shapes allow the transmission and control of certain logical parameter. For an example, at its bare minimum, a cube can express 6 different outcome since it has 6 different sides—just like a dice! If one were to explode this possibility even further, one can also utilize its resting on each edges and corner as a state, with all this eventually amounting to 26 states! Percussive samples on each? That’ll be a fun drum machine.

Furthermore, one can also measure the planar orientation at any point in time just like a knob. This is similar to the case of a cylinder—and in my opinion there is no coincidence for a cylindrical shape to already be used as the default shape of a knob. That said, imagine if this cylinder is plopped out of its base, and still acting as a rotational control point. Now, it can essentially be broken down into 3 different states. The first being it standing on its bottom cap, the second being when it is lying on its side, and the third when it’s standing on its top cap. 3 different interchangeable knobs! One of the basic example would be to have a continuous value control on one cylinder side (a more continuous change in pitch, like sirens for example), and a quantized value control on the other (this would be notes, do, re, mi, etc). 

The last module, the cone, is a color picker. On one hand, it still act as a knob; on the other hand, it is a placeholder that store the color of a surface that it sits on. Currently, the cone determines the character/timbre of a sound. However, in the future I am imagining these to be reprogrammable, i.e. a user can then store different functions on these color ranges. For an example, if the stored color tends to blue, the module will talk to the cylinder. Else, if the color tends to red, it will talk to the cube. With more modules and RSSI detection, these building blocks can be combined into a kind of pseudo-orchestra, or perhaps, as I was talking to one of my friends in Rochester, a tool for outdoor play activities.

Towards the end of the fellowship, I conducted a demo at the museum for the exhibit team. On top of that, my supervisor was really kind as to set up an external critique appointment with Workinman—a local game company. Both parties give feedback that are really helpful and productive, which concerns technicalities, pragmatic and marketing strategies, as well as possible future application. Some of the open-ended questions includes: what would be the tutorial to play with this product? Can the modules become objects that connects different installations, and string together a social experience? If these modules become consoles for a game environment, how can it be used outdoors and outside of the screen, similar to Johann Sebastian Joust?

This fellowship experience has truly rewarded me with the opportunity to learn and explore about the infusion of computational principle into analog play, and the various ways to honor social and spatial interaction. It has definitely set a momentum towards me developing a fully modular and spatial programming language.