Neighbor-ly: IoT, Machine Learning and Social Relationships

What Is Neighbor-ly?

Neighborly Noises and Urban Living

Neighbor-ly is a fictional smart home object designed specifically to be in urban homes. It’s designed to address a problem that is all too common for people sharing walls with neighbors in city apartments, dealing with those obscure and obnoxious sounds that travel to your home in the most inconvenient of times. Is that someone on stilts walking around upstairs? Are they moving and dragging furniture all afternoon? Are those bags of marbles being dropped on the floor? What is that sound?

People want to feel that their homes are grounds for solace from a chaotic life outside, especially those living with roommates in large cities. What happens in reality, however, is that moments meant for peace and relaxation are rudely disrupted by all kinds of noises, often unpredictably. The mundane but disruptive noise leaves the urban dweller with little to do but to feel frustration in silence, growing resentment over time towards their neighbors.

How Does Neighbor-ly Work?

Neighbor-ly uses machine learning to be trained on picking up noises coming from neighbors, classifying them, and then reacting based on that classification. When it hears a sound, it also produces a phrase or a sentence guessing at the source of the sound, just like a person does when they hear something out of the blue. It learns over time which noises are classified as nuisances by the owner. The reaction from Neighbor-ly, a knock against a wall, occurs only after the object has been trained to understand and realize that this is a noise that the owner does not like. The knock is a signal to the neighbor. Quiet, please!

We know that not all noises may be annoying, some may be pleasant and others could be something that people can’t do much about, like a baby crying. However, the sounds that are problematic and obnoxious to the owner are trained by Neighbor-ly. The classified "bad" sounds are stored within the internal system. Being trained on the sound 3 times is enough for Neighborly to detect the "bad" sound. When the noise happens again, the knocking gesture by Neighbor-ly is similar to how people used to do it in the old days with a knock at the neighbor with a broom.

Neighbor-ly externalizes an experience that often develops into unspoken anger between neighbors. A loud party once in a while, which one might feel more comfortable confronting a neighbor about, is different than the more frequent and mundane interruptions that occur with more regularity. It leaves an urban dweller in discomfort without any action they can take to resolve the issue.

Through training and customization, Neighbor-ly allows the owner to feel like something is being done. The object is also a silent but proactive witness to the situation. In addition to that, it plays a role in adding humor to the experience to lighten the frustration for the owner by taking a guess at what the sound could be. It’s just as curious as you are about what that noise is!

Over time, a relationship forms between Neighbor-ly and the owner, because it begins to embody parts of the owner’s personality in some senses. Neighbor-ly and the owner are a duo team on a mission to live more peacefully in their apartments.

What’s Inside?

Neighbor-ly is powered by Arduino and the machine learning software Wekinator. A push button connected with Arduino allows the user to train the device by sending a signal to Wekinator via open sound control (OSC) to be classified. The system needs a few samples of the sound to make better classifications and complete the training. In that way, a user creates their own library of sounds that are classified as “annoying”. When the system hears a sound, it runs it through the library. If a sound is classified as annoying, Wekinator sends a signal to Arduino to trigger the “broom”, activating a servo motor that controls the knocker.

The hardware consists of a WiFi enabled Arduino, a motor that moves the “broom”, a microphone that captures sounds and an LCD screen that provides feedback to show the training progress as well as the humorous guesses about annoying sounds.

How Did We Get to Neighbor-ly?

Measuring Annoyance

We specifically looked at the future of smart products in homes. The home is a rich arena of information, detecting movement, sound and behavior patterns of not just of those living in the homes, but of things in and around the home as well. As we zeroed in on the specific issue of noise, we asked the question: how do you measure annoyance?

To take this question forward, we focused on the creation of a sound data set to execute on the machine learning training. This would help us understand how the owner of the product would think of recording annoyance. We quickly learned that there is no existing sound data set for “annoying neighbor sounds." In fact, machine learning data sets are often with visual data sets.

We decided to create our own sound data set to train Neighbor-ly. We simulated different types of sounds, both outdoors and indoors and created our custom sound data set to simulate training, classification and reaction over time. We collected and recorded different sounds on the streets from drilling, to construction to recording a wide range of sounds that occur in the home that can be overheard by neighbors.

Are You Thinking "What’s the Point of This?"

Good!

IoT and Social Relationships

Customized products have radically changed how we consume and experience the world around us. Continued injection of emerging technologies, like machine learning and the Internet of Things (IoT) to make everything “smart” can sometimes be reckless, and perhaps unnecessary. As is the case with Neighbor-ly perhaps.

Ultimately, our exploration of Neighborly is a social commentary on how IoT devices will become more and more intrinsic to human life, and sneak in closer and closer to a person’s personal space—from wearables to homes. The prominence of ubiquitous devices may ‘solve’ for problems, however, the cost of those solutions may be that it breeds, or worse, amplifies, hostility or avoidance between people.

When objects become mediators between individuals, it becomes increasingly easy to avoid the type of discomfort that is necessary for human relationships to flourish—knowing how to resolve conflicts for instance, or how to develop and act on empathy through active listening or knowing how to compromise through conversation and negotiation. “Solutions” like Neighbor-ly perhaps creates more distance and silos between human beings, while making people feel righteous in their silos.

With the presence of more and more customized devices and services, which learn the user’s behaviors, needs, desires, and wishes to the point of predicting those desires, the line gets blurry between a device correctly predicting what someone will want to do, to directing them to what they should do. As this becomes the normative behavior of humans and objects both, each depending on the other to tailor a human’s experience to specific exposures, the presence of spontaneity, discovery and serendipity from a person’s life is slowly eliminated.

Is This the Type of Future We as Designers Want to Create for Society?

Neighbor-ly is an exploration of cramming in an IoT solution in a space where the solution is perhaps just human conversation, not an additional device that learns the user’s behaviors, patterns, and in this case, aggression. Neighbor-ly is an object that embodies the owner’s frustration. Perhaps over time, the impact of an object like this in society would start off with embodying the personality of the owner, to then amplifying it, which may then have unintended consequences.

Turns out gadgets have already been invented to “solve” the problem of the noisy upstairs neighbor and the demand for them has not only heightened neighborly conflict in some cases, but it also causes damage to the physical home itself.

What’s Next?

Neighbor-ly is an artifact. We intend to continue using it as a device to entice investigations around the potentially intrusive nature of connected devices as they develop further and further. An object like this also raises a bigger question about the ethical use of data. Should we be so comfortable with a device listening to every sound happening in a home for the sake of it providing a small convenience once in a while?

The next step in our process would be to develop a set of objects that mediate more of such deeply nuanced human connections. The aim of producing a series of such IoT devices is to construct a strong narrative to understand our hypothesis and make a stronger case for designers and engineers to consider the ethics of their decisions.

This is a team project by Rina Shumylo, Abhishek Kumar and Fahmida Azad done as a part of the Trainable User Interface course at the Copenhagen Institute of Interaction Design, under the mentorship of Massimo Banzi and Simone Rebaudengo.