Lessons from working with the press: Cultural considerations and terminology surrounding American Sign Language in engineering research

I am the corresponding author (i.e., the professor who runs the lab which does the work) of the paper The Language of Glove: Wireless gesture decoder with low-power and stretchable hybrid electronics. PLoS One 2017. DOI: 10.1371/journal.pone.0179766. I must take ownership of an issue surrounding this paper that has the potential to insult both the Deaf community and the linguistics community. Since the publication of our paper, it has come to our attention that we used wording surrounding American Sign Language imprecisely. This imprecision possibly led to inaccurate portrayals of this work in the media that suggested that the device described in this paper “translates” American Sign Language (ASL), when in fact what it does is convert signs of the American Manual Alphabet to text. This capability is properly called fingerspelling, and is used by signers to convey loan words from English. American Sign Language is a language rich in complexity and consists of movements of the hands, the arms, the body, and the face, which are well outside the ability of the gloves to detect, much less to translate. Any suggestion that a single electronic glove is in any way capable of translating ASL was unintentional. The device was, in fact, never intended to be an accessibility device, nor could it ever be used as such (the signer could simply write the letters and not bother donning any awkward equipment). This episode, nevertheless, taught me important lessons about the ways in which research is perceived in the public, and for which projects and under what circumstances it makes sense to encourage press coverage.

It is common practice in research in materials science and engineering to produce a rudimentary device to demonstrate practical value of a new material or system of materials. This is why the literature in thin-film electronics is littered with demonstrations of inefficient solar cells, transistors with low charge mobilities, and biosensors that are completely impractical to translate from lab to market. These demonstrations are, in effect, required by editors and peer reviewers to show the value of a new type of electronic material. They also help the researchers to discover challenges of integrating materials into devices, and to communicate these challenges to other researchers in the field. The purpose of our paper in PLoS One was to demonstrate integration of soft electronic materials with low-energy wireless circuitry that can be purchased economically by research laboratories. It was our hope that a complete description of such a device in the peer-reviewed literature would lead to innovation in virtual and augmented reality (e.g., for education and surgical training), devices for physical therapy (e.g., for rehabilitation or for neurological disorders when the sensors are coupled with actuators), and devices for non-verbal communication for first responders (e.g., police). As we are fundamentally organic materials scientists, we were excited to find that a common conductive fluoroelastomer exhibited good mechanical robustness and a high degree of piezoresistance, that is, change in electrical resistance with mechanical strain. In other words, it makes a good sensor for wearable electronics. It has the additional benefits that it can be cast from solution: this capability opens the door to large-scale, printed electronics. Indeed, the repurposing or rediscovery of common materials has a long history in materials science. Consider the use of poly(dimethylsiloxane) (PDMS, or "silicone rubber"), which after decades of use in caulking bathtubs and grouting ceramic tiles is also responsible for the development of microfluidics, chip-based biological assays, and implantable electronics. Even if PDMS is not present the final product, it is a safe bet that it was used in the R&D stage. 

We thus sought to showcase the ability of the fluoroelastomer to detect signals by integrating it with a wearable sensor. This device could have been anything, but we chose a glove, because the human hand is an amazing thing: it is capable of creation, exploration, and expression. As a demonstration of the ability of this glove to recognize gestures, we used it to convert the letters of the American Manual Alphabet to text viewable on a smart phone. We chose this alphabet as our model system because it comprises a set of 26 standardized gestures (including stretching strains, pressure strains, and accelerations), which represent a challenge in engineering to detect using our system of materials. Unfortunately, use of the word “translate” with respect to the alphabet used in ASL encouraged some in the media to suggest that the glove could “translate ASL.” This characterization made it possible to interpret this aspect of our paper as an example of cultural appropriation—that is, taking an element from a culture to suit our own needs. Needless to say, this was not our intent. I learned the American Manual Alphabet in elementary school from members of the Deaf community in my hometown of Rochester, NY. In fact, my parents used to take me to the ASL-accessible mass at my church, and I always appreciated the beauty in the expression of language, ideas, and emotions through sign. So, rudimentary fingerspelling as a demonstration made sense to me. I was not so deceived as to think our device would be the first – nor best – device capable of detecting fingerspelling. (A confession: my publication history, of which I am proud, is replete with solar cells that can't power an LED and wearable pulse sensors that you can't wear outside in the rain, but which nevertheless highlight the properties of materials with the idea that they will eventually be made useful for something.) In fact, over the last few years, bright and enthusiastic high school students have used off-the-shelf components to produce other fingerspelling gloves and posted their efforts on YouTube. (For the record, I think it's awesome that young people are taking an interest in working with electronics at a young age. All of my inventions – alarms for my bedroom, and later rudimentary video games made on QBASIC – were, by comparison, characterized by a high degree of crappiness.)

In 2016, a team of students at the University of Washington (UW) won the Lemelson-MIT Student Prize for a glove capable of recognizing gestures. This invention received significant press coverage, which characterized it as capable of “transliterating sign language into text and speech” (SignAloud, http://lemelson.mit.edu/w...). An open letter published by the UW Department of Linguistics to UW’s Office of News and Information objecting to the characterization of this device and similar inventions can be found at the top of the page here, https://catalyst.uw.edu/w... or directly here. We encourage all researchers, tinkerers, and reporters in the field of human-machine interfaces to read and understand the arguments and legitimate concerns made in this letter. To summarize, the open letter takes the position that such devices are designed as a convenience for hearing people as the burden is on the signer to wear the apparatus. (While in principle there exists the possibility of hearing people to use more sophisticated versions of such devices for the purposes of learning ASL—as in the voice-recognition software used for learning spoken languages—such an explicit capability was not mentioned in our paper.) Perhaps a larger issue is that such technologies reinforce the misconception that sign languages represent mere manual versions of spoken languages, when in fact they have their own grammar and syntax, and are capable of expressing ideas and emotions in ways that spoken languages are not: an analogy would be a word or phrase in one spoken language for which there is no adequate translation into another. Characterization of gesture-recognition devices as “sign language translators” suggests that members of the Deaf community “need help” and may contribute to audism—the discrimination against and marginalization of the Deaf.

I learned of the backlash to our paper (and to press coverage thereof) in the linguistics community when a board member from the Linguistics Society of America (LSA) contacted our Dean of Social Sciences at UC San Diego, who herself is a member of the Deaf community and a renowned expert in ASL, with the concerns of the LSA: coverage of our paper contributed to several misunderstandings about ASL and of Deaf Culture. A quick exploration of tweets from the linguistics world around the time our paper was published confirmed that yes, people were upset. My Dean of Engineering forwarded me the concerns of the LSA, and together we all agreed that a clarification of the goals of the paper and an acknowledgement of the cultural issues by me—in the official record of the paper—would be the appropriate response. The LSA board decided not to issue a public statement, on the basis of my official Comment (of which this post is an expanded version), which is now part of the paper as recognized by the rules of PLoS One. While the paper itself was explicit that the device can only convert signs corresponding to the American Manual Alphabet to Latin letters, I  understand why a linguist or a Deaf individual would smack her or his forehead in disbelief in a news story that described my student’s work with the headline “UCSD Engineers Develop Glove that Translates American Sign Language.”

It is critical for researchers and reporters to be aware of cultural issues when communicating scientific results to the media. The business of reporting in science and technology is admittedly difficult. There is an incentive for journalists to focus on the most-clickable, photogenic, or easiest-to-explain aspect of a story. A scientific paper written primarily to teach the results to other scientists (as the vast majority of them are) makes for uninteresting journalism. Moreover, the public may not have the patience (or interest, or time) for learning the values that underpin the choice of a researcher to pursue a particular project. A project is always laden with subtle or implicit arguments that do not make sense outside the frame of reference of the field. For example, our paper, in which the key innovative element was a new system of materials for stretchable hybrid electronics, triggered many of even the most careful outside observers to ask "Why make a 'signing glove' when high school students have already done it on YouTube?" Science journalists cannot possibly be experts in all aspects of every field in which they are asked to report. This limitation is especially consequential when research in the physical sciences and engineering touch on aspects outside of their traditional boundaries. Peer review cannot be expected to catch all instances of overstepping, as the reviewers are generally also physical scientists and engineers, and may also be ignorant of the cultural ramifications. For example, I originally thought the pun in the title of our paper ("The Language of Glove") was cute. I still rather like it, but I now understand its cringeworthiness within the Deaf and linguistics communities (and not in the the good way that all puns aspire to cringeworthiness). The onus is thus on the researcher to be aware of cultural issues and to make sure—to the extent possible—that word choice, nuance, and how the technology may impact a culture is properly conveyed to the journalist and thence to the public. Nevertheless, I remain proud of my students' achievements in the work described in this paper. It was a long slog to get all the materials to play nicely together, and we now have a wonderful new platform on which to test our materials for a wide range of applications.

–Darren

[An earlier version of this post appeared in the Comments section of PLoS One.]