︎

SoftNote



Class Project:  05-499 Building User-Focused Sensing Systems

Team: Catherine Yu, Willa Yunqi Yang, Jiaqi Liu.

Role: Programming lead. 

Date: May, 2021

Length: 8 weeks
Introduction:
In this project we explore gesture detection in connection to resistive stretch sensing. Specifically, our project is motivated by the piano learning experience for people with visual impairments as it is hard for them to keep track of fingerings when practicing. Beyond piano learning, our approach also fits into the larger picture of hand gesture detection as it detects finger tapping and movements.


Materials:
Materials we used include Arduino Lilypad, conductive yarn for sensor fabrication, conductive fabric and threads for knitting and connecting components, resistors, and non-conductive yarn and fabric.

Sensor Fabrication:
When it comes to the fabrication process for each sensor, we got close to the optimal solution for us during multiple experiments. The key stages for it are choices of yarns, exploration of sensor making approaches and sewing on the limited space. The following presents the road blocks we encountered during the sensor fabrication process, and they shaped our current sensor fabrication solution.
Choices of yarns: The behavior of yarns and our access to them both influence our choices of yarns. First, we select conductive yarns instead of resistive yarns. Among the types of yarn we have access to, the latter one provides a wider range of readings, but they showed irregular changes as it is stretched, i.e. even if the yarn is at rest, the resistance changes during our experiments. On the contrary, the conductive yarn presented more consistent readings and it is sensitive enough for our purpose, even with a small range of change. So we moved forward with conductive yarns, which are shown as #3 and #5 in. Furthermore, we chose #5 for our final sensor demo. #3 showed cleaner data than #5 which is available to our team member in China, but two of us who are making the sensors only have access to #5. Besides various conductive yarns, we also experimented with different normal yarns, while considering stretchability and thickness for each of them. The stretchability sets the range of resistance changes, and whether the changes are enough to provide available signals. We should also consider the thickness of yarns, as our toy knitting machine can only knit relatively thin yarn, like the rightmost one of Figure~ref{fig:2.3.1-yarn-normal}. It’s also what we selected for our sensor demo.


Exploration of sensor making approaches: To build the sensor with appropriate stretchability for our finger movements and make it relatively reproducible, we tried multiple combinations of conductive yarns and normal yarns. There are two major approaches standing out. The first one is using knots with thick yarn, which is shown as #1 in left picture of the figure. The knot shows visible resistance changes with only 60cm conductive yarns per sensor, as it uses two strands per row. The approach saves materials for our limited resources. However, it turns out that the total resistance might be too low, and it have even burnt out our first Lilypad.
Then we tried a new approach using longer conductive yarns, which means we should turn to knitting fabrication method because it can knit longer conductive yarns into our sensors. Using knitting machine also makes sensors less difficult to duplicate. The right picture of the figure shows the toy knitting machine we experimented with. We finally adopted the second method, and used 200cm conductive yarn per sensor whose resistance value was 50ohm.
Sewing on the limited space: SoftNote is designed to be a lightweight fabric on wrists, where we are actually limited by the space there for sewing yarns in intricate schematics. It took a lot of efforts for us to avoid conductive touching each other. Our solution behind it was adding in 3 extra fabrics as layers and sewing branches of our schematic on each of them. The figure zooms in on these fabrics.
Signal processing:

Filter choices:
Initially, we recorded some data, shown in the top row in Figure 7, using a sensor made using conductive yarn with cleaner data and tuned the filter parameters for that set of data.
Then we applied the same filters with tuned parameters to another set of data, shown in the bottom row in Figure 7, using a sensor made using conductive yarn with noisier data, which is the one we are using for our final prototype.
At the time, we only had one sensor fabricated and we were hoping to find filters that can easily generalize because our handmade sensors don't have identical behaviors.
Mean filter and low pass filter performed significantly better than the band pass filter.
We ended up using low pass filter because of the high frequency noises in our data.
Pipeline:
In this section, we will discuss the signal processing pipeline(Figure 8).


Firstly, there is the raw data, which is the resistance measured using voltage divider. The sensor resistance decreases as it is stretched, and increases as it is relaxed. We observed that the sensor value does not return to its start value after we return to the start position, but the trend of stretched and restored is quite obvious.

Low pass filter is applied to clean up the high frequency noise as explained in the ‘filter choices’ section.

Recommended by the professors, differenceS between neighboring values are calculated so that the data is not affected by the increasing resistance as the sensor is being used. This step is very important because sensor value does not return to a consistent value and calculating the difference preserves the stretched v.s. restored changes but eliminates the increasing trend. Although the light blue curve has a increasing trend, the dark blue curve does not.

Then mean filter is applied smooth the calculated differences.

Then we applied the median filter for zero crossings.
Finally let's look at the raw data with where stretch and restoration detection are made. Also note how much the raw resistance has increased.
Calibration:
Before we start detecting with the sensors, we need to first calibrate the sensors. We ask the user to tap 5 times with each finger to find the thresholds using the 5 peaks detected. Calibration is needed before every single use because sensor initial resistance is not predictable. We suspect this behavior

could caused by the the following reasons fatigue in the knitted sensor as it becomes looser after it’s used.
Limitations:
The greatest limitation was that data is very noisy because conductive yarn is very noisy. Again we had access to different conductive yarns, the accuracy improves a lot when used with the conductive yarn that is cleaner. We read that using resistive yarn instead of conductive yarn gives a larger range for data change which could improve accuracy[1], but unfortunately, we were not able to purchase a resistive yarn that was not extremely noisy.
Another limitation is finger movement dependency. In Figure 18, we can see that when only one finger(visualized as the pink curve) moves, data for other fingers also change significantly. This is because natural muscle movements dependency. It's hard to keep the other fingers completely quiet when one finger is moving. Noisy data is another factor here. There might also be other factors contributing to the finger movement dependency like sensor placements.
Future Works:
DIY Low-Cost Gesture Recognition:
While our current demo shows a proof of our ideas and applicable results, we could further refine it for better usability and higher accuracy for gesture recognition.
- Communication: To make the sensor more user friendly and practical in multiple scenarios, a bluetooth module could be added to make it wireless.
- Output: The output signals can also be integrated with audio feedback, which could benefit people with visual impairment, or we could provide API interfaces for gesture triggering.
- Sensor: For the sensor itself, maybe we could experiment with more yarns and find one thinner. Adjusting the knitting mechanism might help with removing the folding in our current sensors.
- Signal Processing: More research could be done for handling finger dependences with software.
Other Use Case:
We explored our use case of piano playing, the stretch change is actually quite small compared with existing works like elbow bend detection. And the fact that we were able to achieve acceptable accuracy gives us the confidence with cleaner raw data, there are many opportunities. Using a industry knitting machine allows much tenser knit stitches, less space between yarns, and using thinner yarns. Right now each sensor is 3 stitches wide, but it can easily become 10 stitches wide with the same width using industrial knitting machines. Prior works have shown that resistive yarn have a larger range of changes than conductive yarn, so using other yarns should also yield better results.
So far, we have only detected whether the sensor is being stretched. Future works can explore how much the sensor is being stretched.
Last but not the least, knitting can easily produce flat 2D sheets and even complex 3D shapes. For our prototype, we are really treating the knitted stretch sensor as a 1D string. Future works could look into stretch sensing from different directions.

References:
[1] Maureen Selina Laverky. Knitted Stretch Sensors. 2018. URL: https://www.maureenselinalaverty.com/what#/machine-knitted-sensors/.