Multimodal Human interface using sketch, speech and brain data in Computer Aided Design
Nenminisseri Mana, Amit
MetadataShow full item record
This study illustrates how we could use multi-modal data during collaborative design work in order to distinguish gesture pen strokes indicative of functionality of an object (gesture strokes) from those representing device structure itself, that is, object strokes (non-gestures). In previous related works, classifiers have been developed to discriminate between these strokes while either using the combination of sketch and speech or sketch and brain activity. Experiments indicated that the sketch, speech and brain activity together give us features that help improve the accuracy of these classifiers. In order to test this, a collaborative framework was developed for sharing designs using a server client program and simultaneously the speech and the brain activity were recorded. Furthermore, once all data from multiple subjects were recorded, the speech was aligned to the sketch strokes, using the 3-second window for alignment of speech to a given stroke. Data from the brain activity recordings were aligned with a similar window. After the gathering of the features, 7 classifiers were tested against multiple random training and test data sets gathered across all different subjects. The features associated with different modalities were used individually as well as in combination with the others in order to determine the best possible feature set that gives the highest possible accuracy. The combination of all three modalities gave us the best results in classification of gesture strokes from non-gesture strokes.