Problem

Mobile devices should react to user behaviors or... and something...

Solution

Kinesthetics is the ability to detect movement of the body; while usually applied in a self-aware sense, here it refers to the mobile device using sensors to detect and react to proximity, action and orientation.

The most common sensor is the accelerometer, which as the name says, measures acceleration along a single axis. As utilized in modern mobile devices, accelerometers are very compact, integrated micro electro-mechanical systems, usually with detectors for all three axes mounted to a single frame.

In certain circles the term "accelerometer" is beginning to achieve the same sense of conflating hardware with behavior as the GPS and location. Other sensors can and should be used for detection of Kinesthetic gestures including cameras, proximity sensors, magnetometers (compasses), audio, and close-range radios such as NFC, RFID and Bluetooth. All these sensors should be used in coordination with each other.

Kinesthetic gesturing is largely a matter of detecting incidental or natural movements and reacting in appropriate or expected manners. Unlike on-screen gestures -- which form a language and can become abstracted very easily -- kinesthetic gesture is about context. Is the device moving? In what manner? In relation to what? How close to the user, or other devices?

Design of mobile devices should consider what various types of movement, proximity and orientation mean, and behave appropriately.

Specific subsets of device movement such as Orientation have specialize behaviors and are covered separately. Location is also covered separately. Subsidiary senses of position such as those used in augmented reality and at close range (e.g. within-building location) may have some overlaps, but are not yet established enough for patterns to emerge.

The use of a secondary device for communicating gesture, such as a game controller, is covered under the Remote Gestures pattern.

Variations

Only some of the variations of this pattern are explicitly kinesthetic, and sense user movement. Others are primarily detecting position, such as other device proximity...

• Device Orientation - Relative orientation to the user, or absolute orientation relative to the ground. Either incidental, or natural gestures. For example, when rotated screen down, and no other movement is detected, the device may be be set to "meeting mode" and all ringers are silenced; as the screen is the the "face" of the device, face-down means it is hidden or silenced.

• Device Gesturing - Deliberate gestures, moving the device through space in a specific manner. Usually related to

natural gestures, but may require some arbitrary mapping to behaviors. This may require learning. An example is the relatively common behavior of shaking to clear, reset or delete items.

• User Movement - Incidental movement of the user is detected and processed to divine the user's current behavioral context. For example, gait can be detected either when the device is in hands or in a pocket; the difference between walking, running, and sometimes even restless vs. working (e.g. at a desk at work) can be detected. Vibration generally cannot be detected by accelerometers so riding in vehicles must be detected by location sensors. Other user movements including swinging the arm, such as moving a handset towards the user's head.

• User Proximity - Proximity detectors, whether cameras, acoustical or infra-red, can be used to detect how close the device is to a passive object, such as the user's head. A common example is for touch-centric devices to lock the screen when approaching the user's ear, so talking on the handset does not perform accidental inputs. Moving away from the user is also detected, so the handset is unlocked without direct user input.

• Other Device Proximity - Non-passive devices, such as other mobiles, or NFC card readers can be detected by the use of close-range radio transmitters. As the device approaches, it can open the appropriate application -- sharing or banking -- without user interaction. The user will generally have to complete the action, to confirm that this was not accidental, spurious or malicious.

Proximity to signals or other environmental conditions further than a few inches away is not widely used as yet. When radio, audio or visual cues become commonly available they are likely to be a related contextually related manner.

Interaction Details

do action???

combination gestures. almost all are. See device orientation in the variations... Combinations include things not included, like GPS. When you walk to a car, accelerometer detects you walk, then GPS determines you are moving at vehicle speeds, etc.... combinations are not momentary but over time. The device should monitor full time... privacy concerns though...

User movement (swinging up) and user proximity (your head) have to work together. Otherwise it might lock when you wave it by a table... combinations add intelligence, to make sure actions are more contextually accurate. Sensors can be activated based on other sensors, to reduce output and battery consumption. Proximity detectors, for example, can be activated only when a proper movement type is detected.

Kinesthetic gestures are not as ubiquitous as On-Screen Gestures so may be unexpected to some users. Settings should be provided to disable (or enable) certain behaviors.

Presentation Details

when reacting to a proximate other device, present the relationship on screen when possible... when known, display items on the screen adjacent to the location of the adjacent device...

Antipatterns

Examples