Mobiles are not different from desktops because they are small but because they are connected and personal. Good products don't just meet a niche, but leverage the native intent of the interface. Lately, we have heard some gnashing of teeth as developers try to figure out how to make things that are useful for the Apple Watch. Meanwhile, users of Pebble wonder what the fuss is about as much has already been figured out regarding wearables.

Likewise, we have to design properly for large interactive displays. We cannot just make the same stuff we do on desktop, or mobile, really big and call it suitable on a large display. We have to understand the ways they are interacted with, and recognize large displays are public, and collaborative.

Control Methods

With the apparent ubiquity of the touchscreen smartphone, there's an assumption that all interactivity is touch. But really, there are several ways to control large displays:

• Remote control

• Control at a distance

• Very coarse control

• Fine control up close

Each of them has pros and cons, not just in price or suitability for installation, but in the way they encourage group behaviors, or the detail of control input.

Let's review each one in turn:

Remote Control

Now we remotely control much of our lives, but the first remote controls most people used were for controlling large displays. The basic pushbutton TV remote has much to teach us still. Good remotes are:

Individual

Only one person uses the remote at a time. Poor remote control systems allow multiple simultaneous input methods, and do not clarify when they are in conflict.

Responsive

When an input is performed, the remote indicates the control was sent, and the display device responds in a reasonable time. The response doesn't have to be completion of the request, just an indication that the request has started, and is being processed. Bad remotes have long delays.

Abstracted

The input method is indirect, so has to be abstracted at some level. There is no direct control, but a language of control imposed. Think of how channel changing on the classic old TV remote is up and down. Trackpads that try to allow direct mouse pointer manipulation work poorly because the pointing surface is moving, or at an arbitrary orientation to the display.

Of course, even TV remotes are becoming more complex, but the principle has been extended for many types of control, from industrial automation to collaborative public space wall displays. We can break down remote control into two basic categories

Fixed

Wall mounted units, kiosks, desktop computers and laptops since they are not really usable unless set on a surface. WHAT IT MEANS...

Portable

Simple pushbutton remotes, complex remotes with things like keyboards, smartphones and tablets with remote control apps on them. Gesture-enabled wands are discussed under Control at a Distance. MEANING

multiple workspaces... can replicate on remote, or have separate control... upsides and downsides (reflect mirroring computer experiences like when you do not want the users to see everything, even as simple as notes for a presentation... )

Designing for Remote Control

• Remote controls are unusual in that they are split in need, appealing to both very simple and very complex entry methods.
• Remotes require physical contact with the remote control device. This may be difficult due to environmental conditions, fears of loss or damage, or other concerns. If so, consider one of the other methods.
• Remotes are shared by physically handing control of the unit to another individual. This can have advantages and disadvantages in clarifying who has control, as well as the psychology of control.
  • Systems exist where there are multiple remotes attached to a single system. Limited or group remotes (such as classroom feedback systems) must be designed to clearly emphasize the hierarchy of control, and limits or capabilities of each type.
• Don't make up your own control language. Use standard mappings when such abstractions already exist. Don't change channels by click left and right. Spinning controls increment clockwise and decrement counter-clockwise.
• React instantly. It is okay if there are technical constraints on completing the request, but indicate the request has been received. Reflect inputs to the screen with indicators, or actually begin the action in a visible way immediately.

Control at a Distance

Here I mean direct control at a distance. Instead of using a remote intermediary, you gesture at the display unit (or an attached sensor) itself.

This actually can encompass "wands" or remotes like that used on the Wii, which assist with gestural sensing, but they are not very prominent anymore, with the advent of the Kinect there is no real need to hack those, when you can hack the direct gesture systems. The principles are similar, however.

The control systems work in two basic ways:

Gesture Language

Actions can be performed with simple gestures, or strings of them to form more complex commands, or series of commands. The gestures are generally not going to be very natural, so must be memorized. Typical users will not be able to memorize or apply more than a few gestures, so the design of the information has to be directed and simple.

Gestures are typically things like next, stop, back, details, options and so forth. This can be combined with other methods, such as voice control, to provide for input like typing which would be very difficult to perform otherwise.

Direct Manipulation

The other method is to use gestures to directly control elements on the screen, or to indicate them with a mouse pointer. This is the classic sci fi movie version of VR, and futuristic control systems of all sorts. The user grabs shapes, or moves the focus over a series of objects then indicates. Your first experience on many of these systems is exactly like this, as a virtual mouse is provided and you select options, type WiFi passwords and so on.

From this you may have noticed the key problem with it, that moving like this is tiring, and it is more tiring the more precision you have to use to make the system perform properly.

Designing for Control at a Distance

• Aside from entertainment, this is best for assistive systems where the user's primary task is on another device, or is in a real environment.
• Use these input methods briefly, or as short strings periodically. Allow the user to make a selection, or a gesture, then display information.
• Huge benefits in environments where the user cannot or should not touch the display. Some interesting research is occurring in hospital operating rooms, but the next step will be mechanics, outdoors, and public access devices.
• Be careful selecting gestures, to assure they are unambiguous, are not performed naturally so will only be used deliberately, and do not interfere with their primary job. In tests in the hospital setting, some gestures caused the surgeon to accidentally make contact with his upper body, violating sterile procedures.
• Don't expect to be able to train for part time or brief-use. Museum visitors cannot learn a gesture language quickly enough for it to be useful to them.
• You will have to reflect the input to the screen, which may distract other users of the system.

Very Coarse Control

gesture at a distance, both secondhand (ambient, environmental) and mostly, deliberate...

BRIEF... multi-user at once especially...

Designing for Very Coarse Control

• f
• f
• f

Fine Control Up Close

Large displays can be seen at a distance, and generally fall under the 10-foot-UI principles (meaning, assume users are consuming content from 10 feet away, vs arm's length or in the hand). But this is not fixed. Users can approach the display, and there may be advantages in allowing direct control of the interface from the large display.

Direct control here will mean touch. The touch technology used is not critical for the basic principles, but you should be sure to understand the impact each technology has on your design. Some do not work for multi-touch, or do not support gestures well, and target sizes (even though based on human physiology) are not the same for each technology.

Collaboration systems, especially, can benefit from use of large touchscreen displays. Various installations have had good results from allowing multiple users to organize cards at the same time, such as for sorting visual items, or creating shared task lists.

Presentations such as demonstrations can also use direct control of the display instead of canned or remote control driven responses.

A key problem with many of these systems or uses is that the person interacting with the display necessarily blocks some of the screen. While people are good enough at seeing around their own hand, it is much harder to do for those further away (partly due to parallax) and the user's body may also obscure the screen.

This is not different from the issues encountered with sharing on chart paper, on whiteboards, or even when presenting a slideshow. And these have lessons we can learn from. If it is critical that others be able to see the work while a single user is manipulating the display, consider use of an "on screen remote." An area near the edge can be designed to work at close range, and will encourage the controlling user to stand to the side of the display so their changes can be seen on the larger part of the screen.

Designing for Fine Control Up Close

• Remember it is single touch use, but with many observers. This is unusual in that you have two simultaneous and often conflicting audience types.
  • Try to place controls to the edges, or otherwise so the touching user does not obscure the screen. This also allows the touching user to anchor their hand to the bezel for additional precision.
  • Remember the near user has a different view, so it may be necessary to provide a monitor (duplicate the large view) so they can get an impression of what the room sees. This can even include how much they are obscuring the screen, as a reminder.
  • Interactions that only matter to the touching user, such as button feedback, should not be so strong they are distracting to viewers.
  • If possible, make near-use functions disappear (or become less prominent) when no one is within touching range.
• In some cases, more than one user may be touching the screen at one time.
  • Design this for the actual way it will be used. Is it only to be used up close, or will sometimes users step back to see the results, or share with the room? Considerations above still bear out, but can be more complex to develop as there may be no easy switch of modality.

Public & Collaborative

More important than any technology or method of interaction is the fundamental truth that most large interactive screens are used in public, and used collaboratively. Public doesn't have to mean literal free access, but just means not one-on-one as we are accustomed to with phones, tablets and computers. Large interactive displays are unlike desktop computers or especially mobiles due simply to their size. More often than not, the size is a direct offshoot of the specific need to be in public. At the lowest end of public we mean the TV in your home. Smart TVs mean someone controls a device which the whole family is observing.

Other cases such as museums, business contexts and so forth have the same basic issues as family Smart TV control. As single individual is in physical possession of the control unit (or has the focus of the gestural control mechanism) but may not be fully in control. He has to take input from the entire family, assure the group makes decisions, and make it clear what input he has requested of the device.

When designing for any of the control cases, two audience classes can be considered.

• Group
• Individual

And unusually, we do not just have to design for both audiences, but usually both experiencing some output at the same time. Let's look at some problems with this.

Feedback

Traditional input methods consider the controlling user, so when a selection is made feedback must be quick enough it is clear that the intent was executed. It doesn't have to take effect, but something must happen such as the button indicating click, a delay indicator, even just vibration and noise.

However, those not making the control inputs are slightly disassociated from the cognitive cycle. They may have requested an input, but only verbally. There may be competing requests, the controlling individual may have misheard, or ignored it, and time delays are much higher. The need is still there, so all users can orient themselves to the system properly.

Some systems, such as gesture based sensing, require feedback on the screen of the control. This is different than indicating focus or wayfinding, and the feedback may interfere with other users' ability to consume information. This may make some input methods less suitable for shared environments.

Wayfinding, Orientation and Notification

easier to get lost, especially as it does not hold attention

even the controlling individual can be distracted by the others in the room, or other tasks and miss cues or need to reorient

use mobile methods to remind and label at each point, as few transient indicators as possible

Focus

"no, up one" we've all done that in a meeting... but only because we can see what is going on... systems without focus like gesture? ???

More issues??? BRAINSTORM. GOOGLE SMART TV REVIEWS... ALL SHOULD BE TAKEOFFS ON BASIC SYSTEMS

Conclusion

Summarize findings in bullet list...

And also, general guidelines:

• Refer to TV guidelines on size, contrast and viewing angle when designing interactions and arranging rooms to assure far users can see the display.

future directions of research... call to action for folks to volunteer their projects???

• Accuracy of touch - Bet it changes due to device position. Level, above head, etc.
• Touch enhanced by anchoring to the bezel?
• MEASURE gorilla arm, don't just assume. Make people use for a while and observe, measure changes over time.
• How to account for near/far sizing? Does the small near use annoy and distract?
• Pixel density. Some near guidelines may have to go back to old ones, like no italics, due to low density.
• Perception of no-bezel use. Wall-to-wall or multi-screen interfaces, can user work with close-range control, can they find it, will they stabilize on the screen or not?

References:

• Planar case studies