by: adminPosted on:

Beyond Mobile – Tapworthy’s Josh Clark on the Future

Josh Clark, author of Tapworthy, wowed the crowd at IxDA New York City. Thanks for Bloomberg for hosting and giving a tour of their impressive usability lab. Here are some highlights:

Senors Give Us Superpowers

Mobile should not be the companion – or lite version – of the desktop. ‘Our smartphones and tablets hold so much personal info, it’s the sensors that give rich context and insight to that info and tasks. They have GPS, camera, audio, touch, light detector, accelerometer, compass, a gyroscope. These devices can do more than the desktop.’ We should enhance the data with sensors, but not just because you can – you should do it smartly to fill a need.

Some examples of current apps that take advantage of the sensors:

  • Shopper –  Takes your shopping list and re-orders it on an aisle-by-aisle basis, depending on what store you’re in.
  • Into Now – You “check in” to your TV and it listens to what’s playing and determines which show and episode you’re watching.
  • Skinvaders – Use your face as a gameboard using augmented reality (not avail in US on iOS).
  • Word Lens – Get foreign signs translated by holding up your phone.
  • Layar – Adds an interactive layer to print magazines using augmented reality.
  • Table Drum – Uses the phone as a speaker to allow you to use anything as a drum – You’re free from the tyranny of the screen.
  • AnyTouch – A prototype that allows your environment to become your interface
  • LeapMotion – Controlled using natural gesture – they gave it away to 12K developers to work on apps. It will be bundled with some Asus devices. Intel has a similar technology.

While gesture and speech are in their infancy – “Siri is still marked ‘beta’ by Apple, and alas, it still feels like it” – Gesture and speech together are a magic wand!

Custom Sensors Let Devices Talk to Other Objects

Think social: Share content and control. For example:

  • Square – turns a phone into a credit card reader
  • Proteus Digital Health Feedback System – Patient’s pill has a sensor the size of a grain of sand, with the same stuff you find in a vitamin.
    Copper and magnesium transmit from your stomach acid.
  • Asthmapolis – Clark helped create this, which is a sensor you put on top of your inhaler. It sends data via bluetooth to your phone so you can see where an attack occurred and how well you are controlling your asthma. Epidemiological data is recorded so scientists can track trends.
  • Botanicus Interacticus – An interesting development by Disney to interact with plants using sensors.
  • Anemon – A sensor is attached to a cow and the farmer gets a text when the cow is in heat

Mirroring – Your Device is a Broadcaster

Your device is not only a sensor for input, but a broadcaster. Examples:

  • Airplay – Mirror your screen to show photos/videos with Apple TV. Google, Samsung working on similar initiatives.
  • Smart Kitchens have great promise, but how many interfaces will you have to learn?

Remote Control – Smart Devices Control the Dumb Devices

Clark said “I don’t want a smart TV, smart toaster, smart fridge, each with own OS. Give me a few smart devices, lots of dumb displays.”  Examples:

  • Metal Storm: Wingman – Use your iPad or iPhone to fly the plane on the TV.
  • Microsoft SmartGlass – Control your xbox with any device.

But you need an ecosystem and APIs. But the innovation will come outside of standards – it will be done in private sandboxes first. Clark said, “That’s why Google bought Motorola. That’s why I wouldn’t be surprised if Microsoft buys Nokia. That’s why Amazon makes the Kindle.”

Migrating Interfaces – Primary Shifting Devices Depending on Context

Migrating interfaces allow invisible, ambiguous, shared control. For example:

  • Talk on the phone in your car using your dashboard. When you park, it transfers to the cell phone.
  • Siri integrated in the steering wheel, so you can use it hands-free.
  • Scrabble – Gameboard is on the iPad, the phones hold the tiles.
  • Corning A Day Made of Glass – Corning has a vision of how glass can be used to turn your dashboard into a holodeck or mirror into a cool control panel.

Bill Buxton @wasbuxton at Microsoft Research says “it takes 20 years from the moment an idea is conceived in a lab to when it can go big in the mass market.”

  • Wii-U – Controllers allow you to move the content from place to place.
  • GrabMagic – Clark’s friend Aral Balkan at a hackathon created an app called GrabMagic, which connected a Kinect, Mac, TV, phone for amazing demo of moving data among devices
  • Pinch UI – Pinch interface is used to create apps that make devices react when they’ve been pinched to work together. Open API.
  • Sifteo – Very cool cubes that downloads software on demand to make cubes interact with each other when touched.

Right now, the web is “just in case” – jam-packed with everything you might need. So are apps – there are so many on your phone, you have to weed them out. We need to move to a “just in time” model, where they download software as they need it.

Passive Interfaces – Devices Do The Work

Devices talk to each other, share control, and then they can start doing that on their own. Examples:

  • Nest – Thermostat with proximity sensors to know when you’re home, humidity sensor, temperature, wifi, internet connection to get the weather outside, and talks to apps and websites.
  • Nike FuelBand – Like FitBit, it gives you feedback about your physical activity and talks to your phone/computer. Could a future UI push news to such a dumb display?
  • LUMOback – Just tells you if you need to sit up straighter, but it has an API.
  • HapiFork – Vibrates if you eat too fast.
  • Hue – Philips created a light system where you can control the mood, color.
  • Visa and Mastercard have a system where you type a password onto the card to validate.

In these cases, your API is the application. Your content is chunked up information stripped of presentation, which can be served up through these dumb devices, your app, or website. Since structure is so important, “metadata is the new art direction.” Robots should do the work, such as at The Guardian. They describe data well. The script reads the XML placement and size of the print edition and a robot determines the order of the iPad edition.

A Cloud of Social Devices

We’re building services, presentation deprecates with throwaway interfaces. Content and data lasts. Think about design services. There will be “whole systems that can support lots of devices.” “You are not building an app. You are building a system. Prepare for a multidevice future.”

 

Related articles