Touching People

Final presentation

Camille Moussette — Tue, 15 May 2007 00:07:17 +0100

final presentation [pdf, 11.9 MB]
video files not included

~~preliminary report~~ new version coming soon

New name: HAPI

Camille Moussette — Sun, 13 May 2007 08:35:39 +0100

Say farewell to Touching People. My degree project now has a new name:

HAPI: haptic interaction for mobile devices

The initial name was interesting (in relation to Nokia's Connecting People), but it could also be associated to various meanings that didn't really related to my project. So hence the new HAPI name.

I registered the domain www.hapi-project.com so both addresses now direct to the same site.

NTT DoCoMo's new phones include motion sensors

Camille Moussette — Fri, 27 Apr 2007 22:50:49 +0100

Some of the new FOMA 904i phones now include accelerometers. I couldn't find extensive details on how motion sensing is used for real applications (beside launching apps and playing games). But it's interesting to note that such sensors are finding their way into more phones nowadays. The Nokia 5500 Sport also has 3D accelerometers embedded.

via www.intomobile.com

WHC 11: N73 on steroids

Camille Moussette — Mon, 23 Apr 2007 09:36:40 +0100

This eleventh WHC is a direct refinement of WHC 9 where I was using the 6DOF unit to navigate a simulated menu on the laptop. Using the laptop is very convenient for developing my code. The processing power is ample and I can use various libraries for UI and effects. One thing I realized while testing with the portable computer as a screen is that people don't look at the device while manipulating it. Much like playing with the Nintendo Wii. It's quite normal, you want to see what is going on as you move the portable device. It results in quite large movements or gesture, not really want I intended (and how sensors/parameters are set in the code).

My goal for this WHC was to put the visual feedback (screen) back into the users' hands. I had many options to accomplish this, but I went for the fastest and simplest one: use a real phone in my mock-up. Actually it's not the simplest one, as there are multiple layers to get this working. But the end result is there, things append on the phone as you manipulate it. And all wireless !

The 6DOF unit + battery fitted with the N73

How it works:

The 6DOF unit sends its raw data over Bluetooth to the Powerbook
ImageExpo (a remote-control application for S60 phones) runs on the laptop and takes control of the phone over Bluetooth.
My Processing application receives and analyses the incoming data for the 6DOF.
When a pattern or gesture is recognize by the application, it generates keystrokes. These keystrokes are send via Java (Robot class) to the ImageExpo appliation.
Keystrokes are send to the phone and finally things change on the phone's screen.

There is almost no perceivable delay or lag in this setup, which I find quite amazing. This solution turns out very sucessfull. It was much easier to patch all this together than having to write J2ME code that would need to run natively on the phone and provide the same end result.

Foam core backplane + elastic band. Fancy stuff :-)

The receptacle and the N73

Side view

Testing session, with the powerbook handling the gesture analysis over Bluetooth

Once I was able to remote control the phone via Bluetooth, it was natural to use it with the 6DOF unit instead of using the laptop screen for visual feedback.

Working demo. Not perfect but it is getting there! Gesture let you browse the default contacts list on the N73.

WHC 10: penta-vibrating grip redux!

Camille Moussette — Sun, 22 Apr 2007 01:06:30 +0100

After the quick success of the vibrating duet (WHC 7) it is now time to try out a full 5 nodes version. Actually this project is a direct revisit of one of my first mock-up, the penta-vibrating grip (WHC 2).

Much of this has already been done before, it is more a matter of redoing and repacking the various parts. One nice thing is that my sequencer application works very nicely with this grip (replacing the 5 pistons unit). The Arduino code stays the same also, only the actuator/vibrator relays need to change. For the records, we are now using the regular 4420 IC instead of the 4421 ones. The pistons required inverse signal (no input - output on), where the vibrator motors need direct (no input - output off).

The grip drilled to receive the small vibrators.

Overview and comparison for sizes.

The vibrators glued to their respective disks

The assembled unit with the five vibrating nodes

Covering the disks with a thin foam layer.

The new grip hooked to the Arduino board, ready for action.

Back view. The vibrating motor are mechanically isolated.

Testing the grip. It is quite successful I would say.

Explanation and demo of the new grip.

Surfaces that move

Camille Moussette — Fri, 20 Apr 2007 22:58:32 +0100

How to get tactile feedback through a continuous surface? Is it possible to get good and rich tactile cues without using discrete and physically isolated buttons? To my surprise it seems like it. More and more handset manufacturers are indeed using 'skinned' buttons and keys on their devices, where an outer layer wraps/embeds/hides the physical keys.

This is a quick survey of production phones that uses this idea.

Sony-Ericsson model xxx

Nokia model xxx

Nokia model xxx, side buttons

Sony-Ericsson model xxx, side buttons

Sony-Ericsson model xxx

Samsung model xxx

??? company model xxx

Nokia model xxxx

Nokia model xxxx, side view

WHC 9: 6DOF

Camille Moussette — Thu, 19 Apr 2007 00:36:13 +0100

After playing a bit with the 3-axis accelerometer (WHC 1 and 3), it's now time to get serious with IMUs (Inertial Measurement Units). I am now using the 6DOF v.3 unit from Sparkfun (link). It offers 3 axis-accelerometers (X, Y, Z), 3 gyros (roll, yaw, pitch) and compass headings (X, Y), all packaged with a Bluetooth link. Quite a beast! Check this wikipedia entry to know more about IMUs and how they work.

The goal of this week is to implement some sort of menu navigation (moving along one axis) using small gestures/movements. The tilt-slide interface has already been developed and published before. I am going more for quick hand movement that would allow you to move up (or down) 1 unit, 3 units or go all the way up. This calls for some filtering and peak detection algorithms.

My code in Processing is far from perfect but it fulfills my needs for now. Remember, the goal is not to come up with a production class software. It is more a hardware sketch to be able to test some ideas quickly.

The base is there and hopefully the application and algorithms will be refined to be more precise and recognize more gestures in the coming days and weeks.

The marvelous 6DOF unit from Sparkfun.

Side view of the unit, with 4 AAs battery pack

Close-up on the top board, see the X, Y, Z, Pitch, Roll and Yaw axis. I will be using the pitch axis first.

Graphing the values, and trying to recognize the right gesture

More action shots! Refining the peak detection code.

First working version.

Improved version where I can dynamically adjust the threshold, lock delay and noise values

N73 landed in Ume�

Camille Moussette — Mon, 16 Apr 2007 22:18:46 +0100

What a lovely monday morning. A big box was waiting for me on my desk as I stepped into the studio this morning. Surprise, two Nokia N73 landed in Ume�! Two because my colleague Kristofer also has his degree project supervised by the same tutor as me. So many thanks to Mike Kruzeniski at Nokia for the phones. They are superb. It is bigger than my K750i, but the screen and control keys are much nicer. And it runs the latest Symbian OS with great Java support and Flashlite 2.1. I will definitely put this phone to good use for my degree project. Check back soon for some quick and dirty prototyping with the N73!

WHC 8: poking machine, part 3

Camille Moussette — Wed, 11 Apr 2007 21:52:56 +0100

In this challenge, I decided to migrate the actuators and sensors into some sort of portable devices. I felt that that putting your palm on a desktop device is quite different than holding something in you hand. The results are two matching portable devices: one controller with the light sensors, and one poking device with the electro-mechanical actuators. They now both have 5 nodes and can be remote controlled by the computer running an improved version of the Processing sketch.

Note to me: serial communication is problematic. Incoming buffer data is not handled properly resulting in some sort of inconsistency over time. [fixed now !]

Building the new handheld receptacle for the actuators.

Matching units: actuators on the left, controller on the right.

Soldering new light sensors.

Embedding the sensor in the casing.

Controller unit, with supporting pull-down circuitry for each sensor.

Final working setup.

The matching devices explained.

WHC 7: vibrating duet

Camille Moussette — Mon, 09 Apr 2007 21:06:06 +0100

This week's new Hardware Challenge is very simple. It took only about an hour or so to build. It consists of a cardboard box with two vibrating motors. It is a revisit of the failed penta-vibrating grip (WHC 2). This time, I tried to isolate the vibrating units from each others and minimize their mechanical link to the box/supporting structure. The only thing that keep them together is a thin layer of foam on glued on top.

You know what, it works! It is a very crude experiment/setup but you actually feel if the vibration is coming from the left or the right unit. I have to say that I very happy to have a minimal proof of concept for this idea. It's been puzzling me for months and now I know that vibration can be discerned across the hand. I will definitely move further with this in coming weeks.

Mounting two vibrating motor. Foam block as a cover for rought testing. Not very successful.

We need to isolate the two units. A small cardboard box is used as a support receptacle.

Now each vibrating node is mechanically isolated.

Vibrating motors are upside down. They are linked together with only the outer layer of thin foam.

Final setup after one hour. Buttons to activate the nodes.

Testing successful! We can identify which note is vibrating.

Getting in Touch: Virtual Maps for the Blind

Camille Moussette — Sun, 08 Apr 2007 12:05:05 +0100

Getting in Touch: Virtual Maps for the Blind from Scientific American.

An article on haptic navigation system developed by a group of researchers at Aristotle University of Thessalon�ki in Greece.

zdnet.com also presents the article and adds some notes/references to it: http://blogs.zdnet.com/emergingtech/?p=535

Lemur: the multitouch surface controller

Camille Moussette — Sun, 01 Apr 2007 16:17:43 +0100

This is not totally in line with my degree project but I found it while browsing Processing resources (via Flight 404). So Lemur from the French company JazzMutant is a multitouch controller for live performance. I won't mentioned more since their website is doing a good job of presenting and explaining the device.

As an intro you can watch this video

Interfaces Supporting One-Handed Use of Small Devices

Camille Moussette — Sun, 01 Apr 2007 15:38:32 +0100

Mobile phones today usually support one-handed interaction. You can use two hands to speed up some tasks but most operations can be completed using only one finger or hand. PDAs on the other hand, usually involve two-handed interaction with their touchscreen and point-click actions. Generally one hand is used to hold the device as the other one act as a controller (actuating, moving and pushing buttons).

It is a sure bet that we will see a lot more touch screens appearing in mobile phones over the next few years (iPhone and such). Sure they have some advantages (customizable interface) but also present some challenges. I won't go into details here but I just want to mentioned this nice research done by the HCI group at the University of Maryland. Their work was presented at CHI 2005 and it involves developing interfaces to support one-handed use of small devices.

http://www.cs.umd.edu/hcil/mobile

Immersion: engaging the sense of touch

Camille Moussette — Sun, 01 Apr 2007 11:53:47 +0100

Immersion is probably the biggest name in the haptic technologies market. They have been around since 1993 and hold +600 patents for various haptic systems. Most of their products and applications are targeted at businesses and large corporations (due to the high cost), but new offerings are finding their way into mainstream market.

Their range of products is quite impressive: from automotive systems to medical and gaming applications. Mobile devices are also under their portfolio with the VibeTonz system. At first sight, it looks like personalized tactile ring-tones, but I guess the possibilities are much broader than this. I never encountered such device or demo unit, but I heard some report that it is quite convincing and believable.

If you don't mind cheesy music and an over-excited host, check this 9 min. Flash video from Tech Closeup. It showcases some of the Immersion products and technologies.

The Immersion website also offers good information on haptics:

WHC 6: poking machine, part 2

Camille Moussette — Sat, 31 Mar 2007 23:27:10 +0100

Over the last few days, I have been refining the poking machine with various components. First I completed an analog controller to trigger the actuators. It is composed of simple photo-resistors that are sampled by the Arduino board. Getting my hands on more photo-resistors (I had only two in my stock) turned out to be a real challenge. It took me more than four hours of intense research to browse and find a store that would sell and ship them to Sweden. They finally were delivered pretty fast (next day) from Elfa with a custom order number. The idea of the board is to act as input device to control and record sequences for the array of actuators.

The second thing I did this week was to develop an application to control the actuators. This component is essential if I want to test and evaluate various patterns or sequences with users. It makes recalling and replicating the sequences over time possible and almost enjoyable. I am quite happy with the software I was able to complete in 2 days of work. I still have to code the import/export module for xml config files. Nevertheless it working good, with click and drag functionalities. Processing talks to the Arduino board via serial communication, both for controlling the actuators and polling the sensors for the wooden board.

The next step is definitely testing. I have to get out of my corner and go to the users and see how they understand and feel the various signals. I still have to come up with some sort of plan of action before asking users to put their hand on the device.

the so difficult to find photo resistors. the snapshot is for archival purpose if I need to order again, but I should have enough with these 30 units.

components fitted to the wood board (I should have drilled an extra shot to make them flush, on the to-do list for next week)

sensors are polled to control the solenoids

pull-down circuitry for the photo resistors

software sequencer: screen interface to manage the triggers and sequence's speed. developped in Processing

an overview of the poking actuators controlled by the computer via the Arduino board

the software component to modify the sequence's parameters/triggers. only three tracks are used now.