Haptic Gaming System

Overview

Haptikos is an interactive gaming system that allows the user to physically feel a virtual world. The player controls the cursor on screen by moving the red joystick attached to the linkage system. Two games were created to highlight the capabilities of the feedback system. The first involves feeling a virtual shape displayed on the screen (the display can be toggled off if you want more of a challenge). This game is a simple demonstration of translating a virtual environment into something physical. The second is a side-scrolling level where the player collects jewels while avoiding boxes. This shows possible applications to an interactive gaming environment, where the user can directly feel what is happening to their character on screen. However, using our platform, all it takes is a creative mind and some programming knowledge to develop a wide variety of other games/demonstrations.

Watch Haptikos (and the other projects) in action on YouTube.

The Team

Yang Bai (Mechatronics Masters Student)
Philip Dames (Mechanical Engineering BS/MS)
Megan Welker(Mechanical Engineering BS/BM)

Mechanical Design

Existing System

Haptikos began as a modification of the pantograph used by John Glassmire for finger position tracking for his M.S. thesis work in friction-based tactile displays. This design allows the free point to move in a (roughly) semi-circular area in a plane. Two optical encoders are mounted at the shoulder joints to track the angle of the upper arm links (and thus the finger).

System Modifications

The existing system had the capability to accurately track finger position, but could not put apply any sort of haptic feedback. To add this capability, we chose to use a capstan drive system. In such a system, there is a thread spooled onto the motor shaft that winds in and out depending on how the motor is spinning. This thread is mounted to another element that is pulled back and forth as the thread spools out. Such drive systems can be found in other haptic devices (such as these haptic paddles) and are chosen because they increase the torque of a motor as a gearhead does, but avoid problems such as backlash which is a major concern when the goal is to provide smoothly changing forces to a user. The motors chosen for out project were the Pittman 700935 motors, which are relatively powerful motors without a gearhead.

The arc of the capstan drive was created by milling two 2.75" radius arcs out of Aluminum. A piece of heavy duty thread was set into the groove of the outer side of the arc and secured by screws on either side. To connect the capstan to the motor, the thread was tightly wound around the spool (3/8" diameter brass cylinder, mounted with a set screw) three times and pulled taut in the groove. The spool was covered with a plastic layer to increase the coefficient of friction between the spool and the thread. The gear ratio created by our drive system is 44 : 3 ( $=r_{arc}/r_{spool}$ ).

Pantograph.

Arc.

Thread and Motor attachment.

Box

To enclose the entire system, we made a 12" x 12" x 3" acrylic box using the laser cutter. The top plate is removable so that the circuitry is readily accessible. The backboard of the pantograph is screwed into the bottom. The motors were held in place by a set of holes in the top plate and a second bracket suspended between the top and bottom. Holes were cut in the side and back of the box to allow wires to pass in and out of the box.

Electrical Design

Block diagram of circuit.

Circuitry.

PIC Configuration

We used the PIC32MX460F512L from Microchip on the NU32 board. The board was powered using a USB cable plugging into the socket on the left side of the board. The ribbon cable on the far right of the breadboard connects to the breakout board and has four I2C lines, four lines from the decoding circuit, +5V, and ground. The cable next to it is the RS-232 that connects the PIC to the PC.

Breakout Board

This board serves as a central hub for all of the other components to the circuit, allowing for only one ribbon cable to connect to the PIC's breadboard. Clockwise from the green power supply board, we have connections to the two power amplification circuits, the PIC breadboard, the decoding circuit, and the DAC/level shifting circuit. The power supply provides a separate +5V, -5V, and ground line to power the motors, protecting the PIC from the high amperage needed to drive the motors. In the center of the board is a Cincon EC2A05 DC/DC converter that takes in the +/- 5V lines and outputs +/- 15V used for the level shifting circuit.

DAC/Level Shifter Board

This board takes the force data sent over I2C from the PIC (from the PC) and creates an analog voltage signal ranging from +/- 5V that feeds into the power amplification circuit. Here, we use a MAX518 DAC chip that connects to a level shifting circuit using two LT1001 op-amps, as shown in the circuit diagram. This is necessary to allow the motors to spin in both directions without the use of PWM (which we opted not to use because we want torque, and thus, current control instead of speed control). Note the actual board consists of two identical channels, one for each motor, while the schematic shows only one.

Motor Control Board

This circuit uses current feedback in order to maintain a constant output current, allowing for direct torque control of the motors. This circuit was designed by Prof. Michael Peshkin and detailed information can be found here. Two of these boards were used, one for each motor with the input coming from the DAC board and power from the central breakout board.

Decoder Board

The decoding circuit reads the signals from two optical encoders them into up and down counts (two each) that connect to the timer pins on the PIC. The encoders used are Canon TR-1 (equivalent to TR-36) which are 3600 counts per revolution using x1 decoding. The decoding circuit is that used in the Recommended Receiving Circuit link at the bottom of the page for the TR-36, where the channels first feed into a SN74HC14N Schmitt trigger which cleans up the signals going into the LS7083 quadrature decoding chips. These take the pulse trains and translate them into up and down counts which are then sent over to the PIC and read in on the timer pins.

Mathematical Analysis of the Pantograph

There are two parts to this problem, the forward kinematics and the force generation, which are duals of each other.

Forward Kinematics

The forward kinematics of a system (typically a robotic arm or something similar) is often used in robotics and gaming to calculate the position and orientation of a hand given the joint angles of an arm. The idea is to create a function such that $x=f(\theta )$ where $x$ is a vector giving the location of the finger and $\theta$ is a vector of the joint angles (here the angles read by the encoders). For our system, the pantograph used was originally designed and build by John Glassmire (formerly of the Laboratory of Intelligent Mechanical Systems. Dimensions as well as full calculations of the forward kinematics may be found in Appendix D and Solidworks drawings of the parts found in Appendix E of the paper found at [1]. Encoder 1 is found at angle 1, encoder 2 at angle 5, and the finger at angle 3, with arrows indicating the direction of positive motion, note that angle 5 is backwards to provide reflectional symmetry about the vertical axis.

Force Generation

Obviously, the wider the range of forces we can create, both in terms of magnitude and direction, the better sensation we are able to create with Haptikos. However, we also want to transition between forces in such a way that feels natural to the user. These are fundamental issue addressed by designers for any haptic device. There is a limit to the torques our motors can provide (current supply to each motor is limited to 500 mA), and thus on the strength of the forces we can give to the user and the resolution of these forces was also limited by communication issues (as discussed later).

We independently confirmed the forward kinematic calculations, and through a series of substitutions, put them all into a single function $x=f(\theta )$ . Taking the derivative of this mapping, we have that ${\dot {x}}={\frac {df}{d\theta }}{\dot {\theta }}=J(\theta ){\dot {\theta }}$ , where $J(\theta )$ is the Jacobian of the map. By the duality of velocity and force, we have that the torques required to produce a given force can be found using the relation $\tau =J(\theta )^{T}F$ . For our map, the Jacobian is complicated and would take up nearly a page so it is omitted for brevity. However, we were unsuccessful using this approach to generate forces in arbitrary directions. This may be due to many causes, from data types conflicting to omitting a sign or term when typing out the equations.

Instead we used simple controls to achieve forcing in eight directions. A single byte of data was sent to the PIC containing the motor control information, with the first nibble to control motor 1 and the second to control motor 2. This allows for 16 levels of forcing, with 7 chosen to indicate zero force. Less than 7 provided a torque along the positive direction for each angle (as indicated in the figure) and greater than 7 along the negative direction. We found that activating both motors allowed us to create forces along the y and x axis (depending on if the motors were providing a torque in the same, or opposite, directions, respectively) while only activating a single motor provided forcing along the lines y = x and y = -x (motor 2 or motor 1, respectively). This range of 16 discrete force values was mapped over half of the possible force range because we found that having forces that were too strong caused for large jumps when the forces changed direction while smaller forces allowed for smoother-feeling transitions, at the cost of a more rigid feeling to surfaces.

Software

PIC Programing

All PIC programming was done in C using the MPLAB IDE by Microchip Technologies. To install MPLAB, follow the instructions found here. To create a new project in MPLAB, follow the instructions found here. The source files for our project can be downloaded here (remember to change the include directions if you are trying to use our MPLAB project files).

While the source code is commented, here is a list of the project-specific files and a brief summary of the contents of each.

fingerTrack.c - This is the main file of the project, containing the main function as well as initialization and communication functions
- initEncoder - This initializes the 4 timers (T2CK - T5CK) used to count the up and down pulses from the decoding circuit. Timers 2 and 3 are for the up and down counts of encoder 1, respectively. Timers 4 and 5 are used for up and down counts of encoder 2.
- getEncoderPositionX - This reads the current counts of encoder X and updates the angles, checking for rollover of the 16-bit counters.
- initUART1 - This initializes the UART1 bus of the PIC to be used for RS-232 communication with the PC.
- initInterruptComm - This initializes the Timer1 interrupt to run every 10ms and initializes both I2C buses on the PIC to talk to the DAC's. The interrupt timing could be synched to match the frame rate of the game, but we found this to be reliable and not interfere with the data transmission interrupt.
- MAX518 - This takes the data sent from the PC about how to force the system, parses the data, and sends out the appropriate signal to the DAC's over the I2C buses. Only 1 byte of data is sent between the PC and the PIC as this was found to be the most reliable, though it does limit the resolution of forces we are able to achieve (16 levels instead of 256). We did not find this to be a problem, but is one area of possible improvement.
- UART1 interrupt - When it detects that it has received data from the PC, the PIC will read the data, call the MAX518 function, and toggle LED0 for visual feedback. We had some issues with the interrupt not functioning correctly after the game was reset after a game over (this did not happen every time).
- Timer1 interrupt - This sends the current (x,y) position of the finger as well as the two angles of the encoders to the PC.
calculations.c (and .h) - This library has the functions used to calculate the forward kinematics of the pantograph (i.e., to get finger position from the encoder readings).
- sine and cosine - These functions take in the current encoder angle (measured to the nearest tenth of a degree) as an integer and use a lookup table to calculate the sine. The output is an integer which must be divided by 10000 (the global variable SCALER). This was done to speed up calculations as it only involves integer math instead of floats, and can directly use encoder counts instead of converting these to radians.
- forwardKinematics - This calculates the current (x,y) location of the finger based on the current encoder readings.

RS-232 Communication with PC

We found that reliable communication between the different components of our system was one of the more challenging aspects of the project. Because of this, we decided to post our code here as a quick reference instead of downloading the code from above. The desired baud rates for serial (DESIRED_BAUDRATE) and I2C (BRG_VAL) communication were defined at the beginning of the code to be

#define DESIRED_BAUDRATE       19200      // The desired BaudRate RS-232
#define Fsck   		50000      // Frequency
#define BRG_VAL    		((SYS_FREQ/2/Fsck)-2)   // Baud rate generator value

This first function initializes the UART1 port on the PIC. For an more detailed explanation of the configuration variables, and other UART functions, see chapter 16 of the Peripheral Library Guide.

void initUART1(int pbClk)
{
    // define setup Configuration 1 for OpenUARTx
    // Module Enable 
    // Work in IDLE mode 
    // Communication through usual pins 
    // Disable wake-up 
    // Loop back disabled 
    // Input to Capture module from ICx pin 
    // no parity 8 bit 
    // 1 stop bit 
    // IRDA encoder and decoder disabled 
    // CTS and RTS pins are disabled 
    // UxRX idle state is '1' 
    // 16x baud clock - normal speed
    #define config1    UART_EN | UART_IDLE_CON | UART_RX_TX | UART_DIS_WAKE | UART_DIS_LOOPBACK | UART_DIS_ABAUD | UART_NO_PAR_8BIT | UART_1STOPBIT | UART_IRDA_DIS | UART_DIS_BCLK_CTS_RTS| UART_NORMAL_RX | UART_BRGH_SIXTEEN
   
    // define setup Configuration 2 for OpenUARTx
    // IrDA encoded UxTX idle state is '0'
    // Enable UxRX pin
    // Enable UxTX pin
    // Interrupt on transfer of every character to TSR 
    // Interrupt on every char received
    // Disable 9-bit address detect
    // Rx Buffer Over run status bit clear
    #define config2      UART_TX_PIN_LOW | UART_RX_ENABLE | UART_TX_ENABLE | UART_INT_TX | UART_INT_RX_CHAR | UART_ADR_DETECT_DIS | UART_RX_OVERRUN_CLEAR   
 
    // Open UART2 with config1 and config2
    OpenUART1( config1, config2, SYS_FREQ/16/DESIRED_BAUDRATE-1);   // calculate actual BAUD generate value.
       
    // Configure UART2 RX Interrupt with priority 2
    ConfigIntUART1(UART_INT_PR2 | UART_RX_INT_EN);
}

The next initialization function configures Timer1 which is used to generate an interrupt every 10 ms that sends location data to the PC. The timer is configured to add one count every 64 clock cycles of the PIC with a priority of 7 (the highest value) as the feedback to the computer is important for rendering the game correctly. Because this is higher priority, it may be necessary to increase the period of the interrupt if the PIC is receiving longer packets of information from the PC. This function also initializes the I2C communication ports (further explanation of this below).

void initInterruptComm()
{
   // Initialize timer 1 to generate interrupt to send data to PC
   // init Timer1 mode and period (PR1) // produces 10ms period
   OpenTimer1( T1_ON | T1_PS_1_64 | T1_SOURCE_INT, 12499); 
     
   mT1SetIntPriority(7);    // set Timer1 Interrupt Priority
   mT1ClearIntFlag();       // clear interrupt flag
   mT1IntEnable(1);         // enable timer1 interrupts

   // Initialize I2C lines to communicate data to DACs
   OpenI2C1( I2C_EN, BRG_VAL );
   OpenI2C2( I2C_EN, BRG_VAL );
}

Then we create an interrupt handler for UART1 which is triggered when the PIC detects that there is data in the serial buffer. This is set to a priority level of 2 because it is less important than sending location information to the PC. The function ReadUART1 retrieves a single character from the serial buffer; use getsUART1() for strings.

// UART 1 interrupt handler
// it is set at priority level 2
void __ISR(_UART1_VECTOR, ipl2) IntUart1Handler(void)
{
   unsigned char force;

   // Is this an RX interrupt?
   if(mU1RXGetIntFlag()) {
        // Clear the RX interrupt Flag
        mU1RXClearIntFlag();
   
        force = ReadUART1(); // get data from PC
        mLED_0_Toggle(); // toggle LED for feedback
        MAX518(force); // send signal to motors
   }

   // We don't care about TX interrupt
   if ( mU1TXGetIntFlag() ) {
        mU1TXClearIntFlag();
   }
}

We also create an interrupt controller for Timer1 to send position information back at the pre-defined rate of 10ms. The function uses putsUART1 which sends a string of data to the PC, which then parses the string into the individual components. The end line character "\n" is useful so the PC knows that it has received the entire data string, though any character could be used.

// interrput code for the timer 1
void __ISR( _TIMER_1_VECTOR, ipl7) T1Interrupt( void)
{
    // send data to PC every 10 ms (based on Timer1)
    char RS232_Out_Buffer[32];
    sprintf(RS232_Out_Buffer,"%d %d %d %d\n",X, Y, phi1, phi2);
    putsUART1(RS232_Out_Buffer);

    mLED_3_Toggle(); // toggle LED for feedback
    
    // clear interrupt flag and exit
    mT1ClearIntFlag();
} // T1 Interrupt

I2C Communication with DACs

The two I2C ports were initialized above in the initInterruptComm() function. The I2C communication also uses functions from the Peripheral Library Guide, found in chapter 15. The general structure of the function is to send a series of commands to the DAC and wait for this to complete after each command. First, the Start command is sent, then Idle waits until the I2C line is no longer busy. Then the address of the chip on the I2C line is sent out and the PIC waits until the chip has acknowledged (ACK-ed). This address is defined by the chips used as well as the configuration pins set when wiring the chips. It then sends the command byte to tell the DAC to get ready to output data and finally it sends the data byte telling the DAC the voltage level to output. The I2C line is then closed to avoid sending extraneous data and the process is repeated for the second DAC. This information will change depending on the device being interfaced with (particularly if the PIC has to switch from master to slave mode) but the basic principles outlined here can be used for most applications.

void MAX518(int force)
{
    unsigned char addr = 0b01011000; // address of DAC on I2C line
    unsigned char cmd = 0x00; //command line 
    int force1 = 127+(7-(force >> 4))*8; // upper nibble to motor 1
    int force2 = 127+(7-(force & 15))*8; // lower nibble to motor 2
   
    // Send to DAC 1 //////////////////////
    StartI2C1();   //Send the Start Bit (begin of data send) 
    IdleI2C1();    //Wait to complete 

    MasterWriteI2C1 (addr); //address 
    IdleI2C1(); 
    while( !I2C1STATbits.ACKSTAT==0 ) {}

    MasterWriteI2C1 (cmd); //command line 
    IdleI2C1(); 
    while( !I2C1STATbits.ACKSTAT==0 ) {}

    MasterWriteI2C1(force1); //output 
    IdleI2C1(); 
    while( !I2C1STATbits.ACKSTAT==0 ) {}

    StopI2C1();  //end of data send 
    IdleI2C1();  //Wait to complete 


    // Send to DAC 2 //////////////////////
    StartI2C2();   //Send the Start Bit (begin of data send) 
    IdleI2C2();    //Wait to complete 

    MasterWriteI2C2 (addr); //address 
    IdleI2C2(); 
    while( !I2C2STATbits.ACKSTAT==0 ) {}
      
    MasterWriteI2C2 (cmd); //command line 
    IdleI2C2(); 
    while( !I2C2STATbits.ACKSTAT==0 ) {}

    MasterWriteI2C2(force2); //output 
    IdleI2C2(); 
    while( !I2C2STATbits.ACKSTAT==0 ) {}

    StopI2C2();  //end of data send 
    IdleI2C2();  //Wait to complete 
}

Game Programming

The games are programmed and run in Processing, a Java-based programming language that makes the creation of graphics very easy. We chose to make two games that highlight different aspects of our project, though as you can imagine, our system could be used for many more.

Link's Adventure

This game is a side-scrolling adventure game where the player must avoid the blocks and collect the jewels scrolling across the screen. As the player icon, we chose to use the character Link from the original Legend of Zelda game due to the simple graphics and game style. The three types of jewels each have a different effect on the game. Red replenish health lost from hitting the boxes, green increase the score, and blue slow blocks (whose speed gradually increases over time). When Link runs into a red block, Haptikos will shake the joystick as the box disappears to give the user a physical indication that their player was injured, much like the rumble feature found in many modern game console controllers. While the user has the option of going for a high score, we have found that it is generally more enjoyable to simply run into the blocks as this is the unique part of our device. Future iterations of this game would include some sort of haptic feedback when a jewel is collected to the player strikes the ground, possibly even simulating gravity by pushing the character towards the ground. Commented source code can be found here and code allowing the game to be played with a computer mouse here.

Identification Game

In this game the player will feel a virtual shape (circle, star, triangle, square, or diamond) in the center of the screen. When the cursor (blue circle) hits the shape, the device will push the finger away from the shape, creating the sensation that you are feeling a wall. The player can also play blindly, using only the feedback from Haptikos to attempt to identify the invisible shape. To toggle this mode, press the 'i' key on the keyboard. Press 'r' to generate a new shape at random. Currently there are only 5 shapes, chosen because they are different enough so that the user can identify the shape (i.e., a heptagon and an octagon would be too similar to identify using the discrete forces available to us). Commented source code can be found here.

Debugging Tips

While working on the project, as with all mechatronics projects, we ran into many small problems. We will highlight some of the most common ones so that you don't make the same mistakes that we did and can spend more time improving, rather than fixing, your project.

ALWAYS CHECK YOUR WIRING - When using a breadboard wires often get knocked loose, break off, etc. The thing we spent the most time on was trying to debug circuits where a wire had simply come out of the breadboard or that we had put back in the wrong place. Don't waste your time reading the I2C signals on an oscilloscope to make sure that the data is correct only to find out you accidentally wired the data line to the clock pin of your chip. And remember that the multimeter is your friend; when dealing with lots of ribbon cables it is invaluable for checking if pins are wired correctly.
Use the oscilloscope - While the multimeter is good for checking some things, it missed quick voltage changes that a scope will let you see. We used this for 95% of our debugging, from checking power lines to reading the individual bits being sent over I2C or RS-232 lines.
Check your soldering - Make sure the solder on any solder board, PCB's, or the PIC board is done correctly. This tends to wear out over time and can break loose/come into contact with other connections creating shorts. If your circuit is not working, check this as it only requires a short visual inspection but can save hours of headache from changing software and tracing wires all over your circuit boards.
Remember how your circuit is supposed to work - After we built the power amp PCBs we were testing them to make sure they work. Since they are current control circuits, the motor + and motor - pins must have current flowing between them or the output voltage will float. We forgot to put a power resistor between these pins and spent a long time trying to figure out why the output was not changing before we finally realized it actually did work exactly as it was designed to.
Use sockets when moving your circuit to solder board - If a chip burns out and you have soldered it directly to the board, it takes a long time to remove. Use sockets instead as this makes changing chips very easy. The same goes for data lines, use pin headers and ribbon cables instead of soldering your cables directly to the board. This also makes moving the circuit much simpler as it can be disassembled.
Common ground - When connecting several circuit boards together, make sure they have a common ground. If this is not done, the ground from one board can float relative to the other boards and cause problems when sending signals between boards.
Current limitations - When using DC/DC converters, exceeding the maximum current may cause the converter voltage output to decrease.
Impedance matching - As with all modular circuitry, make sure that the output impedance of one circuit is matched to the input of the connected circuit.

Next Steps

There are several areas for improvement in our project, the primary one being to improve the haptic feedback by allowing forces to be generated in arbitrary directions and increasing the resolution of forces we can generate. The first half of this involves debugging the Jacobian calculations and ensuring that all data types in the calculations are compatible (i.e., multiply ints by ints instead of floats). For the second half of this, we need more reliable serial communication from the PC to the PIC. While we were successful sending a single byte of data, when trying to send a string of data (2 or 4 bytes) the PIC would often, though not not always, fail to receive the data. Fixing this, the full capabilities of the DAC could be used to have 256 levels of forcing instead of the 16 we used. This would be helpful when trying to simulate textures, compliant objects, stickiness (damping), etc, all of which should be possible with a device such as Haptikos. This could be a benefit to games, such as feeling more resistance when your character walks through a patch of mud instead of over grass.

There are also a couple of improvements for the mechanical design. The Pittman motors used had a "clickiness" to them so as the user moved the pantograph around they felt slightly bumps in the motion, even when the motors were unpowered. While this may actually be a benefit for some applications (i.e., simulating rough surfaces), in general we would recommend choosing motors with a smoother feel to them. Also, when mounting the capstan drive system, take care to properly align all parts. Not centering the arcs properly can lead to increased resistance and can limit the size of the workspace. We ran into this issue with the initial build but fixed it for the final design.