Skip to content

Add Gaze Interaction to iOS

Introduction to Gaze interaction

Using a person's point of gaze as a means of input is probably the most widely used application of eye tracking. In this tutorial, we'll go through how to make your interface eye controlled with dwell-based gaze interaction.

Of all the different types of gaze interaction, dwell is normally the one used. The basic principle is that the point of gaze becomes a cursor, as known from regular computer interfaces. Instead of clicking with a physical button, the user "dwells", or fixes their gaze on the element they want to interact with for a certain amount of time - typically one second. The system usually gives feedback on how far an interaction is.

demonstration of gaze interaction

Above is a simple implementation of gaze interaction in iOS. The white cirlce indicates the point of gaze. As the gaze transfers the screen, interactable buttons are highlighted. If the user keeps their gaze on a highlighted element, a yellow border indicates how far a user is from activating the button. When the border reaches 100%, an action is performed. In this case, the button turns green.

From the animation, it is possible to see the other advantages of eye tracking. By tracking the gaze, we can actually infer the intention of the user before any action has happened. We also know if, and where the user has their attention on the screen. Coupled with the interaction, this simple interface gives a powerful insight to the user's thought process, which we can help us understand user behaviour.

Limitations and Challenges of Gaze Interaction

As all other user interfaces, eye tracking has weaknesses and pitfalls. These can usually be solved with clever design, or by simply being aware of them. For more in-depth explanation of what might cause low accuracy or errors, see troubleshooting.

Midas Touch Problem

Midas Touch is a well-known problem in eye tracking. It stems from the legend of King Midas, who turned everything he touched to gold. The same problem occurs in eye tracking (but with less gold): How can avoid interacting with everything we look at? A simple way to avoid this problem is to simply increase the time we have to look at a button (dwell time). But that increases interaction time, and strain on the eyes. In our experience, user's with experience in eye tracking interfaces can avoid the midas touch problem with skill - they learn where to look, and can control their gaze in a manner that doesn't interfere with the system. The midas touch problem should be a major concern when you design gaze interfaces.

Speed and Accuracy

Speed is a major issue when dealing with user interfaces. Research shows that direct manipulation of object on a screen should take 0.1 second or less - far lower than the recommended 1 second for dwell time. We recommend that dwell-based gaze interaction as the only means of input should only be a viable solution for people with no other options of input. For example:

  • People with severe motor disabilities,
  • Use cases where the user has no free hands available
  • Situations where contact-free interaction is essential

For use cases where speed is essential, we recommend combining gaze interaction with other means of selection - for example touch or keyboard interaction.

Other sources of inaccuracy is typically linked to Calibration Deterioation and Environmental Parameters which you can read more about by following the links.

Set up Gaze Interaction

It's time to set up some basic gaze interaction. You need to have calibrated the system in order for the gaze tracking to work.

Project Setup

By following the previous guides, you should have a ViewController which displays a SPCalibrationViewController on startup, and removes it when the calibration has completed through the didFinishCalibration(_) delegate method. We'll use that method to display a UICollectionViewController with cells that can be interacted with through gaze. Start by creatining a new swift file called GazeInteractionViewController.swift, and create a view controller called GazeInteractionViewController:

import Foundation
import UIKit
import Obital_iOS

class GazeInteractionViewController: UIViewController {
    var sessionManager: SessionManager?

    override func viewDidLoad() {
        super.viewDidLoad()
    }

    override func viewDidLayoutSubviews() {
        super.viewDidLayoutSubviews()
    }

    override func viewWillDisappear(_ animated: Bool) {
        super.viewWillDisappear(animated)
    }
}

Notice the SessionManager object - we'll need that later.

Layout the Collection View

Time to add the collection view. There's nothing new here - this is just how you would usually add a collection view in swift through storyboard. Open the storyboard and add a new view controller, assigning the GazeInteractionViewController class to it. Give it the storybard ID "gazeInteractionViewController".

Create View Controller

Add a UICollectionView to the GazeInteractionViewController and make it fill the superview using autolayout.

Auto Layout

Collection View Constraints

Select the UICollectionViewCell in the controller, and give it the identifier "cell"

Set the Data Source for the Collection View

Add an outlet from the UICollectionView to the view controller, and make the controller a delegate of the collection view:

    @IBOutlet var collectionView: UICollectionView!

    override func viewDidLoad() {
        super.viewDidLoad()
        self.collectionView.delegate = self
        self.collectionView.dataSource = self
    }

Add the extensions to conform to the delegates. We'll add them now just to have them, but we still need to configure each method to make it functional.

extension GazeInteractionViewController: UICollectionViewDelegate {

    // Called when the cell is selected
    func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath) {
        guard let cell = collectionView.cellForItem(at: indexPath) else {return}
    }

    // Called when the cell is highlighted
    func collectionView(_ collectionView: UICollectionView, didHighlightItemAt indexPath: IndexPath) {
    }

    // Called when the cell is un-highlighted
    func collectionView(_ collectionView: UICollectionView, didUnhighlightItemAt indexPath: IndexPath) {
    }
}

extension GazeInteractionViewController: UICollectionViewDataSource {
    // The number of cells we're displaying
    func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
        return 4
    }

    // Configure the cell
    func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
        // Create cell
        let cell = collectionView.dequeueReusableCell(withReuseIdentifier: "cell", for: indexPath)
        // Set the background color, so we can see it.
        cell.contentView.backgroundColor = .yellow
    }
}

If you set GazeInteractionViewController as the storyboard entry point and run the project, you'll see four small yellow squares on top of the screen. We'll have to make them a bit bigger to make them gaze controllable. Add an extension conforming the controller to UICollectionViewDelegateFlowLayout and configure the size of the cells to create a stack of four cells filling the full screen width:

extension GazeInteractionViewController: UICollectionViewDelegateFlowLayout {
    func collectionView(_ collectionView: UICollectionView,
                        layout collectionViewLayout: UICollectionViewLayout,
                        sizeForItemAt indexPath: IndexPath) -> CGSize {
        // Margin around the cells
        let cellMarginSize: CGFloat = 16
        // Calculate width. Fills the entire screen width minus margins.
        let width = self.view.frame.size.width - cellMarginSize * 2
        let margin = cellMarginSize * 2
        // Calculate height. Each cell fills 1/4 of screen height minus margins
        let numberOfItems = self.collectionView(collectionView, numberOfItemsInSection: 0)
        let screenHeight = (self.view.frame.size.height - cellMarginSize * CGFloat(numberOfItems) - margin * 2)
        // The cells should just fill the height out
        let height = screenHeight / CGFloat(numberOfItems)
        return CGSize(width: width, height: height)
    }
}

Running the app now should produce a vertical stack of yellow cells, filling the entire screen. Now we need to make them interactable!

Connect the Eye Tracker

We need to make sure that the view is only loaded after calibration. Go back to the main view controller, where you calibrate the eye tracker. In the bottom of the didFinishCalibration(_), replace the code displaying GazePointDotViewController, to instead display your new CollectionViewController. Notice that we need to instantiate it from the storyboard, since the UICollectionView is added through there.

extension ViewController: SPCalibrationViewControllerDelegate {
    func didFinishCalibration(_ quality: NSMutableArray) {
        remove(vc: spCalibrationViewController)
        // Remove old code first.
        // Get the storyboard
        let storyboard = UIStoryboard(name: "Main", bundle: nil)
        // Instantiate you view controller
        guard let gazeInteractionViewController = storyboard.instantiateViewController(withIdentifier: "gazeInteractionViewController") as? GazeInteractionViewController else {return}
        // Add it to the stack.
        add(vc: gazeInteractionViewController)
    }
}

Great! After calibration, your view controller will be displayed. Notice how we injected the sessionManager into our view controller - we need that to create new gaze-based view controllers.

Everything we've done so far has been basic swift. Now it's time to add some eye tracking magic!

In the bottom of viewDidLoad() in GazeInteractionViewController, create a gaze point view:

override func viewDidLoad() {
    // ... code above here
    let gazePointViewController = GazePointDotViewController(sessionManager: sessionManager!)
    // Remember to use the add and remove extensions we created earlier
    add(vc: gazePointDotViewController)
}

You should now be able to see the calibrated gaze point in your collection view.

Adding Gaze Interaction

To make the gaze interface, we need to:

  1. Register the gaze point on the screen, and notify each cell if they are being looked at.
  2. React to attention by highlighting the cell
  3. Run a countdown for dwell time, and visualize it in the cells
  4. React to selection if the dwell time is reached.

Register Views to Gaze Point Changes

To make UIKit based interfaces gaze interactable, Obital manages it's own view class called GazeInteractionView. By adding this view to any existing UI element, and conforming the element to GazeInteractionViewDelegate, you can make any view gaze interactable. Luckily, Obital already comes with default conforms for the most normal UI element. It works by conforming the default class to GazeInteractionViewDelegate, and calling the similar methods that would be called on a regular touch event:

Class didHighlight cancelHighlight didActivate
UIButton sendActions(for: .touchDown) sendActions(for: .touchCancel) sendActions(for: .touchUpInside)
UICollectionViewDelegate collectionView(collectionView: UICollectionView, didGazeHighlightItemAt: IndexPath) collectionView(collectionView: UICollectionView, didGazeUnhighlightItemAt: IndexPath) didSelectItemAt: indexPath

You can modify this behaviour, or completely disable them, by overriding the delegate methods in each class.

Since we're working with UICollectionViewCells, all we need to do is to add the gaze view using another method supplied by Obital, addGazeView (available in the same classes as the delegate methods in the table). Do so in the cellForItemAt in your GazeInteractionViewController:

//GazeInteractionViewController.swift

// Configure the cell
func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
    // Create cell
    let cell = collectionView.dequeueReusableCell(withReuseIdentifier: "cell",for: indexPath)
    cell.contentView.backgroundColor = .yellow
    cell.addGazeView()
    return cell
}

This adds an invisible view with the exact size of the cell to the cell, that notifies it's superview when any change in gaze i registered.

Reacting to Changes

You can identify what methods you need to use from the table above. Since we're using UICollectionViewCells, just use the delegate methods from UICollectionViewDelegate:

extension GazeInteractionViewController: UICollectionViewDelegate {

    // Called when the cell is selected
    func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath) {
        guard let cell = collectionView.cellForItem(at: indexPath) else {return}
        cell.contentView.tintColor = cell.contentView.tintColor == .yellow ? .blue : .yellow
    }

    // Called when the cell is highlighted
    func collectionView(_ collectionView: UICollectionView, didGazeHighlightItemAt indexPath: IndexPath) {
        guard let cell = collectionView.cellForItem(at: indexPath) else {return}
        cell.contentView.backgroundColor = cell.contentView.tintColor.withAlphaComponent(0.5)
    }

    // Called when the cell is un-highlighted
    func collectionView(_ collectionView: UICollectionView, didGazeUnhighlightItemAt indexPath: IndexPath) {
        guard let cell = collectionView.cellForItem(at: indexPath) else {return}
        cell.contentView.backgroundColor = cell.contentView.tintColor
    }
}

This is what happens when running the code:

  1. If a cell is looked on, it will become brighter.
  2. If the gaze is moved away from the cell, it will change color to its default.
  3. If it is activated, it will switch between yellow and blue.

Now you have a gaze controlled interface!

Adding Progress Views

In most cases, you will need some way to provide feedback to the user, on how long they are in an activation. Obital supplies a default GazeProgressView just for this. You can add it to any view the same way as you would add the GazeInteractionView.

func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
    cell.addGazeProgressView()
}

Progress View

The progress view is enabled when the gaze enters the gaze interaction view. When it reaches full circle, the user knows that the view is selected.

Updating progress view in your own classes

You can update the gaze progress view in your own views by overriding didHighlight(_) and and use the following approach:

if let progressView = self.subviews.first(where: {$0 is GazeProgressView}) as? GazeProgressView {
    progressView.animate(to: currentDwell)
}

Saving Gaze Progress

Obital manages all gaze interaction elements through the shared singleton GazePointViewManager.shared. You cannot create or modify most of this class, but you do have some options for customization. When interacting with multiple elements, it may be advantagerous to "save" the progress of a number of cells. The default number of cells able to be highlighted simultaneously is 1. You can change this value by using maxHighlightedViews of the gaze point manager:

// in GazeInteractionViewController.swift

override func viewDidLoad() {
    // ... previous code above
    GazePointViewManager.shared.maxHighlightedViews(2)
}