\documentclass[twoside,11pt]{article}

\renewcommand{\topfraction}{0.9}
\renewcommand{\bottomfraction}{0.9}
\renewcommand{\textfraction}{0.1}
\renewcommand{\floatpagefraction}{0.9}
\setcounter{topnumber}{4}

%\usepackage[utf8]{inputenc}
\usepackage{lscape}
\usepackage{subfigure}
\usepackage[table]{xcolor}
\usepackage{amsmath}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfig}
\usepackage{xspace}
\usepackage{pslatex}
\usepackage[nolist]{acronym}
\usepackage{tikz}
\usetikzlibrary{trees}

%\ac{acronym}   Expand and identify the acronym the first time; use only the acronym thereafter
%\acf{acronym}  Use the full name of the acronym.
%\acs{acronym}  Use the acronym, even before the first corresponding \ac command
%\acl{acronym}  Expand the acronym without using the acronym itself.

\newcommand{\kinect}{Kinect$^{TM}$}
\newcommand{\eg}{e.g.}
\newcommand{\ie}{i.e.}
\newcommand{\vs}{vs.}
\newcommand{\matlab}{Matlab\textregistered~}
\renewcommand{\matlab}{Matlab~}


%\newcolumntype{x}[1]{>{\centering\hspace{0pt}}p{#1}}

\newcommand{\mif}{\mbox{if}}
\newcommand{\motherwise}{\mbox{otherwise}}

\usepackage[hyphens]{url}
\usepackage{jmlr2e}
% Definitions of handy macros can go here

% Add a period to the end of an abbreviation unless there's one
% already, then \xspace.

%\xspace adds a space unless followed by punctuation.
\newcommand{\ha}{\emph{Hand-Arrangement}\xspace}
\newcommand{\tab}{\emph{Location}\xspace}
\newcommand{\sig}{\emph{Motion}\xspace}
\newcommand{\dez}{\emph{HandShape}\xspace}
\newcommand{\ori}{\emph{Orientation}\xspace}

% Heading arguments are {volume}{year}{pages}{submitted}{published}{author-full-names}

\jmlrheading{17}{2016}{1-54}{9/14; Revised 1/15}{4/16}{Sergio Escalera, Vassilis Athitsos, and Isabelle Guyon}

% Short headings should be running head and authors last names

\ShortHeadings{Challenges in multimodal gesture recognition}{Escalera, Athitsos, and Guyon} \firstpageno{1}

\begin{document}
\definecolor{tred}{rgb}{0.6,0,0}    \newcommand{\crgb}[1]{\textcolor{tred}{#1}}
\definecolor{tgreen}{rgb}{0,0.4,0}  \newcommand{\cdepth}[1]{\textcolor{tgreen}{#1}}
\definecolor{torange}{rgb}{0.4,0,0.4} \newcommand{\cthermal}[1]{\textcolor{torange}{#1}}
\definecolor{tblue}{rgb}{0,0,0.6} \newcommand{\cnocat}[1]{\textcolor{tblue}{#1}}



\title{Challenges in multimodal gesture recognition}

\author{\name Sergio Escalera \email sergio@maia.ub.es \\
       \addr Computer Vision Center UAB and University of
       Barcelona\\
    \name Vassilis Athitsos \email athitsos@uta.edu \\
       \addr University of Texas\\
              \name Isabelle Guyon \email guyon@chalearn.org \\
       \addr ChaLearn, Berkeley, California\\
}

\editor{Zhuowen Tu}

\maketitle

\begin{abstract}%   <- trailing '%' for backward compatibility of .sty file
This paper surveys the state of the art on multimodal gesture
recognition and introduces the JMLR special topic on gesture
recognition 2011-2015. We began right at the start of the \kinect
revolution when inexpensive infrared cameras providing image depth
recordings became available. We published papers using this
technology and other more conventional methods, including regular
video cameras, to record data, thus providing a good overview of
uses of machine learning and computer vision using multimodal data
in this area of application. Notably, we organized a series of
challenges and made available several datasets we recorded for
that purpose, including tens of thousands of videos, which are
available to conduct further research. We also overview recent
state of the art works on gesture recognition based on a proposed
taxonomy for gesture recognition, discussing challenges and future
lines of research.
\end{abstract}

\begin{keywords}
Gesture Recognition, Time Series Analysis, Multimodal Data
Analysis, Computer Vision, Pattern Recognition, Wearable sensors,
Infrared Cameras, \kinect.
 \end{keywords}

\section{Introduction}


Gestures are naturally performed by humans. Gestures are produced
as part of deliberate actions, signs or signals, or subconsciously
revealing intentions or attitude. They may involve the motion of
all parts of the body, but the arms and hands, which are essential
for action and communication, are often the focus of studies.
Facial expressions are also considered gestures and provide
important cues in communication.

Gestures are present in most daily human actions or activities,
and participate to human communication by either complementing
speech or substituting themselves to spoken language in
environments requiring silent communication (under water, noisy
environments, secret communication, etc.) or for people with
hearing disabilities. The importance of gestures in communication
is rooted in primal behaviors: the gesture-first theory, supported
by the analysis of mirror neurons in
primates~\citep{gesturefirst2}, indicated that the first steps of
language phylogenetically were not speech, nor speech with
gesture, but were gestures
alone~\citep{gesturefirst1,gesturefirst2}. See examples of primate
communication by means of gestures in Figure~\ref{fig:primates}.

\begin{figure*}
    \begin{center}
    \includegraphics*[width=\linewidth]{primates.png}
    \end{center}\vspace{-0.5cm}
    \caption{\label{fig:primates} Example of possible bonobo iconic gestures. (a) Start of swing gesture (or shove); (b) End of swing gesture (or shove); (c) Start of iconic swing, other bonobo starts to move; (d) End of iconic swing, other moving. Image from~\citep{gesturefirst1}.}
\end{figure*}

Given the indubitable importance of gestures in human activities,
there has been huge interest by the Computer Vision and Machine
Learning communities to analyze human gestures from visual data in
order to offer new non-intrusive technological solutions. For
completeness, in this paper we also review some gesture
recognition systems with data acquired from wearable sensors,
although the comprehensive review of papers focus on the analysis
of different visual modalities.

Applications are countless, like Human Computer Interaction (HCI),
Human Robot Interaction (HRI) (also named human machine
interaction HMI), communication, entertainment, security, art,
semiotics, commerce and sports, while having an important social
impact in assistive technologies for the handicapped and the
elderly. Some examples of applications are illustrated in
Fig.~\ref{fig:app}.

In addition to the recent advances in human and gesture
recognition from classical RGB visual data, the automatic analysis
of human body from sensor data keeps making rapid progress with
the constant improvement of (i) new published methods that
constantly push the state-of-the-art and (ii) the recent
availability of inexpensive 3D video sensors such as \kinect,
providing a complementary source of information, and thus allowing
the computation of new discriminative feature vectors and improved
recognition by means of fusion strategies. In
section~\ref{background} we review the state of the art in gesture
recognition.


\begin{figure*}
    \begin{center}
\begin{tabular}{ccc}
    \includegraphics*[width=4.7cm,height=3.5cm]{appDriver.png}&
\includegraphics*[width=4.7cm,height=3.5cm]{appsign.png}&
\includegraphics*[width=4.7cm,height=3.5cm]{appretrieval.jpg}\\
(a)&(b)&(c)\\
    \includegraphics*[width=4.7cm,height=3.5cm]{appsurveillance.png}&
\includegraphics*[width=4.7cm,height=3.5cm]{appHCI.JPG}&
\includegraphics*[width=4.7cm,height=3.5cm]{appinpatient.png}\\
(d)&(e)&(f)\\
\end{tabular}{}
    \end{center}\vspace{-0.5cm}
    \caption{\label{fig:app} Some applications of gesture recognition. (a) Gesture recognition for driver assistance, from~\citep{drive}, (b) Sign Language Recognition,
(c) action/gesture recognition for content retrieval and
categorization, from~\citep{appretrieval}, (d) surveillance, (e)
Human Computer/Robot/Machine Interaction, and (f) Assistive
technology for people with reduced autonomy.}
\end{figure*}








In order to push research and analyze the gain of multimodal
methods for gesture recognition, in 2011 and 2012, ChaLearn
organized a challenge on single user one-shot-learning gesture
recognition with data recorded with \kinect in which 85 teams
competed. Starting from baseline methods making over 50\% error
(measured in Leveinshtein distance, a metric counting the number
of substitutions, insertions and deletions, analogous to an error
rate), the winners brought the error rate below 10\%. While there
was still some margin of improvement on such tasks to reach human
performance (which is below 2\% error), we were encouraged to make
the task harder to push the state of the art in computer vision.
In our second ChaLearn challenge on Multimodal Gesture Recognition
in 2013, we proposed a user-independent task with data recorded
with \kinect, with a larger vocabulary and continuously performed
gestures. Of 60 participating teams, the winner attained an error
rate of 10\% on this data set, in terms of Leveinshtein distance.
In 2014, we used the same Multimodal Gesture Recognition dataset
with the objective of performing gesture spotting. The winner of
the competition, with a deep learning architecture, obtained an
overlapping near 0.9. Lastly, in 2014 and 2015 we ran an action
spotting challenge with a new dataset consisting of RGB sequences
of actors performing different isolated and collaborative actions
in outdoor environments. Future challenges we are planning include
the analysis of gestures taking into account face and contextual
information, involving many modalities in the recognition process.
In this paper we also review other existing international
challenges related to gesture recognition.

Our first workshop at CVPR from our 2011 challenge emphasized
mostly 2D video data meanwhile our second and third workshops at
CVPR, ICPR, ICMI, ECCV conferences from our 2012, 2013, and 2014
challenges were focused on affordable 3D sensors for gesture
recognition research, also including audio information. In ECCV
2014 and CVPR 2015 workshops we also promoted different aspects of
looking at people, including pose recovery, activity recognition,
and scene understanding where humans are present. In addition to
best challenge results, many research papers devoted to gesture
recognition were published and presented in our challenge
workshops. We also invited keynote speakers in diverse areas of
pose and gesture research, including sign language recognition,
body posture analysis, action and activity recognition, and facial
expression or emotion recognition.

In this special topic on gesture recognition, extension of best
challenge and workshop papers from previous events have been
published. In addition, new description and learning strategies
papers related to gesture recognition have been published. All of
them will be shortly reviewed in the following sections.

The rest of the paper is organized as follows: Section 2 reviews
the state of the art on gesture recognition, defining a taxonomy
to describe existing works as well as available databases for gesture
and action recognition. Section 3 describes the series of gesture
and action recognition challenges organized by ChaLearn,
describing the data, objectives, schedule, and achieved results by
the participants. For completeness we also review other existing
gesture challenge organizations. In Section 4 we review the
published papers in this gesture recognition topic which are
related to ChaLearn competitions. Section 5 describes special
topic published papers related to gesture recognition which are
not based on ChaLearn competitions. Finally, Section 6 discusses
main observations about the published papers.






















\section{Related Work in Gesture
Recognition}\label{background}

In this section we present a taxonomy for action/gesture
recognition, we review most influential works in the field, and
finally we review existing datasets for action/gesture recognition
together with the performance obtained by state of the art
methods.

\subsection{Taxonomy for gesture recognition}

Fig.~\ref{fig:taxonomy} is an attempt to create a taxonomy of the various components involved in
conducting research in action/gesture recognition. We include various aspects relating to the problem setting, the data acquisition, the tools, the solutions, and the applications.


\begin{figure}
\centering
\resizebox{1\textwidth}{!}{%
\begin{tikzpicture}[grow'=right,level distance=1.65in]
    \tikzstyle{edge from parent} = [thick, draw, edge from parent fork right]
    \tikzstyle{every node}=[font=\Huge]
    \tikzstyle{root} = [text width=85mm,align=center]

    \tikzstyle{nodeL1} = [text width=80mm,align=center]
    \tikzstyle{level 1} = [level distance=95mm]

    \tikzstyle{nodeL2} = [text width=10mm,align=center]
    \tikzstyle{level 2} = [level distance=105mm]

    \tikzstyle{nodeL3} = [text width=40mm,align=center,xshift=5mm]
    \tikzstyle{level 3} = [level distance=85mm]

    \tikzstyle{nodeL4} = [text width=90mm,align=center,xshift=20mm]
    \tikzstyle{level 4} = [level distance=145mm]

    \tikzstyle{termL2} = [text width=460mm,align=left,xshift=190.0mm]

    \tikzstyle{termL3} = [text width=80mm,align=left,xshift=5mm]
    \tikzstyle{termL4} = [text width=360mm,align=left,xshift=150mm]
    \tikzstyle{termL5} = [text width=135mm,align=left,xshift=5mm]

    % Supervisor
    \node[root]{Gesture Recognition}
    child[sibling distance=70mm]{
        node[nodeL1,yshift=30mm]{Aspects of gesture meaning}
        child[sibling distance=10mm]{
            node[termL2]{Spatial information: where it occurs; eg.  \citep{malicICMI2004}.}
        }child[sibling distance=10mm]{
            node[termL2]{Pathic information: the path it takes; eg. \citep{alon_pami2009}.}
        }child[sibling distance=10mm]{
            node[termL2]{Symbolic information: the sign it makes; eg. \citep{vogler_iccv1999}.}
        }child[sibling distance=10mm]{
            node[termL2]{Action: the type of action that is being performed; eg. \citep{schuldt2004recognizing}.}
        }child[sibling distance=10mm]{
            node[termL2]{Body pose: limb locations and joint angless; eg. \citep{modec13}.}
        }child[sibling distance=10mm]{
            node[termL2]{Handshape: the posture of the hand; eg. \citep{trieschPAMI2001}.}
        }child[sibling distance=10mm]{
            node[termL2]{Affective information: its emotional quality; eg. \citep{castellano2007recognising,kapurSPRINGER2005}.}
        }
    }
    child[sibling distance=70mm]{
        node[nodeL1,yshift=30mm]{Actors involved in gesture recognition}
        child[sibling distance=10mm]{
            node[termL2]{Human â€“ machine/robot/computer interaction; eg. \citep{malicICMI2004}.}
        }child[sibling distance=10mm]{
            node[termL2]{Isolated human gestures; eg. signs in  \citep{vogler_iccv1999}.}
        }child[sibling distance=10mm]{
            node[termL2]{Human-human gestures; eg. \citep{patronBMVC2010}.}
        }child[sibling distance=10mm]{
            node[termL2]{Human-object interaction; eg. \citep{kjellstromECCV2008}.}
        }
    }
    child[sibling distance=75mm]{
        node[nodeL1,yshift=30mm]{Body limbs usage}
        child[sibling distance=10mm]{
            node[termL2]{Hands /arms (1 or 2); eg. \citep{vogler_iccv1999}. }
        }child[sibling distance=10mm]{
            node[termL2]{Body; eg. \citep{shotton_cvpr2011}.}
        }child[sibling distance=10mm]{
            node[termL2]{Head; eg. \citep{erdemICPR2002}.}
        }
    }
    child[sibling distance=88mm]{
        node[nodeL1,yshift=30mm]{Static / dynamic gestures}
        child[sibling distance=10mm]{
            node[termL2]{Static; eg. \citep{trieschPAMI2001}. }
        }child[sibling distance=10mm]{
            node[termL2]{Dynamic; eg. \citep{chairgest,shotton_cvpr2011}.}
        }
    }
    child[sibling distance=120mm]{
        node[nodeL1,yshift=30mm]{Context aware}
        child[sibling distance=10mm]{
            node[termL2]{Computer Vision scene analysis; eg. \citep{pieropan_iros2014,shapovalova_pria2011}.}
        }child[sibling distance=10mm]{
            node[termL2]{Intelligent Objects; eg. \citep{czabke_pcth2010}.}
        }
    }
    child[sibling distance=200mm]{
        node[nodeL1,yshift=30mm]{Data source}
        child[sibling distance=28mm]{
            node[nodeL3]{Visual}
            child[sibling distance=15mm]{
                node[termL4]{RGB; eg. \citep{linICCV2009}.}
            }
            child[sibling distance=15mm]{
                node[termL4]{Depth; eg. \citep{shotton_cvpr2011}.}
            }
            child[sibling distance=15mm]{
                node[termL4]{Thermal; eg. \citep{larsonCHI2011}.}
            }
        }child[sibling distance=28mm]{
            node[termL2]{Audio; eg. \citep{escalera2013multi}.}
        }child[sibling distance=20mm]{
            node[nodeL3]{Wearable}
            child[sibling distance=20mm]{
                node[termL4]{Magnetic field trackers, body suits; eg. \citep{kapurSPRINGER2005}.}
            }
            child[sibling distance=20mm]{
                node[nodeL4]{Instrumented (data) gloves; eg. \citep{ma_fg2000}.}
                child[sibling distance=10mm]{
                    node[termL5]{Active}
                }
                child[sibling distance=10mm]{
                    node[termL5]{Passive}
                }
            }
        }
    }
    child[sibling distance=20mm]{
        node[nodeL1,yshift=60mm]{Ambient / egocentric}
        child[sibling distance=10mm]{
            node[termL2]{ambient; eg. \citep{bloomCVPRW2012,sadeghipourBMVC2012}. }
        }child[sibling distance=10mm]{
            node[termL2]{egocentric; eg. \citep{baraldi2014_379,starner_pami1998}.}
        }
    }
    child[sibling distance=40mm]{
        node[nodeL1,yshift=30mm]{Databases}
        child[sibling distance=10mm]{
            node[termL2]{Data modalities; eg. color \citep{Johnson10}, depth \citep{renACMICM2011}, skeletal data \citep{sadeghipourBMVC2012}.}
        }child[sibling distance=10mm]{
            node[termL2]{Viewing field: Upper body \citep{modec13} / full body  \citep{sigal_ijcv2010} / egocentric \citep{fathi_cvpr2011}.}
        }child[sibling distance=10mm]{
            node[termL2]{Occlusions; eg. \citep{pamipascal,modec13}.}
        }child[sibling distance=10mm]{
            node[termL2]{Frontal view; eg. \citep{liuIJCAI2015} / different viewpoints; eg. \citep{andriluka14benchmark}.}
        }child[sibling distance=10mm]{
            node[termL2]{Controlled; eg. \citep{sigal_ijcv2010} / uncontrolled; eg. \citep{tran2010improved}.}
        }child[sibling distance=10mm]{
            node[termL2]{Complex background; eg. \citep{laptev2008learning} / homogeneous background; eg. \citep{neidleLREC2012}}
        }child[sibling distance=10mm]{
            node[termL2]{Ethnic, gender and age variabilities; eg. \citep{andriluka14benchmark,fothergillCHI2012}.}
        }child[sibling distance=10mm]{
            node[termL2]{Segmented gestures; eg. \citep{sadeghipourBMVC2012} / unsegmented gestures; eg. \citep{bloomCVPRW2012}.}
        }child[sibling distance=10mm]{
            node[termL2]{Amount of data, subjects and gesture categories }
        }child[sibling distance=12mm]{
            node[termL2]{Type of gestures: iconic; eg. \citep{sadeghipourBMVC2012}, emblematic; eg. \citep{song_fg2011b}, sign language; eg. \citep{neidleLREC2012}, action; eg. \citep{neuro}.}
        }child[sibling distance=14mm]{
            node[termL2]{Individual gestures; eg. \citep{Johnson10} / collaborative gestures; eg. \citep{laptev2008learning}.}
        }
    }
    child[sibling distance=55mm]{
        node[nodeL1,yshift=0mm]{Evaluation metrics}
        child[sibling distance=25mm]{
            node[termL2]{Spotting: Jaccard (overlap) \citep{neuro}, F1 score (or accuracy related) \citep{fothergillCHI2012}.}
        }child[sibling distance=25mm]{
            node[termL2]{Classification (sequences containing gestures or segmented gestures): classification accuracy \citep{renACMICM2011}, F1 score (or accuracy related), Levenstein \citep{guyon-mva-2014}}
        }
    }
    child[sibling distance=60mm]{
        node[nodeL1,yshift=10mm]{Gesture challenge organizations, eg. \citep{guyon-mva-2014,pamipascal}}
    }
    child[sibling distance=73mm]{
        node[nodeL1,yshift=60mm]{Recognition (Appearance / model based)}
        child[sibling distance=20mm]{
            node[termL2]{Preprocessing: noise removal, signal smoothing; eg. \citep{song_fg2011a}.}
        }child[sibling distance=20mm]{
            node[termL2]{Feature extraction: body limbs localization and tracking (distances and angles); eg. \citep{song_fg2011a}, appearance features of body parts; eg. \citep{roussos_jmlr2013}.}
        }child[sibling distance=20mm]{
            node[termL2]{Segmentation: sliding windows; eg. \citep{wan-2013}, dynamic programming; eg. \citep{alon_pami2009}.}
        }child[sibling distance=20mm]{
            node[termL2]{Recognition: deep learning strategies \citep{natalia2014}, DTW \citep{wang_sga2010}, HMM \citep{starner_pami1998,vogler_iccv1999}, SVM \citep{schuldt2004recognizing}, RF \citep{demirdjianICMI2009}, HCRF \citep{quattoni_pami2007}}
        }
    }
    child[sibling distance=65mm]{
        node[nodeL1,yshift=10mm]{Applications}
        child[sibling distance=5mm]{
            node[termL2]{HRI (\citep{leeFG2006}), HCI \citep{malicICMI2004}, HMI \citep{drive}, sign language \citep{starner_pami1998}, gesture to speech \citep{fels_thesis}, presentations \citep{licsar2004hand}, virtual environments \citep{wexelblatTCHI1995}, 3d modeling, multimodal interaction \citep{rauschertGIS2002}, TV controlling \citep{Freeman94televisioncontrol}}
        }
    };
\end{tikzpicture}}
\centering \vspace{-0.3cm} \caption{\scriptsize{Taxonomy for
gesture recognition.}} \label{fig:taxonomy} \vspace{-0.5cm}
\end{figure}

First, regarding the problem setting, the interpretation of
gestures critically depends on a number of factors, including the
environment in which gestures are performed, their span in time
and space, and the intentional meaning in terms of symbolic
description and/or the subconscious meaning revealing
affective/emotional states. The problem setting also involves
different actors who may participate in the execution of gestures
and actions: human(s) and/or machine(s) (robot, computer, etc.),
performing with or without tools or interacting or not with
objects. Additionally, independently of the considered modality,
for some gestures/actions different parts of the body are
involved. While many gesture recognition systems only focus on
arms and hands, full body motion/configuration and facial
expressions can also play a very important role. Another aspect of
the problem setting involves whether recognized gestures are
static or dynamic. For the first case, just considering features
from an input frame or any other acquisition device describing
spatial configuration of body limbs, a gesture can be recognized.
In the second case, the trajectory and pose of body limbs provide
the highest discriminative information for gesture recognition. In some settings, gestures are defined based not only on the pose and motion of the human, but also on the surrounding context, and more specifically on the objects that the human interacts with. For such settings, one approach for achieving context awareness is scene analysis, where information is extracted from the scene around the subject (e.g., \cite{pieropan_iros2014,shapovalova_pria2011}). Another approach is to have the subject interact with intelligent objects. Such objects use embedded hardware  and software to facilitate object recognition/localization, and in some cases to also monitor interactions between such objects and their environment (e.g., \cite{czabke_pcth2010})


Second, the data are, of course, of very central importance, as in every machine learning application. The data sources may vary: when recognizing gestures, input data can come from different
modalities, visual (RGB, 3D, or thermal, among others), audio, or
wearable sensors (magnetic field trackers, instrumented (data)
gloves, or body suits, among others). In the case of gloves, they
can be active or passive. Active ones make use of a variety of
sensors on a glove to measure the flexing of joints or the acceleration
and communicates data to the host device using wired or
wireless technology. Passive ones consist only of markers or
colored gloves for finger detection by an external device such as a
camera. Although most gestures are recognized by means of ambient
intelligent systems, looking at the person from outside, some gesture
recognition approaches are based on egocentric computing, using
wearable sensors or wearable cameras that analyze, for instance,
hand behaviors. Additionally, it is well-known that context provides rich
information that can be useful to better infer the meaning of some
gestures. Context information can be obtained by means of
computer vision scene analysis, interaction with objects, but also via intelligent objects in the scene (objects with sensors
that emit signals related to proximity and interaction).
Some examples of acquisition devices are shown in
Figure~\ref{fig:acquisition}.

Third, the field of gesture recognition has shaped up thanks to the adoption of standard methodology.
In order to advance in the design of robust action/recognition
approaches, several datasets with different complexity have been
published, and several world challenges helped to push the
research in the area. This required the definition of standard
evaluation metrics to render methods comparable. Notably, when one wants to recognize actions/gestures from data,
common steps involve pre-processing of the acquired data, feature
extraction, segmentation of begin-end of gesture and its final
gesture/action label classification. Many datasets include preprocessed and/or thoroughly annotated data.


\begin{figure*}
    \begin{center}
\begin{tabular}{ccc}
    \includegraphics*[width=4.7cm,height=3.5cm]{sensor8.png}&
\includegraphics*[width=4.7cm,height=3.5cm]{sensor7.png}&
\includegraphics*[width=4.7cm,height=3.5cm]
{thermal.png}\\
(a)&(b)&(c)\\
    \includegraphics*[width=4.7cm,height=3.1cm]{sensor2.png}&
\includegraphics*[width=4.7cm,height=3.1cm]{sensor3.png}&
\includegraphics*[width=4.7cm,height=3.1cm]{sensor4.png}\\
(d)&(e)&(f)\\
\end{tabular}{}
    \end{center}
    \caption{\label{fig:acquisition} Some examples of acquisition devices for gesture recognition. (a) left: mobile with GPS and accelerometer, right: inertial sensor with accelerometer and gyroscope,
(b) Google Glasses for egocentric computing, (c) thermal imagery
for action recognition, (d) audio-RGB-depth device, (e) active
glove, and (f) passive glove.}
\end{figure*}


Fourth, gesture recognition has offered many opportunities to
algorithm developers to innovate. The approaches, which
essentially can be categorized into appearance-based and
model-based methods, are going to be reviewed in the next section.
We will mention only the most influential works for action/gesture
recognition illustrating various aspects of the problem setting,
data acquisition, and methodology defined in our taxonomy. Note
that although we defined a general taxonomy for gesture
recognition, in this paper, we put special emphasis on computer
vision and machine learning methods for action/gesture
recognition.

Finally, our taxonomy would not be complete without the wide array
of applications of gesture/action recognition, already mentioned
in the introduction.


\subsection{Overview of gesture recognition methods}


Different surveys have been published so far reviewing gesture
recognition
systems~\citep{LaViola,survey2,48200257,survey3,survey4,survey5,survey6}.
In this section, we present an up-to-date review of most
influential works in the field.

% STATIC AND DYNAMIC
\subsubsection{Recognizing static gestures and hand pose}
In the case of static gestures, frequently hand shape is the
important differentiating feature
\citep{cui_cviu2000,s_freeman1996,kelly_prl2010,ren_acmmm2011,triesch_ivc2002},
although the pose of the rest of the body can also be important,
e.g., \citep{lang_cvpr2010, vandenbergh_ijcv2009}. For static hand
pose classification, some approaches rely on visual markers, such
as a color glove with a specific color for each finger, e.g.,
\citep{wang_tog2009}. Other approaches can recognize the hand pose
on unadorned hands. Appearance-based methods, like
\citep{moghaddam95, triesch_ivc2002,s_freeman1996,wu_huang2}, can
be used for recognizing static hand postures observed from
specific viewpoints.

% appearance and model based

Model-based methods for hand pose estimation
\citep{oikonomidis_iccv2011,delagorce_pami2011,oikonomidis_accv2010,rehg_iccv1995}
typically match visual observations to instances of a predefined
hand model. Single frame pose estimation methods try to solve the
hand pose estimation problem without relying on temporal
information \citep{Athitsos_cvpr2003}. Most recently, due to the
advent of commercially available depth sensors, there is an
increased interest in methods relying on depth data
\citep{keskin_eccv2012,mo_cvpr2006,oikonomidis_iccv2011,pugeault_iccvw2011,spherical}.



\subsubsection{From body part detection to holistic pattern detection}

Dynamic gestures are characterized by both the pose and the motion of the relevant body parts. Much effort has traditionally be put into detecting first {\bf body parts} and then tracking their motion.
In color videos, detecting hands can be quite challenging,
although better performance can be achieved by placing additional
constraints on the scene and the relative position of the subject
and the hands with respect to the camera
\citep{cui_cviu2000,isard_ijcv1997,kolsch_rtvhci2004,ong_fg2004,stefanov_rtvhci2005,stenger_iccv2003,sudderth_nips2004}.
Commonly-used visual cues for hand detection such as skin color,
edges, motion, and background subtraction
\citep{Chen03,martin_afgr1998} may also fail to unambiguously
locate the hands when the face, or other ``hand-like" objects are
moving in the background.


In \citep{ChengLi2013} the authors propose a hand segmentation approach
from egocentric RGB data by the combination of color and texture
features. In\citep{baraldi2014_379}, dense features are extracted
around regions selected by a new hand segmentation technique that
integrates superpixel classification, temporal and spatial
coherence. Bag of visual words and linear SVM are used for final
representation and classification.

Depth cameras have become widely available in recent years, and hand detection (in
tandem with complete body pose estimation) using such cameras (and
also in combination with other visual modalities) can be performed
sufficiently reliably for many applications
\citep{shotton_cvpr2011,cvprtoni}.
The authors of \citep{Zhou13} propose a part-based hand gesture
recognition system using \kinect sensor. Finger-EarthMover's
Distance (FEMD) metric is proposed to measure the dissimilarity
between hand shapes. It matches the finger parts while not the
whole hand based on hand segmentation and contour analysis. The
method is tested on their own 10-gesture dataset.

Instead of estimating hand position and/or body pose before
recognizing the gesture, an alternative is to customize the
recognition module so that it does not require the exact
knowledge of hand positions, but rather accepts as input a list of several
candidate hand locations \citep{alon_pami2009,sato_icpr02,gctoni}.

Another approach is to use {\bf global image/video features}. Such
global features include motion energy
images~\citep{bobick_pami2001}, thresholded intensity images and
difference images \citep{dreuw_smvp2006}, 3D shapes extracted by
identifying areas of motion in each video frame
\citep{gorelick_pami2007} and histograms of pairwise distances of
edge pixels \citep{nayak_vhci2005}. Gestures can also be modelled
as rigid 3D patterns \citep{ke_iccv2005}, from which features can
be extracted using 3D extensions of rectangle filters
\citep{viola_face2001}. The work of~\citep{kernel15} uses pixel-level attributes in a hierarchical architecture of 3D kernel descriptors, and efficient match kernel is used to recognize gestures from depth data.


Along similar lines, \citep{ali2010} propose a set of kinematic features that
are derived from the optical flow for human action recognition in
videos: divergence, vorticity, symmetric and antisymmetric flow
fields, second and third principal invariants of flow gradient and
rate of strain tensor, and third principal invariant of rate of
rotation tensor, which define spatiotemporal patterns. These
kinematic features are computed by Principal Component Analysis
(PCA). Then multiple instance learning (MIL) is applied for
recognition in which each action video is represented by a bag of
kinematic modes. The proposal is evaluated on the RGB Weizmann and
KTH action data sets, showing comparable result to state of the
art performances.


Much effort has also been put into {\bf spatiotemporal invariant features}.
In \citep{Junsong11} the authors propose a RGB action recognition system
based on a pattern matching approach, named naive Bayes mutual
information maximization (NBMIM). Each action is characterized by
a collection of spatiotemporal invariant features which are
matched with an action class by measuring the mutual information
between them. Based on this matching criterion, action detection
is to localize a subvolume in the volumetric video space that has
the maximum mutual information toward a specific action class. A
novel spatiotemporal branch-and-bound (STBB) search algorithm is
designed to efficiently find the optimal solution. Results show
high recognition results on KTH, CMU, and MSR data sets, showing
speed up inference in comparison with standard 3D
branch-and-bound.

Another example is the paper of \citep{Derpanis13} in which a compact local descriptor of video dynamics
is proposed for action recognition in RGB data sequences. The
descriptor is based on visual spacetime oriented energy
measurements. An associated similarity measure is introduced that
admits efficient exhaustive search for an action template, derived
from a single exemplar video, across candidate video sequences.
The method is speeded up by means of a GPU implementation. Method
is evaluated on UCF and KTH data sets, showing comparable results
to state of the art methods.


The work of \citep{Yang14} presents a coding scheme to aggregate
low-level descriptors into the super descriptor vector (SDV). In
order to incorporate the spatio-temporal information, the super
location vector (SLV) models the space-time locations of local
interest points in a compact way. SDV and SLV are combined as the
super sparse coding vector (SSCV) which jointly models the motion,
appearance, and location cues. The approach is tested on HMDB51
and Youtube with higher performance in comparison to state of the
art approaches.


\subsubsection{Segmentation of gestures and gesture spotting}

Dynamic gesture recognition methods can be further categorized
based on whether they make the assumption that gestures have
already been segmented, so that the start frame and end frame of
each gesture is known. Gesture spotting is the task of recognizing gestures in
unsegmented video streams, that may contain an unknown number of
gestures, as well as intervals were no gesture is being performed.
Gesture spotting methods can be broadly classified into two
general approaches: the direct approach, where temporal
segmentation precedes recognition of the gesture class, and the
indirect approach, where temporal segmentation is intertwined with
recognition:
\begin{itemize}
\item {\bf Direct methods} (also called heuristic segmentation) first
compute low-level motion parameters such as velocity,
acceleration, and trajectory curvature~\citep{kang_prl2004} or
mid-level motion parameters such as human body
activity~\citep{kahol_fg2004}, and then look for abrupt changes
(e.g., zero-crossings) in those parameters to identify candidate
gesture boundaries.
\item {\bf Indirect methods} (also called recognition-based segmentation) detect gesture
boundaries by finding, in the input sequence, intervals that give
good recognition scores when matched with one of the gesture
classes. Most indirect methods
\citep{alon_pami2009,lee_pami99,oka_tcj1998} are based on
extensions of Dynamic Programming (DP) e.g., Dynamic Time Warping (DTW)
\citep{darrell_pami1996,kruskal83}, Continuous Dynamic Programming
(CDP) \citep{oka_tcj1998}, various forms of Hidden Markov
Models (HMMs)
\citep{brand_cvpr1997,Chen03,stefanov_rtvhci2005,lee_pami99,starner_pami1998,vogler_iccv1999,wilson_pami1999},
and most recently, Conditional Random Fields
\citep{lafferty_icml2001,quattoni_pami2007}. Also hybrid
probabilistic and dynamic programming approaches have been
recently published~\citep{pdtw}. In those methods, the gesture endpoint is detected by comparing
the recognition likelihood score to a threshold. The threshold can
be fixed or adaptively computed by a non-gesture garbage model
\citep{lee_pami99,yang_pami2009}, equivalent to silence models in
speech.
\end{itemize}
When attempting to recognize unsegmented gestures, a frequently
encountered problem is the {\em subgesture problem}: false
detection of gestures that are similar to parts of other longer
gestures. \citep{lee_pami99} address this issue using heuristics to
infer the user's completion intentions, such as moving the hand
out of camera range or freezing the hand for a while. An
alternative is proposed in \citep{alon_pami2009}, where a learning
algorithm explicitly identifies subgesture/supergesture
relationships among gesture classes, from training data.

Another common approach for gesture spotting is to first
extract features from each frame of the observed video, and then
to provide a sliding window of those features to a recognition module, which
performs the classification of the gesture
\citep{Corradini01,cutler_fg1998,darrell_pami1996,oka_cga2002,starner_pami1998,yang_pami2002}).
Oftentimes, the extracted features describe the position and
appearance of the gesturing hand or hands
\citep{cutler_fg1998,darrell_pami1996,starner_pami1998,yang_pami2002}). This approach can be integrated with recognition-based segmentation methods.


\subsubsection{Action and activity recognition}

The work of \citep{Wanqing00} presents an action graph to model
explicitly the dynamics of 3D actions and a bag of 3D points to
characterize a set of salient postures that correspond to the
nodes in the action graph. The authors propose a projection based
sampling scheme to sample the bag of 3D points from the depth
maps. In \citep{Sminchisescu06} it is proposed the first
conditional/discriminative chain model for action recognition.

The work of \citep{Zanfir2013} propose the non-parametric Moving
Pose (MP) framework for low-latency human action and activity
recognition. The moving pose descriptor considers both pose
information as well as differential quantities (speed and
acceleration) of the human body joints within a short time window
around the current frame. The descriptor is used with a modified
kNN classifier that considers both the temporal location of a
particular frame within the action sequence as well as the
discrimination power of its moving pose descriptor compared to
other frames in the training set. The method shows comparable
results to state of the art methods on MSR-Action3D and
MSR-DailyActivities3D data sets.


In \citep{omae13}, it is proposed a new descriptor for activity
recognition from videos acquired by a depth sensor. The depth
sequence is described using a histogram capturing the distribution
of the surface normal orientation in the 4D space of time, depth,
and spatial coordinates. To build the histogram, 4D projectors are
created, which quantize the 4D space and represent the possible
directions for the 4D normal. Projectors are initialized using the
vertices of a regular polychoron. Projectors are refined using a
discriminative density measure, such that additional projectors
are induced in the directions where the 4D normals are more dense
and discriminative. The proposed descriptor is tested on MSR
Actions 3D, MSR Gesture 3D, and MSR Daily Activity 3D, slightly
improving state of the art results.

In \citep{Wang14}, the authors propose to characterize the human actions
with an ``actionlet" ensemble model, which represents the interaction of
a subset of human joints. Authors train an ensemble of SVM
classifiers related to actionlet patterns, which includes 3D joint
features, Local Occupancy Patterns, and Fourier Temporal Pyramid.
Results on CMU MoCap, MSR-Action3D, MSR-DailyActivity3D, Cornell
Activity, and Multiview 3D data sets show comparable and better
performance than state of the art approaches.

The work of \citep{Tian14} presents an approach for activity
recognition in depth video sequences. Authors cluster hypersurface
normals in a depth sequence to form the polynormal which is used
to jointly characterize the local motion and shape information. In
order to globally capture the spatial and temporal orders, an
adaptive spatio-temporal pyramid is introduced to subdivide a
depth video into a set of space-time grids. It is then proposed a
scheme of aggregating the low-level polynormals into the super
normal vector (SNV) which can be seen as a simplified version of
the Fisher kernel representation. Authors validate the proposed
approach on MSRAction3D, MSRDailyActivity3D, MSRGesture3D, and
MSRActionPairs3D data sets slightly improving in all cases state
of the art performances.


In \citep{gang14} the authors propose the orderlets to capture
discriminative information for gesture recognition from depth
maps. Orderlet features are discovered looking for frequent sets
of skeleton joints that provide discriminative information.
Adaboost is used for orderlets selection. Results on the ORGBD
data set shows a recognition rate of 71.4\% mean class average
accuracy, improving by near 5\% state of the art results on this
data set, and near 20\% improvement regarding frame level
classification. However the results showed on the
MSR-DailyActivity3D data set are inferior to the ones reported in
\citep{Luo14}.

The work of \citep{Hui14} presents a depth-based method for hand
detection and pose recognition by segmentation of different hand
parts. Authors based on RF for initial multipart hand
segmentation. Then, a Superpixel-Markov Random Field (SMRF)
parsing scheme is used to enforce the spatial smoothness and the
label co-occurrence prior to remove the misclassified regions.

\subsubsection{Approaches using non-video modalities and multimodal approaches}

In terms of multimodal approaches for gesture recognition,
\citep{Luo14} propose a sparse coding-based temporal
pyramid matching approach (ScTPM) for feature representation using
depth maps. The authors also propose the Center-Symmetric Motion
Local Ternary Pattern (CS-Mltp) descriptor to capture
spatial-temporal features from RGB videos. By fusing both RGB and
Depth descriptors, the authors improve state of the art results on
MSR-Action3D and MSR-DailyActivity3D data sets, with a 6\% and 7\%
of improvement, respectively.

In \citep{ionescuPAMI2014}, it is presented the Human3.6M data set,
consisting of 3.6Million accurate 3D Human poses, acquired by
recording the performance of 5 female and 6 male subjects, under 4
different viewpoints, for training realistic human sensing systems
and for evaluating the next generation of human pose estimation
models and algorithms. Authors also provide a set of large scale
statistical models and evaluation baselines for the dataset
illustrating its diversity.

In \citep{Xiao14} a wearable Immersion CyberGlove II is used to
capture the hand posture and the vision-based Microsoft \kinect
takes charge of capturing the head and arm posture. An effective
and real-time human gesture recognition algorithm is also
proposed.

In \citep{Hui2014} it is proposed to detect and segment different
body parts using RGB and Depth data sequences. The method uses
both temporal constraints and spatial features, and performs hand
parsing and 3D fingertip localization for hand pose estimation.
The hand parsing algorithm incorporates a spatial-temporal feature
into a Bayesian inference framework to assign the correct label to
each image pixel. The 3D fingertip localization algorithm adapts
is based on geodesic extrema extraction to fingertip detection.
The detected 3D fingertip locations are finally used for hand pose
estimation with an inverse kinematics solver. The work
of~\citep{random15} use random forest for both segmenting and
classifying gesture categories from data coming from different
sensors.


Although many works base only on inertial
data~\citep{benba01,siamese15}, multimodal approaches are often
considered in order to combine trajectory information will pose
analysis based on visual data. The works
of~\citep{LCJK2014,citizen} present approaches for gesture
recognition based on the combination of depth and inertial data.
In~\citep{LCJK2014} skeleton obtained from depth data and data
from inertial sensors are train within HMM in order to perform
hand gesture recognition. A similar approach is presented
in~\citep{citizen}, but also recognizing objects present in the
scene and using DTW for recognition with the objective of
performing ambient intelligent analysis to support people with
reduced autonomy. In~\citep{Gowing2014}, it is presented a
comparison of WIMU aWireless/Wearable Inertial Measurement Unit
and \kinect. However, comparison is performed independently,
without considering a fusion strategy.


The work of \citep{gathering} is one of the few that compare the
performance of different segmentation approaches for gesture
recognition comparing RGB, depth, and thermal modalities. They
propose a simple segmentation approach of faces and one hand for
recognizing letters and numbers for HCI. They obtained higher
performance by the use of depth maps. Unfortunately no mutimodal
fusion approaches are tested in order to analyze when each
modality can complement the information provided by the rest of
modalities.

The work of~\citep{escalera2013multi} summarizes a 2013 challenge
on multimodal gesture recognition, where in addition to RGB and
depth data, audio can be used to identify the performed gestures.


Few works considered context information in order to improve
gesture/action recognition systems. In~\citep{context1} it is
proposed to adapt gesture recognition based on a dialogue manager
as a partially observable Markov decision process (POMDP).
In~\citep{context2} two \kinect devices and smart objects are used
to estimate proximity and adapt the recognition prior of some
gestures.

The recent emergence of deep learning systems in computer vision
have also been applied to action/gesture recognition systems. In
\citep{NeverovaWolf14}, it is presented a deep learning based
approach for hand pose estimation, targeting gesture recognition.
The method integrates local and global structural information into
the training objective. In~\citep{deep2}, deep neural network (NN)
combining convolution and max-pooling (MPCNN) is proposed for
supervised feature learning and classification of RGB hand
gestures given by humans to mobile robots using colored gloves.
The hand contour is retrieved by color segmentation, then smoothed
by morphological image processing which eliminates noisy edges.
The system classifies 6 gesture classes with 96\% accuracy,
improving performance of several state of the art methods. The
work of~\citep{deep3} presents an approach that classifies 3D
gestures using jointly accelerometer and gyroscope signals from a
mobile device using convolutional neural network with a specific
structure involving a combination of 1D convolution.
In~\citep{pavlo15} convolutional deep neural networks are used to
fuse data from multiple sensors (short-range radar, a color
camera, and a depth camera) and to classify the gestures in a
driver assistance scenario.



\subsection{Sign language recognition}

An important application of gesture recognition is sign language
recognition. American Sign Language (ASL) is used by 500,000 to
two million people in the U.S. \citep{lane_1996, schein_1989}.
Overall, national and local sign languages are used all over the
world as the natural means of communication in deaf communities.

Several methods exist for recognizing isolated signs, as well as
continuous signing. Some researchers have reported results on continuous signing with vocabularies of thousands of signs, using input from digital gloves, e.g., \citep{yao_icpr2006}. However, glove-based interfaces are typically expensive for adoption by the general public, as well as intrusive, since the user has to wear one or two gloves connected with wires to a computer.

Computer vision methods for sign language recognition offer hope for cheaper, non-intrusive interfaces compared to methods using digital gloves. Several such methods have been proposed
\citep{bauer_icpr2000,cui_cviu2000,dreuw_smvp2006,kadir_bmvc2004,starner_pami1998,vogler_iccv1999,wang_sga2010,zieren_ibpria2005}. However, computer vision methods typically report lower accuracies compared to methods using digital gloves, due to the difficulty of extracting accurate information about the articulated pose and motion of the signer.

An important constraint limiting the accuracy of computer vision methods is the availability of training data. Using more examples per sign typically improves accuracy (see, e.g., \citep{kadir_bmvc2004,zieren_ibpria2005}). However, existing datasets covering large vocabularies have only a limited number of examples per sign. As an example, the ASLLVD dataset \citep{Athitsos_cvpr4hb2008} includes about 3,000 signs,
but only two examples are available for most of the signs. Some interesting research has aimed at enabling automated construction of large datasets. For example, \citep{cooper_cvpr2009} aim at automatically generating large corpora by automatically segmenting signs from close-captioned sign language videos. As another example, \citep{farhadi_cvpr2007} propose a method where sign models are learned using avatar-produced data, and then transfer learning is used to create models adapted for specific human signers.

The recent availability of depth cameras such as \kinect has changed the methodology and improved performance. Depth cameras provide valuable 3D information about the position and trajectory of hands in signing. Furthermore, detection and tracking of articulated human motion is significantly more accurate in depth video than in color video.
Several approaches have been  published in recent years that use depth cameras to improve accuracy in sign language recognition \cite{conly_petra2015,wang_fg2015,Zafrulla_American_2011}.



\subsection{Data sets for gesture and action recognition}

Tens of gesture recognition datasets have been made available to
the research community over the last several years. A summary of
available datasets is provided in Table \ref{tbl::datasets}. In
that table, for each data set we mark some important attributes of
the dataset, such as the type of gestures it contains, the data
modalities it provides, the viewing field, background, amount of
data, and so on. Regarding the ``occlusions'' attribute in that
table, we should clarify that it only refers to occlusions of the
subject by other objects (or subjects), and not to self
occlusions. Self occlusions are quite common in gestures, and are
observed in most datasets. We should also note that, regarding the
complexity of the background, dynamic and/or cluttered backgrounds
can make gesture recognition challenging in color images and
video. At the same time,  a complex background can be quite easy
to segment if depth or skeletal information is available, as is
the case in several datasets on Table \ref{tbl::datasets}.

In order to be able to fit Table \ref{tbl::datasets} in a single
page, we had to use abbreviations quite heavily. Table
\ref{tbl::dataset_acronyms} defines the different acronyms and
abbreviations used in Table \ref{tbl::datasets}.

The datasets we have created for our challenges have certain
unique characteristics, that differentiate them from other
existing datasets. The CDG2011 dataset \citep{guyon-mva-2014} has
a quite diverse collection of gesture types, including static,
pantomime, dance, signs, and activities. This is in contrast to
other datasets, that typically focus on only one or maybe two
gesture types. Furthermore, the CDG2011 dataset uses an evaluation
protocol that emphasizes one-shot learning, whereas existing data
sets typically have several training examples for each class. The
CDG2013 dataset introduces audio data to the mix of color and
depth that is available in some other data sets, such as
\citep{sadeghipourBMVC2012,bloomCVPRW2012}.

\begin{landscape}

\begin{table}\centering
\scriptsize
\renewcommand\tabcolsep{0pt}
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
\hline Dataset                    & Actors/                    &
Body          & Static/    & Modalities & Viewing       & Occl.
& Viewp.     & Controlled & Background & Variab.       & Seg.
& Amount     & Subjects   &
Classes    & Type          & Indiv./       & Eval.      \\

                           &
Objects                    & parts         & Dynamic    &
           & field         &               & Var.       &            &
           &               &               &            &            &
           &               & collab.       &            \\ \hline

HUMANEVA \citep{sigal_ijcv2010}             &
IH                         & F             & SF         &
MC,ST      & F             & No            & V          & C          &
SF,C       & E,G           & SF            & 40000 F    & 5          &
CBP        & BP            & I             & MPJPE      \\ \hline

Human3.6M \citep{ionescuPAMI2014}           &
IH,O                       & F             & SF         &
MC,D,ST    & F             & No            & V          & C          &
SF,C       & G,AU,EU       & SF            & 3.6M       & 11         &
CBP        & BP            & I             & Multiple   \\ \hline

LEEDS SPORTS \citep{Johnson10}              &
IH,O                       & F             & SF         &
C          & F             & No            & V          & U          &
C,SF       & AY,E,G        & SF            & 2000 F     & U1         &
CBP        & BP            & I             & Fer.          \\ \hline

Pascal VOC people \citep{pamipascal}        &
IH,HH,O                    & F             & SF         &
C          & F             & Some          & V          & U          &
C,SF       & A,E,G         & SF            & 632 F      & U1         &
CBP        & BP            & B             & Pascal        \\ \hline

UIUC People \citep{tran2010improved}        &
IH,O                       & F             & SF         &
C          & F             & No            & V          & U          &
C,SF       & AY,E,G        & SF            & 593 F      & U1         &
CBP        & BP            & I             & Fer.          \\ \hline

Buffy \citep{Ferrari08}                     &
IH,HH,O                    & U             & SF         &
C          & U             & Some          & V          & U          &
C,SF       & AY,G          & SF            & 748 F      & UM         &
CBP        & BP            & B             & Fer.       \\ \hline

Parse \citep{ramanan2006learning}           &
IH,O                       & F             & SF         &
C          & F             & No            & V          & U          &
C,SF       & AY,E,G        & SF            & 305 F      & U1         &
CBP        & BP            & MI            & Ram.       \\ \hline

MPII Pose \citep{andriluka14benchmark}      &
IH,O                       & F             & SF         &
C          & F             & Yes           & V          & U          &
C,SF       & A,E,G         & SF            & 25000 F    & U1         &
CBP        & BP            & MI            & Fer.       \\ \hline

FLIC Pose \citep{modec13}                   &
IH,HH,O                    & U             & SF         &
C          & U             & Some          & V          & U          &
C,SF       & AY,E,G        & SF            & 5003 F     & UM         &
CBP        & BP            & MI            & Sap.     \\ \hline

H3D \citep{bourdev2009poselets}                   &
IH,HH,O                    & U             & SF         &
C          & U             & Some          & V          & U          &
C,SF       & A,E,G         & SF            & 520 F      & US         &
CBP        & BP            & MI            & Sap.     \\ \hline

CDG2011 \citep{guyon-mva-2014}               &
IH                         & Mixed         & Mixed      &
C,D        & MU            & Few           & Fixed      & C          &
ST,C       & E,G           & No            & 50000 G    & 20         &
CDG11      & Mixed         & I             & L            \\ \hline

CDG2013 \citep{escalera2013multi}           &
             IH            & H             & Dynamic    &
A,C,D,ST   & U             & No            & F          & C          &
ST,C       & G             & No            & 13858 G    & 27         &
20         & E             & I             & L          \\ \hline

3DIG \citep{sadeghipourBMVC2012}            &
IH                         & H             & Dynamic    &
C,D,ST     & U             & No            & F          & C          &
ST,U       & AY,E,G        & Yes           & 1739 G     & 29         &
20         & I             & I             & F             \\ \hline

HuPBA8K+ \citep{neuro}                      &
IH,HH                      & F             & Dynamic    &
C          & F             & Yes           & Fixed      & C & ST,U
& G, AY         & No            & 8000 F     & 14
& 11         & A             & B             & L          \\
\hline

HOHA \citep{laptev2008learning}             &
IH,HH,HO                   & F             & Dynamic    &
C          & F             & Some          & V          & U          &
SD,C       & AY,E,G        & Yes           & 475 V      & UM         &
8          & A             & B             & CA       \\ \hline

KTH \citep{schuldt2004recognizing}          &
IH                         & F             & Dynamic    &
G          & F             & No            & SV         & C          &
ST,U       & G             & Yes           & 2391 A     & 25         &
6          & A             & I             & CA       \\ \hline

MSRC-12 \citep{fothergillCHI2012}           &
IH                         & F             & Dynamic    &
ST         & F             & No            & F          & C          &
N/A        & A,E,G         & No            & 719359 F   & 30         &
12         & E,I           & I             & F             \\ \hline

G3D \citep{bloomCVPRW2012}                  &
IH                         & F             & Dynamic    &
C,D,ST     & F             & No            & Fixed      & C          &
ST,C       & U             & No            & 80000 F    & 10         &
20         & A             & I             & F             \\ \hline

ASLLVD \citep{neidleLREC2012}               &
IH                         & H             & Dynamic    &
MC         & U             & No            & F          & C          &
ST,U        & G             & Yes           & 9794 G     & 6          &
3314       & S             & I             & RRC        \\ \hline

UTA ASL \citep{conly_petra2013}            &
IH                         & H             & Dynamic    &
C,D        & U             & No            & F          & C          &
ST,U       & E,A           & Yes           & 1313 G     & 2          &
1113       & S             & I             & RCC           \\ \hline

ChAirGest \citep{chairgest}          &
IH                         & H             & Dynamic    &
ChAir      & U             & No            & F          & C          &
ST,C       & U             & No            & 1200 G     & 10         &
10         & I,E           & I             & F,ATSR        \\ \hline

SKIG \citep{liuIJCAI2015}                   &
IH                         & H             & Dynamic    &
C,D        & A             & No            & F          & C          &
ST,CU      & U             & Yes           & 1080 G     & 10         &
6          & I,E           & I             & CA            \\ \hline

6DMG \citep{chenMSC2012}                    &
IH                         & H             & Dynamic    &
6DMG       & H             & No            & F          & C          &
N/A        & G,AU,EU       & Yes           & 5600 G     & 28         &
20         & I             & I             & CA            \\ \hline

MSRGesture3D \citep{kurakinEUSIPCO2012}     &
IH                         & H             & Dynamic    &
B          & H             & No            & Fixed      & C          &
N/A        & U             & Yes           & 336 G      & 10         &
12         & S             & I             & CA            \\ \hline

NATOPS \citep{song_fg2011b}                 &
IH                         & U             & Dynamic    &
S,D,B      & U             & No            & Fixed      & C          &
ST,U       & U             & Yes           & 9600 G     & 20         &
24         & E             & I             & CA           \\ \hline

NTU Dataset \citep{renACMICM2011}           &
IH                         & H             & Static     &
C,D        & U             & No            & F          & C          &
SF,C       & G             & SF            & 1000 G     & 10         &
10         & HS            & I             & CA           \\ \hline

Keck Gesture \citep{linICCV2009}            &
IH                         & H             & Dynamic    &
C          & F             & No            & F          & C          &
ST,U        & No            & Yes           & 294 G      & 3          &
14         & E             & I             & CA         \\ \hline

Cambridge Gesture \citep{kimCVPR2007}       &
IH                         & H             & Dynamic    &
C          & H             & No            & F          & C          &
ST,U        & U             & Yes           & 900 G      & 2          &
9          & E             & I             & CA         \\ \hline

\citep{trieschPAMI2001}                     &
IH                         & H             & Static     &
G          & H             & No            & Fixed      & C          &
SF,CU      & U             & SF            & 717 G      & 24         &
10         & HS            & I             & CA         \\ \hline

%Dataset                                    &
%Modalities & Upper/        & Occl.         & Viewp.     & Controlled &
%Background & Variab.       & Seg.          & Amount     & Subjects   &
%Classes    & Type          & Collab.       & Eval.      \\ \hline


\end{tabular}
\caption{Datasets.
``Occl.'' stands for ``occlusions''.
``View. variab.'' stands for ``viewpoint variabilities''.
``Variab.'' stands for ``variabilities in gender, age, ethnicity''.
``Seg.'' stands for ``segmented''.
``Indiv/Collab.'' stands for ``individual or collaborative gestures''.
``Type'' stands for ``gesture type''.
``Eval.'' stands for ``evaluation''.}
\label{tbl::datasets}
\end{table}

\end{landscape}


\begin{table}[h!]\centering
\tiny
\begin{tabular}{|l|l|l|}
\hline
Taxonomy Attribute                    & Acronym/Abbreviation & Meaning           \\ \hline
Actors/Objects                        & HH                   & human-human interactions \\ \hline
Actors/Objects                        & IH                   & isolated human    \\ \hline
Actors/Objects                        & O                    & humans with objects    \\ \hline
Body parts                            & F                    & full body         \\ \hline
Body parts                            & H                    & hands             \\ \hline
Static/Dynamic & SF & Subjects are in motion, but each frame is individually classified \\ \hline
Modalities & 6DMG  & WorldViz PPT-X4 (position + 3D orientation) +  Wii Remote Plus (acceleration and angular speeds) \\ \hline
Modalities                            & A                    & audio             \\ \hline
Modalities                            & B                    & binary segmentation mask \\ \hline
Modalities                            & C                    & RGB (color)       \\ \hline
Modalities & ChAir & RGB, depth, skeletal, four inertial motion units  \\ \hline
Modalities                            & D                    & depth             \\ \hline
Modalities                            & G                    & grayscale         \\ \hline
Modalities       & MC            & multiple cameras         \\ \hline
Modalities                            & S                    & stereo images     \\ \hline
Modalities                            & ST                   & skeletal tracking \\ \hline
Viewing field                         & A                    & arm and hand      \\ \hline
Viewing field                         & E                    & egocentric        \\ \hline
Viewing field                         & F                    & full body         \\ \hline
Viewing field                         & MU     & upper body in most cases        \\ \hline
Viewing field                         & U                    & upper body        \\ \hline
Occlusions                            &                      &                   \\ \hline
Viewpoints                            & F                    & Frontal           \\ \hline
Viewpoints                            & Fixed                & Fixed viewpoint for each class\\ \hline
Viewpoints                            & SV                   & Fixed viewpoint for some
classes, variable for other classes \\ \hline
Viewpoints                            & V                    & Variable viewpoint\\ \hline
Controlled/Uncontrolled               & C                    & Controlled        \\ \hline
Controlled/Uncontrolled               & U                    & Uncontrolled      \\ \hline
Background                            & CU  & some cluttered, some uncluttered  \\ \hline
Background                            & C                    & cluttered         \\ \hline
Background                            & D                    & dynamic           \\ \hline
Background  & SF    & each frame is individually classified, so background is seen only from a single frame \\ \hline
Background                            & SD                   & static in some cases, dynamic in some cases           \\ \hline
Background                            & ST                   & static            \\ \hline
Background                            & U                    & uncluttered       \\ \hline
Variabilities in gender/age/ethnicity & A                    & variabilities in age \\ \hline
Variabilities in gender/age/ethnicity & AU & unspecified whether there are variabilities in age \\ \hline
Variabilities in gender/age/ethnicity & AY                   & mostly non-senior adults \\ \hline
Variabilities in gender/age/ethnicity & E                    & variabilities in ethnicity \\ \hline
Variabilities in gender/age/ethnicity & EU & unspecified whether there are variabilities in ethnicity \\ \hline
Variabilities in gender/age/ethnicity & G                    & variabilities in gender \\ \hline
Variabilities in gender/age/ethnicity & U                    & unspecified             \\ \hline
Segmented/Unsegmented                 & SF  & each frame is individually classified \\ \hline
Amount of data                        & A                    & action samples    \\ \hline
Amount of data                        & F                    & frames            \\ \hline
Amount of data                        & G                    & gesture samples   \\ \hline
Amount of data                        & V                    & video clips       \\ \hline
Number of subjects  & U1   & Unspecified, but most subjects appear in only one sample \\ \hline
Number of subjects  & UM   & Unspecified, but most subjects appear in several samples \\ \hline
Number of subjects  & US   & Unspecified, but some subjects appear in more than one sample \\ \hline
Classes                        & CBP                  & continuous space of body \\ \hline
Classes        & CDG11      & about 300, but broken into subsets of 8-12 classes \\ \hline
Gesture type                          & A                    & action            \\ \hline
Gesture type                          & BP                   & body pose         \\ \hline
Gesture type                          & D                    & deictic           \\ \hline
Gesture type                          & E                    & emblematic        \\ \hline
Gesture type                          & HS                   & handshape         \\ \hline
Gesture type                          & I                    & iconic            \\ \hline
Gesture type                          & S                    & sign              \\ \hline
Individual or collaborative           & B & both individual and collaborative    \\ \hline
Individual or collaborative           & C                    & collaborative     \\ \hline
Individual or collaborative           & I                    & individual        \\ \hline
Individual or collaborative           & MI               & mostly individual     \\ \hline
Evaluation criteria  & ATSR         & Defined in \citep{chairgest}, based on difference between detected and  \\
                     &              & ground truth endpoints, normalized by duration of the gesture \\ \hline
Evaluation criteria                   & CA         & Classification accuracy \\ \hline
Evaluation criteria                   & F          & F-score                 \\ \hline
Evaluation criteria  & Fer.           & Defined in \citep{Ferrari08}, checks if detected endpoints are within distance of  \\
                     &                &  half length (of the body part in question) from the ground truth position. \\ \hline
Evaluation criteria                   & L          & Levenshtein distance\\ \hline
Evaluation criteria     & MPJPE & Mean per-joint position error (measured as Euclidean distance). \\ \hline
Evaluation criteria     & Pascal & At least 50\% overlap of bounding boxes on all body parts. \\ \hline
Evaluation criteria     & Ram. & Defined in \citep{ramanan2006learning}, average negative log likelihood of correct pose. \\ \hline
Evaluation criteria     & RCC & Defined in \citep{wang_sga2010}, based on rank of the correct class for each test sign. \\
                        &     & For any R, report percentage of test signs for which the correct class was in the top R classes. \\ \hline
Evaluation criteria     & Sap. & Defined in \citep{modec13}, Accuracy is based on (variable) threshold pixel distance between    \\
                        &      & joint location and ground truth, scaled so that the torso length in the ground truth is 100 pixels. \\ \hline
\end{tabular}
\caption{Acronyms and abbreviations used in the table of datasets.} \label{tbl::dataset_acronyms}
\end{table}




\section{Gesture recognition challenges}

In this section we review the series of gesture and action
recognition challenges organized by ChaLearn from 2011 to 2015, as
well as other international challenges related to gesture
recognition.


\subsection{First ChaLearn Gesture Recognition Challenge
(2011-2012): One shot learning}

ChaLearn launched in 2012 a challenge with prizes donated by
Microsoft using datasets described in~\citep{guyon-mva-2014}. We
organized two rounds in conjunction with the CVPR conference
(Providence, Rhode Island, USA, June 2012) and the ICPR conference
(Tsukuba, Japan, November 2012). Details on the challenge setting and results are found in~\citep{guyon-lncs-2013}. We briefly summarize the setting and results.

\subsubsection{2011-2012 Challenge Protocol and Evaluation}

The task of the challenge was to
built a learning system capable of learning a gesture classification problem from a {\bf single training
example} per class, from dynamic video data complemented by a depth map obtained with \kinect. The rationale behind this setting is that, in many computer interface applications to gesture recognition, users want to customize the interface to use their own gestures. Therefore they should be able to retrain the interface using a small vocabulary of their own gestures. We have also experimented with other use cases in gaming and teaching gesture vocabularies. Additionally, the problem of one-shot-learning is of intrinsic interest in machine learning and the solutions deviced could carry over to other applications. It is in a certain way an extreme case of transfer learning.

To implement this setting in the challenge, we collected a large dataset consisting of batches, each batch corresponding to the video recording of short sequences of gestures performed by the same person. The gestures in one batch pertained to a small vocabulary of gestures taken from a variety of application domains (sign language for the deaf, traffic signals, pantomimes, dance postures, etc.). During the development phase, the participants had access to hundreds of batches of diverse gesture vocabularies. This played the role of ``source domain data" in the transfer learning task. The goal of the participants was to get ready to receive new batches from different gesture performers and different gesture vocabularies, playing the role of ``transfer domain data". Their system would then need to learn from a single example of gesture performed by the particular performer, before being capable of recognizing the rest of the gestures in that batch. The full dataset is available from \url{http://gesture.chalearn.org/data}.

More specifically, each batch was split into a training set (of
one example for each gesture) and a test set of short sequences of
one to 5 gestures. Each batch contained gestures from a different
small vocabulary of 8 to 12 gestures, for instance diving signals,
signs of American Sign Language representing small animals,
Italian gestures, etc. The test data labels were provided for the
development data only (source domain data), so the participants
could self-evaluate their systems and pre-train parts of it as is
expected from transfer learning methods. The data also included 20
validation batches and 20 final evaluation batches as transfer
domain data used by the organizers to evaluate the participants.
In those batches, only the labels of the training gestures (one
example each) was provided, the rest of the gesture sequences were
unlabelled and the goal of the participants was to predict those
labels. We used the Kaggle platform to manage
submissions\footnote{For round 1:
\url{http://www.kaggle.com/c/GestureChallenge}. For round 2:
\url{http://www.kaggle.com/c/GestureChallenge2}.} The participants
received immediate feed-back on validation data on a on-line
leaderboard. The final evaluation was carried out on the final
evaluation data, and those results were only revealed after the
challenge was over. The participants had a few days to train their
systems and upload their predictions. Prior to the end of the
development phase, the participants were invited to submit
executable software for their best learning system to a software
vault. This allowed the competition organizers to check their
results and ensure the fairness of the competition.

To compare prediction labels for gesture sequences to the truth
values, we used the generalized Levenshtein distances (each gesture
counting as one token). The final evaluation score was computed as
the sum of such distances for all test sequences, divided by the
total number of gestures in the test batch. This score is
analogous to an error rate. However, it can exceed one.
Specifically, for each video, the participants provided an ordered
list of labels $R$ corresponding to the recognized gestures. We
compared this list to the corresponding list of labels T in the
prescribed list of gestures that the user had to play. These are
the ``true" gesture labels (provided that the users did not make
mistakes). We computed the generalized Levenshtein distance $L(R,
T)$, that is the minimum number of edit operations (substitution,
insertion, or deletion) that one has to perform to go from $R$ to
$T$ (or vice versa). The Levenhstein distance is also known as
``edit distance". For example: $L([1 2 4], [3 2]) = 2; L([1], [2])
= 1; L([2 2 2], [2]) = 2$.

We provided code to browse though the data, a library of computer
vision and machine learning techniques written in \matlab
featuring examples drawn from the challenge datasets, and an
end-to-end baseline system capable of processing challenge data
and producing a sample submission. The competition pushed the
state of the art considerably. The participants narrowed down the
gap in performance between the baseline recognition system
initially provided ($\simeq60\%$ error) and human performance
($\simeq2\%$ error)  by reaching $\simeq7\%$ error in the second
round of the challenge. There remains still much room for
improvement, particularly to recognize static postures and subtle
finger positions.

\subsubsection{2011-2012 Challenge data}

The datasets are described in details in a companion
paper~\citep{guyon-mva-2014}. Briefly, the data are organized in
batches: development batches devel01-480, validation batches
valid01-20, and final evaluation batches final01-20 (for round 1)
and final21-40 (for round 2). For the development batches, we
provided all the labels. To evaluate the performances on
``one-shot-learning" tasks, the valid and final batches were
provided with labels only for {\bf one example of each gesture
class} in each batch (training examples). The goal was to
automatically predict the gesture labels for the remaining
unlabelled gesture sequences (test examples).

Each batch includes 100 recorded gestures grouped in sequences of
1 to 5 gestures performed by the same user. The gestures are drawn
from a small vocabulary of 8 to 12 unique gestures, which we call
a ``lexicon". For instance a gesture vocabulary may consist of the
signs to referee volleyball games or the signs to represent small
animals in the sign language for the deaf. We selected lexicons
from nine categories corresponding to various settings or
application domains (Figure~\ref{fig:many_faces}):

\begin{enumerate}
\item {\bf Body language} gestures (like scratching your head, crossing your arms).
\item {\bf Gesticulations} performed to accompany speech.
\item {\bf Illustrators} (like Italian gestures).
\item {\bf Emblems} (like Indian Mudras).
\item {\bf Signs} (from sign languages for the deaf).
\item {\bf Signals} (like referee signals, diving signals, or Marshalling signals to guide machinery or vehicle).
\item {\bf Actions} (like drinking or writing).
\item {\bf Pantomimes} (gestures made to mimic actions).
\item {\bf Dance postures.}

\end{enumerate}
During the challenge, we did not disclose the identity of the
lexicons and of the users.
\begin{figure*}
    \begin{center}
    \includegraphics*[width=\linewidth]{MANY_FACES3.png}
    \end{center}
    \caption{\label{fig:many_faces} {\bf Types of gestures.} We created a classification of gesture types according to purpose defined by three complementary axes: communication, expression and action. We selected 85 gesture vocabularies, including Italian gestures, Indian Mudras, Sign language for the deaf, diving signals, pantomimes, and body language.}
\end{figure*}

\subsubsection{2011-2012 Challenge Results}

The results of the top ranking participants were checked by the
organizers who reproduced their results using the code provided by
the participants before they had access to the final evaluation
data. All of them passed successfully the verification process.
These results are shown in Tables~\ref{tab:resu} and
\ref{tab:resu2}.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|c|c|}
\hline
Team & Public score & Private score & For comparison score\\
 & on validation set & on final set \#1 & on final set \#2 \\
\hline\hline
Alfnie  &   0.1426  & 0.0996 & 0.0915 \\
Pennect &   0.1797  & 0.1652 & 0.1231 \\
OneMillionMonkeys   &   0.2697  & 0.1685 & 0.1819\\
Immortals   &   0.2543  & 0.1846 & 0.1853 \\
Zonga   &   0.2714  & 0.2303 & 0.2190 \\
Balazs Godeny   &   0.2637 &    0.2314 & 0.2679 \\
SkyNet  &   0.2825  & 0.2330 & 0.1841 \\
XiaoZhuwWudi    &   0.2930  & 0.2564 & 0.2607 \\
Baseline method 1 & 0.5976 & 0.6251 & 0.5646 \\
\hline
\end{tabular}
\end{center}
\caption{\label{tab:resu} Results of round 1. In round 1 the
baseline method was a simple template matching method (see text).
For comparison, we show the results on the final set number 2 not
available in round 1.}
\end{table}


\begin{table}
\begin{center}
\begin{tabular}{|l|c|c|c|}
\hline
Team & Public score & For comparison score & Private score \\
 & on validation set & on final set \#1 & on final set \#2 \\
\hline\hline
Alfnie  &   0.0951  & 0.0734 & 0.0710 \\
Turtle Tamers   &   0.2001  & 0.1702 & 0.1098 \\
Joewan  &   0.1669  & 0.1680 & 0.1448 \\
Wayne Zhang &   0.2814  & 0.2303 & 0.1846 \\
Manavender  &   0.2310  & 0.2163 & 0.1608 \\
HIT CS  &   0.1763 & 0.2825 &   0.2008 \\
Vigilant    &   0.3090  & 0.2809 & 0.2235 \\
Baseline method 2 & 0.3814 & 0.2997 & 0.3172 \\
\hline
\end{tabular}
\end{center}
\caption{\label{tab:resu2} Results of round 2. In round 2, the
baseline method was the ``Principal Motion" method (see text).}
\end{table}

\subsubsection{2011-2012 Challenge, Summary of the Winner Methods}

The results of the challenge are analyzed in details, based on
papers published in this special topic and on descriptions
provided by the top ranking participants in their fact
sheets~\citep{guyon-lncs-2013}. We briefly summarize notable
methods below.

\textbf{The winner of both rounds} (Alfonso Nieto Casta\~non of
Spain, a.k.a. alfnie) used a novel technique called ``Motion
Signature analyses", inspired by the neural mechanisms underlying
information processing in the visual system. This is an
unpublished method using a sliding window to perform
simultaneously recognition and temporal segmentation, based solely
on depth images. The method, described by the authors as a
``Bayesian network", is similar to a Hidden Markov Model (HMM). It
performs simultaneous recognition and segmentation using the
Viterbi algorithm.  The preprocessing steps include Wavelet
filtering replacement of missing values and outlier detection.
Notably, this method is one of the fastest despite the fact that
he implemented it in Matlab (close to real time on a regular
laptop). The author claims that it has linear complexity in image
size, number of frames, and number of training examples.

\textbf{The second best ranked participants} (team Pennect of
Universit of Pennsylvania, USA, in round 1 and team Turtle Tamers
of Slovakia, in round 2) used very similar methods and performed
similarly. The second team published their results in this special
topic~\citep{konecny-2014}. Both methods are based on an HMM-style
model using HOG/HOF features to represent movie frames. They
differ in that Pennect used RGB images only while Turtle Tamers
used both RGB and depth. Another difference is that Pennect used
HOG/HOF features at 3 different scales while Turtle Tamers created
a bag of features using K-means clustering from only 40x40
resolution and 16 orientation bins. Pennect trained a one-vs-all
linear classifier for each frame in every model and used the
discriminant value as a local state score for the HMM while Turtle
Tamers used a quadratic-chi kernel metric for comparing pairs of
frames in the training and test movie. As preprocessing, Pennect
uses mean subtraction and compensates for body translations while
Turtle Tamers replaces the missing values by the median of
neighboring values. Both teams claim a linear complexity in number
of frames, number of training examples, and image size. They both
provided Matlab software that processes all the batches of the
final test set on a regular laptop in a few hours.

The next best ranked participants (who won \textbf{third place in
round 2}), the Joewan team, who published in this special
topic~\citep{wan-2013}, used a slightly different approach. They
relied on the motion segmentation method provided by the
organizers to pre-segment videos. They then represented each video
as a bag of 3D MOSIFT features (integrating RGB and depth data),
and then used a nearest neighbor classifier. Their algorithm is
super-quadratic in image size, linear in number of frames per
video, and linear in number of training examples. The method is
rather slow and takes over a day to process all the batches of the
final test set on a regular laptop.

\textbf{The third best ranked team} in round 1 (OneMillionMonkeys)
also used an HMM in which a state is created for each frame of the
gesture exemplars. This data representation is based on edge
detection in each frame.  Edges are associated with several
attributes including the X/Y coordinates, their orientation, their
sharpness, their depth and location in an area of change.  To
provide a local state score to the HMM for test frames,
OneMillionMonkeys calculated the joint probability of all the
nearest neighbors in training frames using a Gaussian model. The
system works exclusively from the depth images. The system is one
of the slowest proposed. Its processing speed is linear in number
of training examples but quadratic in image size and number of
frames per video. The method is rather slow and takes over a day
to process all the batches of the final test set on a regular
laptop.

Methods robust against translation include those of
Joewan~\citep{wan-2013} and Immortals/Manavender (this is the same
author under two different pseudonyms for round 1 and round
2). The team Immortals/Manavender
published their method in this special
topic~\citep{malgireddy-2013}. Their representations are based on a bag of
visual words, inspired by techniques used in action
recognition~\citep{Laptev:2005}. Such representations are inherently
shift invariant. The slight performance loss in translated data
may be due to partial occlusions.

Although the team Zonga did not end up ranking among top ranking
participants, the authors, who published their method in this
special topic, proposed a very original method and ended up
winning the best paper award. Notably, their outperformed all
baseline methods early on in the challenge by applying their
method without tuning it to the tasks of the challenge and
remained at the top of the leaderboard for several weeks. The used
a novel technique based on tensor geometry, which provides a data
representation exhibiting desirable invariances and yields a very
discriminating structure for action recognition.

ChaLearn also organized demonstration competitions of gesture
recognition systems using \kinect, in conjunction with those
events. Novel data representations were proposed to tackle with
success, in real time, the problem of hand and finger posture
recognition. The demonstration competition winners showed systems
capable of accurately tracking in real time hand postures in
application for touch free exploration of 3D medical images for
surgeons in the operating room, finger spelling (sign language for
the deaf), virtual shopping, and game controlling. Combining the
methods proposed in the demonstration competition tackling the
problem of hand postures and those of the quantitative evaluation
focusing on the dynamics of hand and arm movements is a promising
direction of future research. For a long lasting impact, the
challenge platform, the data and software repositories have been
made available for further
research\footnote{\url{http://gesture.chalearn.org/}}.


\subsection{ChaLearn Multimodal Gesture Recognition Challenge
2013}


The focus of this second challenge was on \emph{multiple instance,
user independent learning of gestures from multimodal data}, which
means learning to recognize gestures from several instances for
each category performed by different users, drawn from a
vocabulary of 20 gesture
categories\citep{escalera2013multi,escalera2013multi2}. A gesture
vocabulary is a set of unique gestures, generally related to a
particular task. In this challenge we focus on the recognition of
a vocabulary of 20 Italian cultural/anthropological
signs~\citep{escalera2013multi}, see Figure~\ref{fig:categories} for one
example of each Italian gesture.


\begin{figure*}[!ht]\centering\scriptsize
\begin{tabular}{p{2.6cm}p{2.6cm}p{2.6cm}p{2.6cm}p{2.6cm}}
   \includegraphics[height=4cm,width=2.6cm]{01.png}&
   \includegraphics[height=4cm,width=2.6cm]{02.png}& \includegraphics[height=4cm,width=2.6cm]{03.png} &
\includegraphics[height=4cm,width=2.6cm]{04.png}&\includegraphics[height=4cm,width=2.6cm]{05.png}\\
   (1) \emph{Vattene}&(2) \emph{Viene qui}&(3) \emph{Perfetto}&(4) \emph{E un furbo}&(5) \emph{Che due palle}\\
   \includegraphics[height=4cm,width=2.6cm]{06.png}& \includegraphics[height=4cm,width=2.6cm]{07.png} &
\includegraphics[height=4cm,width=2.6cm]{08.png}&\includegraphics[height=4cm,width=2.6cm]{09.png}&
   \includegraphics[height=4cm,width=2.6cm]{10.png}\\\\
   (6) \emph{Che vuoi}&(7) \emph{Vanno d'accordo}&(8) \emph{Sei pazzo} &(9) \emph{Cos hai combinato}&(10) \emph{Nonme me friega niente}\\
    \includegraphics[height=4cm,width=2.6cm]{11.png} &
\includegraphics[height=4cm,width=2.6cm]{12.png}& \includegraphics[height=4cm,width=2.6cm]{13.png}&
   \includegraphics[height=4cm,width=2.6cm]{14.png}& \includegraphics[height=4cm,width=2.6cm]{15.png}\\
  (11) \emph{Ok}&(12) \emph{Cosa ti farei} &   (13) \emph{Basta}&(14) \emph{Le vuoi prendere}&(15) \emph{Non ce ne piu}\\

\includegraphics[height=4cm,width=2.6cm]{16.png}&
   \includegraphics[height=4cm,width=2.6cm]{17.png}&
   \includegraphics[height=4cm,width=2.6cm]{18.png}& \includegraphics[height=4cm,width=2.6cm]{19.png} &
\includegraphics[height=4cm,width=2.6cm]{20.png}\\
(16) \emph{Ho fame}&  (17) \emph{Tanto tempo fa}&(18)
\emph{Buonissimo}&(19) \emph{Si sono messi d'accordo}&(20)
\emph{Sono
   stufo}\\
\end{tabular}
   \caption{\scriptsize Data set gesture categories.}
\label{fig:categories}
\end{figure*}

\subsubsection{2013 Challenge Data}

In all the sequences, a single user is recorded in front of a
\kinect, performing natural communicative gestures and speaking in
fluent Italian. The main characteristics of the
dataset of gestures are: \\
\noindent $\bullet$ $13.858$ gesture samples recorded with the
\kinect camera, including audio, skeletal model, user mask, RGB,
and depth images. \\$\bullet$ RGB video stream, 8-bit VGA
resolution (640$\times$480) with a Bayer color filter, and depth
sensing video stream in VGA resolution (640$\times$480) with
11-bit. Both are acquired in 20 fps on average.\\$\bullet$ Audio
data is captured using \kinect 20 multiarray
microphone.\\$\bullet$ A total number of 27 users appear in the
data set.\\$\bullet$ The data set contains the following number of
sequences, development: 393 ($7.754$ gestures), validation: 287
($3.362$ gestures), and test: 276 ($2.742$ gestures), each
sequence lasts between 1 and 2 minutes and contains between 8 and
20 gesture samples, around $1.800$ frames. The total number of
frames of the data set is $1.720.800$.\\$\bullet$ All the gesture
samples belonging to 20 main gesture categories from an Italian
gesture dictionary are annotated at frame level indicating the
gesture label.\\$\bullet$ 81\% of the participants were Italian
native speakers, while the remaining 19\% of the users were not
Italian, but Italian-speakers.\\$\bullet$ All the audio that
appears in the data is from the Italian dictionary. In addition,
sequences may contain distractor words and gestures, which are not
annotated since they do not belong to the main dictionary of 20
gestures.

This dataset, available at \url{http://sunai.uoc.edu/chalearn},
presents various features of interest as listed in
Table~\ref{features}. Examples of the provided visual modalities are shown in Figure~\ref{fig:modalities}.\\


\begin{figure*}[!ht]\centering
\begin{tabular}{cccc}
   \includegraphics[height=2.5cm]{data01.png}&
   \includegraphics[height=2.5cm]{data02.png}&
   \includegraphics[height=2.5cm]{data03.png}&
   \includegraphics[height=2.5cm]{data04.png}\\
RGB&Depth&User mask&Skeletal model\\
\end{tabular}
   \caption{\scriptsize Different data modalities of the provided data set.}
\label{fig:modalities}
\end{figure*}


\begin{table}\scriptsize
\centering \caption{\scriptsize Easy and challenging aspects of
the data.}
\begin{tabular}{|p{8cm}|}
\hline
\textbf{Easy}\\
\hline Fixed camera\\ Near frontal view acquisition\\ Within a
sequence the same user \\ Gestures performed mostly by arms and
hands \\ Camera framing upper body \\ Several available
modalities: audio, skeletal model, user mask, depth, and RGB\\
Several instances of each gesture for training\\
Single person present in the visual field\\ \hline
\textbf{Challenging}\\
\hline \emph{Within each sequence}:\\ Continuous gestures without
a resting pose\\ Many gesture instances are present\\
Distracter gestures out of the vocabulary may be present in terms
of both gesture and audio\\\emph{Between sequences}:
\\High inter and intra-class variabilities of gestures in terms of both gesture and audio\\
Variations in background, clothing, skin color, lighting,
temperature, resolution\\ Some parts of the body may be
occluded\\Different Italian dialects \\
\hline\end{tabular}\label{features}
\end{table}



\subsubsection{2013 Challenge Protocol and Evaluation}


As in our previous 2011-2012 challenge, it consisted of two main
components: a development phase (April 30th to Aug 1st) and a
final evaluation phase (Aug 2nd to Aug 15th). The submission and
evaluation of the challenge entries was via the $Kaggle$
platform~\footnote{\scriptsize
https://www.kaggle.com/c/multimodal-gesture-recognition}. The
official participation rules were provided on the website of the
challenge. In addition, publicity and news on the ChaLearn
Multimodal Gesture Recognition Challenge were published in
well-known online platforms, such as LinkedIn, Facebook, Google
Groups and the ChaLearn website.



 During the development phase,
the participants were asked to build a system capable of learning
from several gesture samples a vocabulary of 20 Italian sign
gesture categories. To that end, the teams received the
development data to train and self-evaluate their systems. In
order to monitor their progress they could use the validation data
for which the labels were not provided. The prediction results on
validation data could be submitted online to get immediate
feed-back. A real-time leaderboard showed to the participants
their current standing based on their validation set predictions.

During the final phase, labels for validation data were published
and the participants performed similar tasks as those performed in
previous phase, using the validation data and training data sets
in order to train their system with more gesture instances. The
participants had only few days to train their systems and upload
them. The organizers used the final evaluation data in order to
generate the predictions and obtain the final score and rank for
each team. At the end, the final evaluation data was revealed, and
authors submitted their own predictions and fact sheets to the
platform.

As an evaluation metric we also used the Levenshtein distance
described in previous section. A public score appeared on the
leaderboard during the development period and was based on the
validation data. Subsequently, a private score for each team was
computed on the final evaluation data released at the end of the
development period, which was not revealed until the challenge was
over. The private score was used to rank the participants and
determine the prizes.


\subsubsection{2013 Challenge Results}

The challenge attracted high level of participation, with a total
of 54 teams and near 300 total number of entries. This is a good
level of participation for a computer vision challenge requiring
very specialized skills. Finally, 17 teams successfully submitted
their prediction in final test set, while providing also their
code for verification and summarizing their method by means of a
fact sheet questionnaire.

After verifying the codes and results of the participants, the
final scores of the top rank participants on both validation and
test sets were made public: these results are shown in
Table~\ref{results1}, where winner results on the final test set
are printed in bold. In the end, the final error rate on the test
data set was around 12\%.

\begin{table}\scriptsize
\centering \caption{\scriptsize Top rank results on validation and
test sets.}
\begin{tabular}{|c|c|c|} \hline
\textbf{TEAM }&  \textbf{Validation score} & \textbf{Test score} \\
\hline IVA MM & 0.20137& \textbf{0.12756}
\\ \hline
WWEIGHT &0.46163&\textbf{0.15387}
\\ \hline
ET &0.33611&\textbf{0.16813}
\\ \hline
MmM &0.25996&0.17215
\\ \hline
PPTK &0.15199&0.17325
\\ \hline
LRS &0.18114&0.17727
\\ \hline
MMDL & 0.43992&0.24452
\\ \hline
TELEPOINTS&0.48543& 0.25841
\\ \hline
CSI MM &0.32124&0.28911
\\ \hline
SUMO &0.49137&0.31652
\\ \hline
GURU &0.51844&0.37281
\\ \hline
AURINKO &0.31529&0.63304
\\ \hline
STEVENWUDI &1.43427&0.74415
\\ \hline
JACKSPARROW &0.86050&0.79313
\\ \hline
JOEWAN &0.13653&0.83772
\\ \hline
MILAN KOVAC&0.87835& 0.87463
\\ \hline
IAMKHADER&    0.93397& 0.92069
\\ \hline
\end{tabular}\label{results1}
\end{table}


\subsubsection{2013 Challenge Summary of the Winner Methods}

Table~\ref{resumenTabla} shows the summary of the strategies
considered by each of the top ranked participants on the test set.
Interestingly, the three top ranked participants agree in the
modalities and segmentation strategy considered, although they
differ in the final applied classifier. Next, we briefly describe
in more detail the approach designed by the three winners of the
challenge.

\begin{table*}\scriptsize
\centering \caption{\scriptsize Team methods and results. Early
and late refer to early and late fusion of features/classifier
outputs. HMM: Hidden Markov Models. KNN: Nearest Neighbor. RF:
Random Forest. Tree: Decision Trees. ADA: Adaboost variants. SVM:
Support Vector Machines. Fisher: Fisher Linear Discriminant
Analysis. GMM: Gaussian Mixture Models. NN: Neural Networks. DGM:
Deep Boltzmann Machines. LR: Logistic Regression. DP: Dynamic
Programming. ELM: Extreme Learning Machines.}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline
\textbf{TEAM }&  \textbf{Test score} & \textbf{Rank} &
\textbf{Modalities}& \textbf{Segmentation} &\textbf{Fusion} & \textbf{Classifier}\\
\hline IVA MM & 0.12756 &1 & Audio,Skeleton &Audio& None &
HMM,DP,KNN
\\ \hline
WWEIGHT &0.15387 &2 &  Audio,Skeleton & Audio &Late  &  RF,KNN
\\ \hline
ET &0.16813 &3 &  Audio,Skeleton & Audio &  Late &Tree,RF,ADA
\\ \hline
MmM &0.17215 &4& Audio,RGB+Depth& Audio  & Late&
SVM,Fisher,GMM,KNN
\\ \hline
PPTK &0.17325 &5& Skeleton,RGB,Depth & Sliding windows& Late&
GMM,HMM
\\ \hline
LRS &0.17727 &6& Audio,Skeleton,Depth& Sliding windows &Early& NN
\\ \hline
MMDL &0.24452& 7 &Audio,Skeleton &Sliding windows &Late &   DGM+LR
\\ \hline
TELEPOINTS& 0.25841 &8& Audio,Skeleton,RGB & Audio,Skeleton &
Late& HMM,SVM
\\ \hline
CSI MM &0.28911 &9&   Audio,Skeleton & Audio  & Early &  HMM
\\ \hline
SUMO &0.31652& 10 &Skeleton  &  Sliding windows &None &   RF
\\ \hline
GURU &0.37281 &11& Audio,Skeleton,Depth &   DP & Late & DP,RF,HMM
\\ \hline
AURINKO &0.63304 &12& Skeleton,RGB &   Skeleton &   Late&    ELM
\\ \hline
STEVENWUDI &0.74415& 13 &Audio,Skeleton & Sliding windows &Early&
DNN,HMM
\\ \hline
JACKSPARROW &0.79313& 14 & Skeleton  &  Sliding windows &None& NN
\\ \hline
JOEWAN &0.83772& 15 & Skeleton&    Sliding windows &None &   KNN
\\ \hline
MILAN KOVAC& 0.87463 &16 &Skeleton   & Sliding windows& None & NN
\\ \hline
IAMKHADER& 0.92069& 17 &Depth &Sliding windows &None &   RF
\\ \hline
\end{tabular}\label{resumenTabla}
\end{table*}


\textbf{The first ranked team $IVA MM$} on the test set used a
feature vector based on audio and skeletal information, and
applied late fusion to obtain final recognition results. A simple
time-domain end-point detection algorithm based on joint
coordinates is applied to segment continuous data sequences into
candidate gesture intervals. A Gaussian Hidden Markov Model is
trained with $39$-dimension MFCC features and generates confidence
scores for each gesture category. A Dynamic Time Warping based
skeletal feature classifier is applied to provide complementary
information. The confidence scores generated by the two
classifiers are firstly normalized and then combined to produce a
weighted sum. A single threshold approach is employed to classify
meaningful gesture intervals from meaningless intervals caused by
false detection of speech intervals.

\textbf{The second ranked team $WWEIGHT$} combined audio and
skeletal information, using both joint spatial distribution and
joint orientation. The method first searches for regions of time
with high audio-energy to define 1.8-second-long windows of time
that potentially contained a gesture. This had the effect that the
development, validation, and test data were treated uniformly.
Feature vectors are then defined using a log-spaced audio
spectrogram and the joint positions and orientations above the
hips. At each time sample the method subtracts the average 3D
position of the left and right shoulders from each 3D joint
position. Data is down-sampled onto a 5 Hz grid considering 1.8
seconds. There were 1593 features total (9 time samples $\times$
177 features per time sample). Since some of the detected windows
can contain distractor gestures, an extra 21st label is
introduced, defining the `not in the dictionary' gesture category.
Python's scikit-learn was used to train two models: an ensemble of
randomized decision trees (ExtraTreesClassifier, 100 trees, 40\%
of features) and a K-Nearest Neighbor model (7 neighbors, L1
distance). The posteriors from these models are averaged with
equal weight. Finally, a heuristic is used (12 gestures maximum,
no repeats) to convert posteriors to a prediction for the sequence
of gestures.


\begin{figure}[!t]\centering
\begin{tabular}{c}
   \includegraphics[width=8.3cm,height=1.8cm]{weight2.png}
\end{tabular}
   \caption{\scriptsize ExtraTreesClassifier Feature Importance.} \label{fig:weight1}
\end{figure}


Figure~\ref{fig:weight1} shows the mean feature importance for the
windows size of 1.8 seconds for the three sets of features: joint
coordinates, joint orientations, and audio spectogram. One can
note that features from the three sets are selected as
discriminative by the classifier, although skeletal features
becomes more useful for the ExtraTreesClassifier. Additionally,
the most discriminative features are those in the middle of the
windows size, since begin-end features are shared among different
gestures (transitions) and thus are less discriminative for the
classifier.

\begin{figure}[!t]\centering
\begin{tabular}{c}
   \includegraphics[width=8.3cm]{winhist.png}
\end{tabular}
   \caption{\scriptsize Recognition of test sequence by the three challenge winners. Black bin means that the complete list of ordered gestures has been successfully recognized.} \label{fig:winhist}
\end{figure}

\begin{figure*}[ht!]\centering
\begin{tabular}{c}
   \includegraphics[width=15cm]{gestures.png}
\end{tabular}
   \caption{\scriptsize Deviation of the number of gesture samples for each category by the three winners in relation to the GT data.} \label{fig:gestures}
\end{figure*}


\textbf{The third ranked team $ET$} combined the output decisions
of two designed approaches. In the first approach, they look for
gesture intervals (unsupervised) using the audio files and extract
these features from intervals (MFCC). Using these features,
authors train a random forest and gradient boosting classifier.
The second approch uses simple statistics (median, var, min, max)
on the first $40$ frames for each gesture to build the training
samples. The prediction phase uses a sliding window. The authors
create a weighted average of the output of these two models. The
features considered were skeleton information and audio signal.

Finally, we extracted some statistics from the results of the
three challenge winners in order to analyze common points and
difficult aspects of the challenge. Figure~\ref{fig:winhist} shows
the recognition of the 276 test sequences by the winners. Black
bin means that the complete list of ordered gestures was
successfully recognized for those sequences. Once can see that
there exists some kind of correlation among methods. Taking into
account that consecutive sequences belong to the same user
performing gestures, it means that some some gestures are easier
to recognize than others. Since different users appear in the
training and test sequences, it is sometimes difficult for the
models to generalize to the style of new users, based on the
gesture instances used for training.


We also investigated the difficulty of the problem by gesture
category, within each of the 20 Italian gesture categories.
Figure~\ref{fig:gestures} shows for each winner method the
deviation between the number of gesture instances recognized and
the total number of gestures, for each category. This was computed
for each sequence independently, and adding the deviation for all
the sequences. In that case, a zero value means that the
participant method recognized the same number of gesture instances
for a category that was recorded in the ground truth data.
Although we cannot guarantee with this measure that the order of
recognized gesture matches with the ground truth, it gives us an
idea of how difficult the gesture sequences were to segment into
individual gestures. Additionally, the sum of total deviation for
all the gestures for all the teams was 378, 469, and 504, which
correlates with the final rank of the winners. The figure suggests
a correlation between the performance of the three winners. In
particular, categories 6, 7, 8, 9, 16, 17, 18, and 19 were the
ones that achieved most accuracy for all the participants,
meanwhile 1, 2, 3, 5, and 12 were the ones that introduced the
highest recognition error. Note that the public data set provides
accurate label annotations of end-begin of gestures, and thus, a
more detailed recognition analysis could be performed applying a
different recognition measurement to Leveinstein, such as Jaccard
overlapping or sensitivity score estimation, which will also allow
for confusion matrix estimation based on both inter and intra user
and gesture category variability. This is left to future work.




\subsection{ChaLearn Multimodal Gesture Spotting Challenge
2014}

In ChaLearn LAP 2014~\citep{ChaLearnLAP2014} we focused on the
user-independent automatic spotting of a vocabulary of 20 Italian
cultural/anthropological signs in image sequences, see
Figure~\ref{fig:categories}.

\subsubsection{2014 Gesture Challenge Data}

This challenge was based on an Italian gesture data set, called
{\em Montalbano gesture dataset}, an enhanced version of the
ChaLearn 2013 multimodal gesture recognition
challenge~\citep{escalera2013multi,escalera2013multi2} with more
ground-truth annotations. In all the sequences, a single user is
recorded in front of a \kinect, performing natural communicative
gestures and speaking in fluent Italian. Examples of the different
visual modalities are shown in Figure~\ref{fig:modalities}.

The main characteristics of the data set are:

$\bullet$ Largest data set in the literature, with a large
duration of each individual performance showing no resting poses
and self-occlusions.

$\bullet$ There is no information about the number of gestures to
spot within each sequence, and several distracter gestures (out of
the vocabulary) are present.

$\bullet$ High intra-class variability of gesture samples and low
inter-class variability for some gesture categories.

A list of data attributes for data set used in track 3 is
described in Table~\ref{tbl::set3}.

\begin{table}[t!]\centering
\scriptsize
\begin{tabular}{|l|l|l|l|l|}
\hline Training seq. & Validation seq. & Test seq. &
Sequence duration & FPS  \\
\hline 393 (7,754 gestures) & 287 (3,362 gestures) & 276 (2,742
gestures) & 1-2 min & 20 \\\hline\hline Modalities & Num. of users
& Gesture categories & Labeled sequences & Labeled frames\\\hline
RGB, Depth, User mask, Skeleton & 27 & 20 & 13,858 &
1,720,800\\\hline
\end{tabular}
\caption{Main characteristics of the $Montalbano$ gesture
dataset.} \label{tbl::set3}
\end{table}




\subsubsection{2014 Gesture Challenge Protocol and Evaluation}
\label{jaccard}

The challenge was managed using the Microsoft Codalab
platform\footnote{https://www.codalab.org/competitions/}. We
followed a development (February 9 to May 20 2014) and tests
phases (May 20th - June 1st 2014)) as in our previous challenges.

To evaluate the accuracy of action/interaction recognition, we use
the Jaccard Index, For the $n$ action, interaction, and gesture
categories labelled for a RGB/RGBD sequence $s$, the Jaccard Index
is defined as:
\begin{equation}
J_{s,n}=\frac{A_{s,n} \bigcap B_{s,n}}{A_{s,n} \bigcup B_{s,n} },
\end{equation}

\noindent where $A_{s,n}$ is the ground truth of
action/interaction/gesture $n$ at sequence $s$, and $B_{s,n}$ is
the prediction for such an action at sequence $s$. $A_{s,n}$ and
$B_{s,n}$ are binary vectors where $1$-values correspond to frames
in which the $n-$th action is being performed. The participants
were evaluated based on the mean Jaccard Index among all
categories for all sequences, where motion categories are
independent but not mutually exclusive (in a certain frame more
than one action, interaction, gesture class can be active).

In the case of false positives (e.g. inferring an action,
interaction or gesture not labelled in the ground truth), the
Jaccard Index is $0$ for that particular prediction, and it will
not count in the mean Jaccard Index computation. In other words
$n$ is equal to the intersection of action/interaction/gesture
categories appearing in the ground truth and in the predictions.

An example of the calculation for two actions is shown in Figure
\ref{fig:1}. Note that in the case of recognition, the ground
truth annotations of different categories can overlap (appear at
the same time within the sequence). Also, although different
actors appear within the sequence at the same time,
actions/interactions/gestures are labelled in the corresponding
periods of time (that may overlap), there is no need to identify
the actors in the scene. The example in Figure \ref{fig:1} shows
the mean Jaccard Index calculation for different instances of
actions categories in a sequence (single red lines denote ground
truth annotations and double red lines denote predictions). In the
top part of the image one can see the ground truth annotations for
actions walk and fight at sequence $s$. In the center part of the
image a prediction is evaluated obtaining a Jaccard Index of
$0.72$.  In the bottom part of the image the same procedure is
performed with the action fight and the obtained Jaccard Index is
$0.46$. Finally, the mean Jaccard Index is computed obtaining a
value of $0.59$.

\begin{figure}[t!]
\centering \begin{tabular}{cc}
\includegraphics[width=7cm]{caso1.png}&\includegraphics[width=7cm]{caso2.png}\\
\end{tabular}\caption{Example of mean
Jacquard Index calculation for gesture and action/interaction
spotting.} \label{fig:1}
\end{figure}


\subsubsection{2014 Gesture Challenge Results}

Table~\ref{tbl::results3} summarizes the methods of the 17
participants that contributed to the test set of track 3. Although
DTW and HMM (and variants) were in the last edition of the
ChaLearn Multimodal Gesture
competition~\citep{escalera2013multi,escalera2013multi2}, random
forest has been widely applied in this 2014 edition. Also, three
participants used deep learning architectures.

\begin{landscape}

\begin{table}[h!]\centering
\tiny
\begin{tabular}{|l|l|l|l|l|l|l|l|l|}
\hline Team name & Accuracy & Rank &  Features  & Dimension
reduction & Clustering & Classifier & Temporal coherence & Gesture
representation\\\hline



CUHK-SWJTU  & \textbf{0.507173} & 1  & Improved
trajectories~\citep{Wang2013} & PCA &- & SVM &Sliding windows
&Fisher Vector\\\hline


ADSC    & \textbf{0.501164} & 2 &  Improved
trajectories~\citep{Wang2013} & - & -  & SVM &Sliding windows
&-\\\hline


SBUVIS  & \textbf{0.441405} & 3 &  Improved
trajectories~\citep{Wang2013} & - & - & SVM &Sliding windows
&-\\\hline


DonkeyBurger  & 0.342192 & 4 &  MHI, STIP &  - &  Kmeans  &Sparse
code &Sliding windows &-\\\hline


UC-T2  & 0.121565 & 5  & Improved trajectories~\citep{Wang2013} &
PCA& - & Kmeans & Sliding windows &Fisher Vector\\\hline


MindLAB & 0.008383 & 6   &MBF &   -  & Kmeans&  RF & Sliding
windows& BoW\\\hline\hline\hline


MMLAB & \textbf{0.5385} & 1 & IDT & PCA & - & SVM & - & Fisher
Vector\\\hline

FIKIE & \textbf{0.5239} & 2 & IDT & PCA & - & HMM &
Appearance+Kalman filter & -\\\hline

\end{tabular}
\caption{Top rows: Action/interaction 2014 recognition results.
MHI: Motion History Image; STIP: Spatio-Temporal interest points;
MBF: Multiscale Blob Features; BoW: Bag of Visual Words; RF:
Random Forest. Bottom two rows: Action/interaction 2015
recognition results. IDT: Improved Dense
Trajectories~\citep{Wang2013}.} \label{tbl::results2}
\end{table}


\begin{table}[h!]\centering
\tiny
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|}
\hline Team &  Accuracy &   Rank &   Modalities & Features  &
Fusion& Temp. segmentation &Dimension reduction &Gesture
representation& Classifier\\\hline


LIRIS  & \textbf{0.849987} & 1 &  SK, Depth, RGB  &  RAW, SK
joints& Early& Joints motion &  - &  - &  DNN\\\hline


CraSPN  & \textbf{0.833904} & 2  & SK, Depth, RGB &   HOG, SK&
Early& Sliding windows &-  & BoW &Adaboost\\\hline

JY & \textbf{0.826799}& 3 &  SK, RGB  & SK, HOG &  Late  &  MRF
&PCA &- &MRF, KNN\\\hline


CUHK-SWJTU  & 0.791933 & 4 &  RGB &Improved
trajectories~\citep{Wang2013} & - &Joints motion& PCA& Fisher
Vector, VLAD  & SVM\\\hline


Lpigou & 0.788804& 5 &  Depth, RGB  &RAW, SK joints  &  Early
&Sliding windows& Max-pooling CNN& -&   CNN\\\hline


stevenwudi  & 0.787310&6 &  SK, depth &RAW& Late &   Sliding
windows& -& - &  HMM, DNN \\\hline


Ismar  & 0.746632& 7  & SK &   SK &   -   &Sliding windows& - &-&
RF\\\hline


Quads  & 0.745449& 8 &  SK &   SK quads  &-&   Sliding windows& -&
Fisher Vector  & SVM\\\hline


Telepoints & 0.688778& 9 &  SK, Depth, RGB  &  STIPS, SK& Late&
Joints motion&   - &  - &  SVM\\\hline


TUM-fortiss &0.648979& 10 & SK, Depth, RGB  &  STIPS &  Late&
Joints motion &  - &  - &  RF, SVM\\\hline


CSU-SCM & 0.597177&11 & Skeleton, Depth, mask &  HOG, Skeleton &
Late &Sliding windows &-&   2DMTM  & SVM, HMM\\\hline


iva.mm  & 0.556251& 12 & Skeleton, RGB, depth  &  Skeleton, HOG  &
Late& Sliding windows& -  & BoW &SVM, HMM\\\hline


Terrier &0.539025& 13 & Skeleton   & Skeleton &   -  & Sliding
windows &-& -& RF\\\hline


Team Netherlands  &0.430709&  14 & Skeleton, Depth, RGB &   MHI&
Early& DTW& Preserving projections &-&   SVM, RT\\\hline


VecsRel &0.408012& 15 & Skeleton, Depth, RGB &   RAW, skeleton
joints& Late& DTW& - &  - &  DNN\\\hline


Samgest & 0.391613&16 & Skeleton, Depth, RGB, mask  &Skeleton,
blobs, moments &Late &   Sliding windows& - &  - &  HMM\\\hline


YNL& 0.270600& 17 & Skeleton &   Skeleton   & - &  Sliding windows
&-& Fisher Vector  & HMM, SVM \\\hline


\end{tabular}
\caption{Multimodal gesture recognition results. SK: Skeleton;
DNN: Deep Neural Network; RF: Ranfom Forest; 2DMTM: 2D motion
trail model; RT: Regression Tree. } \label{tbl::results3}
\end{table}
\end{landscape}


\subsubsection{2014 Gesture Challenge Summary of the Winner Methods}


Next, we describe the main characteristics of the three winning
methods.

\textbf{First place:} The proposed method was based on a deep
learning architecture that iteratively learned and integrated
discriminative data representations from individual channels,
modelling cross-modality correlations and short- and long-term
temporal dependencies. This framework combined three data
modalities: depth information, grayscale video and skeleton stream
("articulated pose"). Articulated pose served as an efficient
representation of large-scale body motion of the upper body and
arms, while depth and video streams contained complementary
information about more subtle hand articulation. The articulated
pose was formulated as a set of joint angles and normalized
distances between upper-body joints, augmented with additional
information reflecting speed and acceleration of each joint. For
the depth and video streams, the authors did not rely on
hand-crafted descriptors, but on discriminatively learning joint
depth-intensity data representations with a set of convolutional
neural layers. Iterative fusion of data channels was performed at
output layers of the neural architecture. The idea of learning at
multiple scales was also applied to the temporal dimension, such
that a gesture was considered as an ordered set of characteristic
motion impulses, or dynamic poses. Additional skeleton-based
binary classifier was applied for accurate gesture localization.
Fusing multiple modalities at several spatial and temporal scales
led to a significant increase in recognition rates, allowing the
model to compensate for errors of the individual classifiers as
well as noise in the separate channels.

\textbf{Second place:} The approach combined a sliding-window
gesture detector with multimodal features drawn from skeleton
data, color imagery, and depth data produced by a first-generation
\kinect sensor. The gesture detector consisted of a set of boosted
classifiers, each tuned to a specific gesture or gesture mode.
Each classifier was trained independently on labeled training
data, employing bootstrapping to collect hard examples. At
run-time, the gesture classifiers were evaluated in a one-vs-all
manner across a sliding window. Features were extracted at
multiple temporal scales to enable recognition of variable-length
gestures. Extracted features included descriptive statistics of
normalized skeleton joint positions, rotations, and velocities, as
well as HOG descriptors of the hands. The full set of gesture
detectors was trained in under two hours on a single machine, and
was extremely efficient at runtime, operating at 1700 fps using
skeletal data.

\textbf{Third place:} The proposed method was based on four
features: skeletal joint position feature, skeletal joint distance
feature, and histogram of oriented gradients (HOG) features
corresponding to left and right hands. Under the na\"ive Bayes
assumption, likelihood functions were independently defined for
every feature. Such likelihood functions were non-parametrically
constructed from the training data by using kernel density
estimation (KDE). For computational efficiency, k-nearest neighbor
(kNN) approximation to the exact density estimator was proposed.
Constructed likelihood functions were combined to the multimodal
likelihood and this serves as a unary term for our pairwise Markov
random field (MRF) model. For enhancing temporal coherence, a
pairwise term was additionally incorporated to the MRF model.
Final gesture labels were obtained via 1D MRF inference
efficiently achieved by dynamic programming.


\subsection{ChaLearn Action and Interaction Spotting Challenge
2014}\label{action}

The goal of this challenge was to perform automatic action and
interaction spotting of people appearing in RGB data sequences.

\subsubsection{2014 Action Challenge Data}

We presented a novel fully limb labelled dataset, the Human Pose
Recovery and Behavior Analysis \emph{HuPBA} $8k+$
dataset~\citep{neuro}. This dataset is formed by more than $8000$
frames where $14$ limbs are labelled at pixel precision, thus
providing $124,761$ annotated human limbs. The characteristics of
the data set are:

$\bullet$ The images are obtained from $9$ videos (RGB sequences)
and a total of
    $14$ different actors appear in the sequences. The image sequences have been recorded using a stationary camera with the same static background.

$\bullet$ Each video (RGB sequence) was recorded at $15$ fps rate,
and each RGB image was stored with resolution $480\times 360$ in
BMP file format.

$\bullet$ For each actor present in an image $14$ limbs (if not
occluded) were manually tagged: Head, Torso, R-L Upper-arm, R-L
Lower-arm, R-L Hand, R-L Upper-leg, R-L Lower-leg, and R-L Foot.

$\bullet$ Limbs are manually labelled using binary masks and the
minimum bounding box containing each subject is defined.

$\bullet$ The actors appear in a wide range of different poses and
performing
    different actions/gestures which vary the visual
appearance of human limbs. So there is a large variability of
human poses, self-occlusions and many variations in clothing and
skin color.

$\bullet$ Several actions and interactions categories are labelled
at frame level.

A key frame example for each gesture/action category is shown in
Figure~\ref{fig::gestures}. The challenges the participants had to
deal with for this new competition are:

$\bullet$ 235 action/interaction samples performed by 14 actors.

$\bullet$ Large difference in length about the performed actions
and interactions. Several distracter actions out of the 11
categories are also present.

$\bullet$ 11 action categories, containing isolated and
collaborative actions: Wave, Point, Clap, Crouch, Jump, Walk, Run,
Shake Hands, Hug, Kiss, Fight. There is a high intra-class
variability among action samples.

Table~\ref{tbl::set2} summarizes the data set attributes for the
case of action/interaction spotting.

\begin{figure}[t!]\centering
\subfigure[Wave]{\includegraphics[width=.3\textwidth]{wave.png}}
\subfigure[Point]{\includegraphics[width=.3\textwidth]{point_at.png}}
\subfigure[Clap]{\includegraphics[width=.3\textwidth]{clap.png}}\\
\subfigure[Crouch]{\includegraphics[width=.3\textwidth]{crouch.png}}
\subfigure[Jump]{\includegraphics[width=.3\textwidth]{jump.png}}
\subfigure[Walk]{\includegraphics[width=.3\textwidth]{walk.png}}\\
\subfigure[Run]{\includegraphics[width=.3\textwidth]{run.png}}
\subfigure[Shake
hands]{\includegraphics[width=.3\textwidth]{shake_hands.png}}
\subfigure[Hug]{\includegraphics[width=.3\textwidth]{hug.png}}\\
\subfigure[Kiss]{\includegraphics[width=.3\textwidth]{kiss.png}}
\subfigure[Fight]{\includegraphics[width=.3\textwidth]{fight.png}}
\subfigure[Idle]{\includegraphics[width=.3\textwidth]{idle.png}}
\caption{\label{fig::gestures}Key frames of the \emph{HuPBA} $8K+$
dataset used in the tracks 1 and 2, showing actions ((a) to (g)),
interactions ((h) to (k)) and the idle pose (l).}
\end{figure}





\begin{table}[t!]\centering
\scriptsize
\begin{tabular}{|l|l|l|l|l|}
\hline Training actions & Validation actions & Test actions &
Sequence duration & FPS  \\
\hline 150 & 90 & 95 & 9$\times$ 1-2 min & 15 \\\hline\hline
Modalities & Num. of users & Action categories & interaction
categories & Labelled sequences\\\hline RGB & 14 & 7 & 4 &
235\\\hline
\end{tabular}
\caption{Action and interaction data characteristics.}
\label{tbl::set2}
\end{table}

\subsubsection{2014 Action Challenge Protocol and Evaluation}

To evaluate the accuracy of action/interaction recognition, we use
the Jaccard Index as defined in Section~\ref{jaccard}.

\subsubsection{2014 Action Challenge Results}

In this section we summarize the methods proposed by the
participants and the winning methods. Six teams submitted their
code and predictions for the test sets. Top rows of
Table~\ref{tbl::results2} summarizes the approaches of the
participants who uploaded their models. One can see that most
methods are based on similar approaches. In particular,
alternative representations to classical BoW were considered, as
Fisher Vector and VLAD~\citep{agregating}. Most methods perform
sliding windows and SVM classification. In addition, to refine the
tracking of interest points, 4 participants used improved
trajectories~\citep{Wang2013}.



\subsubsection{2014 Action Challenge Summary of the Winner Methods}


Next, we describe the main characteristics of the three winning
methods.

\textbf{First place:} The method was composed of two parts: video
representation and temporal segmentation. For the representation
of video clip, the authors first extracted improved dense
trajectories with HOG, HOF, MBHx, and MBHy descriptors. Then, for
each kind of descriptor, the participants trained a GMM and used
Fisher vector to transform these descriptors into a high
dimensional super vector space. Finally, sum pooling was used to
aggregate these codes in the whole video clip and normalize them
with power L2 norm. For the temporal recognition, the authors
resorted to a temporal sliding method along the time dimension. To
speed up the processing of detection, the authors designed a
temporal integration histogram of Fisher Vector, with which the
pooled Fisher Vector was efficiently evaluated at any temporal
window. For each sliding window, the authors used the pooled
Fisher Vector as representation and fed it into the SVM classifier
for action recognition.

\textbf{Second place:} a human action detection framework called
"mixture of heterogeneous attribute analyzer" was proposed. This
framework integrated heterogeneous attributes learned from various
types of video features including static and dynamic, local and
global features, to boost the action detection accuracy. The
authors first detected a human from the input video by SVM-HOG
detector and performed forward-backward tracking. Multiple local
human tracks are linked into long trajectories by spatial-temporal
graph based matching. Human key poses and local dense motion
trajectories were then extracted within the tracked human bounding
box sequences. Second, the authors proposed a mining method that
learned discriminative attributes from three feature modalities:
human trajectory, key pose and local motion trajectory features.
The mining framework was based on the exemplar-SVM discriminative
middle level feature detection approach. The learned
discriminative attributes from the three types of visual features
were then mixed in a max-margin learning algorithm which also
explores the combined discriminative capability of heterogeneous
feature modalities. The learned mixed analyzer was then applied to
the input video sequence for action detection.

\textbf{Third place:} The framework for detecting actions in video
is based on improved dense trajectories applied on a sliding
windows fashion. Authors independently trained 11 one-versus-all
kernel SVMs  on the labelled training set for 11 different
actions. The feature and feature descriptions used are improved
dense trajectories, HOG, HOF, MBHx and MBHy. During training, for
each action, a temporal sliding window is applied without
overlapping. For every action, a segment was labelled 0 (negative)
for a certain action only if there is no frame in this segment
labelled 1. The feature coding method was bag-of-features. For a
certain action, the features associated with those frames which
are labelled 0 (negative) are not counted when we code the
features of the action for the positive segments with
bag-of-features. On the basis of the labelled segments and their
features, a kernel SVM was trained for each action. During
testing, non-overlap sliding window was applied for feature coding
of the video. Every frame in a segment was consistently labelled
as the output of SVM for each action. The kernel type, sliding
window size and penalty of SVMs were selected during validation.
When building the bag-of-features, the clustering method was
$K$-means and the vocabulary size is 4000. For one trajectory
feature in one frame, all the descriptors were connected to form
one description vector. The bag-of-features were built upon this
vector.




\subsection{ChaLearn Action and Interaction Spotting Challenge
2015}

The goal of this challenge was to perform automatic action and
interaction spotting of people appearing in RGB data sequences.
This corresponds to the second round of 2014 Action/Interaction
challenge~\citep{chalearnLAP2015}. Data, protocol, and evaluation
were defined as explained in Section~\ref{action}.


\subsubsection{2015 Action Challenge Results}

Results of the two top ranked participants are shown in bottom
rows of Table~\ref{tbl::results2}. One can see that the methods of
the participants are similar to the ones applied in the 2014
challenge for the same dataset (Top rows of
Table~\ref{tbl::results2}). Results of this second competition
round improved by 2\% the results obtained in the first round of
the challenge.



\subsubsection{2015 Action Challenge Summary of the Winner Methods}


\textbf{First winner:} This method is an improvement of the system
proposed in~\citep{PengWang2015}, which is composed of two parts:
video representation and temporal segmentation. For the
representation of video clip, the authors first extracted improved
dense trajectories with HOG, HOF, MBHx, and MBHy descriptors.
Then, for each kind of descriptor, the participants trained a GMM
and used Fisher vector to transform these descriptors into a high
dimensional super vector space. Finally, sum pooling was used to
aggregate these codes in the whole video clip and normalize them
with power L2 norm. For the temporal recognition, the authors
resorted to a temporal sliding method along the time dimension. To
speed up the processing of detection, the authors designed a
temporal integration histogram of Fisher Vector, with which the
pooled Fisher Vector was efficiently evaluated at any temporal
window. For each sliding window, the authors used the pooled
Fisher Vector as representation and fed it into the SVM classifier
for action recognition. A summary of this method is shown in
Figure~\ref{fig:MMLAB_t1}.

\begin{figure}[t!]
\centering
\includegraphics[width=13cm]{MMLAB1.png}\\
\caption{Method summary for MMLAB team~\citep{CVPR_paper5}.}
\label{fig:MMLAB_t1}
\end{figure}

\textbf{Second winner:} The method implements an end-to-end
generative approach from feature modelling to activity
recognition. The system combines dense trajectories and Fisher
Vectors with a temporally structured model for action recognition
based on a simple grammar over action units. The authors modify
the original dense trajectory implementation of~\citep{Wang2013}
to avoid the omission of neighborhood interest points once a
trajectory is used (the improvement is shown in
Figure~\ref{fig:FKIE}).  They use an open source speech
recognition engine for the parsing and segmentation of video
sequences. Because a large data corpus is typically needed for
training such systems, images were mirrored to artificially
generate more training data. The final result is achieved by
voting over the output of various parameter and grammar
configurations.

\begin{figure}[t!]
\centering \subfigure[]
{\includegraphics[width=6cm]{DTSampling_old.png}} \subfigure[]
{\includegraphics[width=6cm]{DTSampling_new.png}} \caption{Example
of DT feature distribution for the first 200 frames of Seq01 for
FKIE team, (a) shows the distribution of the original
implementation, (b) shows the distribution of the modified
version.} \label{fig:FKIE}
\end{figure}


\subsection{Other international competitions for gesture and action recognition}

In addition to the series of ChaLearn Looking at People
challenges, different international challenges have also been
performed in the field of action/gesture recognition. Some of them
are reviewed below.

The ChAirGest challenge~\citep{chairgest} is a research oriented
competition designed to compare multimodal gesture recognizers.
The provided data came from one Kinect camera and 4 Inertial
Motion Units (IMU) attached to the right arm and neck of the
subject. The dataset contains 10 different gestures, started from
3 different resting postures and recorded in two different
lighting conditions by 10 different subjects. Thus, the total
dataset contains 1200 annotated gestures split in continuous video
sequences containing a variable number of gestures. The goal of
the challenge was to promote research on methods using multimodal
data to spot and recognize gestures in the context of close
human-computer interaction.

In 2015, OpenCV ran at CVPR different
challenges\footnote{http://code.opencv.org/projects/opencv/wiki/VisionChallenge},
one of them focusing on gesture recognition, using the ChaLearn
gesture recognition data. The winner of the competition was also
the work that won the ChaLearn challenge at ECCV
2014~\citep{natalia2014}.

Another recent action recognition competition is the THUMOS
challenge~\citep{THUMOS15}. Last round of the competition was ran
at CVPR 2015. The last round of the challenge contained a
forward-looking data set with over 430 hours of video data and 45
million frames (70\% larger than THUMOS'14). All videos were
collected from YouTube. Two tracks were performed: 1) Action
Classification, for whole-clip action classification on 101 action
classes, and 2) Temporal Action Localization, for action
recognition and temporal localization on a subset of 20 action
classes.



\section{Summary of Special Topic Papers Not Related to the Challenges}

In this special topic, in addition to the papers that described
systems that participated in the ChaLearn gesture challenges,
there are also several papers relating to broader aspects of
gesture recognition, including topics such as sign language recognition, facial expression analysis, and facilitating development of real-world systems.

Three of these papers propose new methods for gesture and action recognition \citep{malgireddy-2013,fanello_jmlr2013,lui_jmlr2012}, that were also evaluated on parts of the CDG2011 dataset \citep{guyon-mva-2014}, used in the ChaLearn challenges held in 2011 and 2012. More specifically, \citep{malgireddy-2013} present an approach for detecting and
recognizing activities and gestures. Hierarchical models are built
to describe each activity as a combination of other, more simple
activities. Each video is recursively divided into segments
consisting of a fixed number of frames. The relationship between
observed features and latent variables is modelled using a
generative model that combines aspects of dynamic Bayesian
networks and hierarchical Bayesian models.
\citep{fanello_jmlr2013} describe a system for real-time action recognition using depth video. The paper proposes specific
features that are well adapted to real-time constraints, based on
histograms of oriented gradients and histograms of optical flow.
Support vector machines trained on top of such features perform
action segmentation and recognition.
\citep{lui_jmlr2012} propose a method for representing gestures as
tensors. These tensors are treated as points on a product
manifold. In particular, the product manifold is factorized into
three manifolds, one capturing horizontal motion, one capturing
vertical motion, and one capturing 2D appearance. The difference
between gestures is measured using a geodesic distance on the
product manifold.

One paper \citep{wang_jmlr2012} studied an action recognition problem outside the scope of the ChaLearn contests, namely recognizing poses and actions from a single image. Hierarchical models are
used for modelling body pose. Each model in this hierarchy covers
a part of the human body that can range from the entire body to a
specific rigid part. Different levels in this hierarchy correspond
to different degrees of coarseness vs. detail in the models at
each level.

Three papers proposed novel methods within the area of sign language recognition \citep{cooper_jmlr2012,nayak_jmlr2012,roussos_jmlr2013}.
\citep{cooper_jmlr2012} describe a method for sign language
recognition using linguistic subunits that are learned
automatically by the system. Different types of such subunits are
considered, including subunits based on appearance and local
motion of the hand, subunits based on combining tracked 2D hand
trajectories and hand shape, and subunits based on tracked 3D hand
trajectories.
\citep{nayak_jmlr2012} address the problem of learning a model for
a sign that occurs multiple times in a set of sentences. One
benefit from such an approach is that it does not require the
start and end frame of each sign as training data. Another benefit
is that the method identifies the aspects of a sign that are least
affected by movement epenthesis, i.e., by signs immediately
preceding or following the sign in question.
\citep{roussos_jmlr2013} present a method for classifying
handshapes for the purpose of sign language recognition. Cropped
hand images are converted to a normalized representation called
``shape-appearance images'', based on a PCA decomposition of skin
pixel colors. Then, active appearance models are used to model the
variation in shape and appearance of the hand.

One paper focused on the topic of facial expression analysis \citep{martinez_jmlr2012}. The paper proposes a model for describing how humans
perceive facial expressions of emotion. The proposed model
consists of multiple distinct continuous spaces. Emotions can be
represented using linear combinations of these separate spaces.
The paper also discusses how the proposed model can be used to
design algorithms for facial expression recognition.


Another set of papers contributed methods that address different practical problems, that are important for building
real-world gesture interfaces \citep{Van_jmlr2014,Gillian_jmlr2014,kohlsdorf_jmlr2013}.
One such problem is obtaining manual annotations and ground truth for large amounts of training data. Obtaining such manual annotations can be time consuming, and can be an important bottleneck in building a real system. Crowdsourcing is a potential solution, but crowdsourced annotations often suffer from noise, in the form of  discrepancies in how different humans annotate the same data. In \citep{Van_jmlr2014}, two template-matching methods are proposed, called SegmentedLCSS and WarpingLCSS, that explicitly deal with the noise present in crowdsourced annotations of gestures. These methods are designed for spotting gestures using wearable motion sensors. The methods quantize signals into strings of characters and then apply variations of the longest common subsequence algorithm (LCSS) to spot gestures.

In designing a real-world system, another important problem is rapid development. \citep{Gillian_jmlr2014} present a gesture recognition toolkit, a
cross-platform open-source C++ library. The toolkit features a
broad range of classification and regression algorithms and has
extensive support for building real-time systems. This includes
algorithms for signal processing, feature extraction and automatic
gesture spotting.

Finally, choosing the gesture vocabulary can be an important implementation parameter. \citep{kohlsdorf_jmlr2013} propose a method for choosing a
vocabulary of gestures for a human-computer interface, so that
gestures in that vocabulary have a low probability of being
confused with each other. Candidate gestures for the interface can
be suggested both by humans and by the system itself. The system
compares examples of each gesture with a large repository of
unlabelled sensor/motion data, to check how often such examples
resemble typical session/motion patterns encountered in that
specific application domain.


\section{Summary of Special Topic Papers Related to 2011-2012 Challenges}

Next we briefly review the contributions of the accepted papers
for the special topic whose methods are applied on the data
provided by 2011-2012 ChaLearn gesture recognition challenges. Interestingly, none of the methods proposed in these papers uses skeletal tracking. Instead, these methods use different features based on appearance and/or motion. Where the papers differ is in their choice of specific features, and also in the choice of gesture models that are built on top of the selected features.

\citep{konecny-2014} combine appearance (Histograms of Oriented
Gradients) and motion descriptors (Histogram of Optical Flow) from
RGB and depth images for parallel temporal segmentation and
recognition. The Quadratic-Chi distance family is used to measure
differences between histograms to capture cross-bin relationships.
Authors also propose trimming videos by removing unimportant
frames based on motion information. Finally, proposed descriptors
with different Dynamic Time Warping variants are applied for final
recognition.


In contrast to \citep{konecny-2014}, which employes commonly used features, \citep{wan-2013} proposes a new multimodal descriptor, as well as a new sparse coding method. The multimodal descriptor is called 3D EMoSIFT, is invariant to scale and rotation, and has more compact and richer visual representations than other state-of-the-art descriptors. The proposed sparse coding method is named simulation orthogonal matching pursuit (SOMP), and is a variant of BoW. Using SOMP, each feature can be represented by some linear combination of a small number of codewords.

A different approach is taken in \citep{Feng_jmlr2014}, which combines three different methods for classifying gestures. The first method uses an improved principal motion representation. In the second
method, a particle-based descriptor  and a weighted dynamic time
warping are proposed for the location component classification. In
the third method, the shape of the human subject is used, extracted from the frame in the gesture that exhibits the least motion. The explicit use of shape in this paper is in contrast to \citep{konecny-2014,wan-2013}, where shape information is implicitly coded in the extracted features.

In \citep{Norberto2014}, the focus is on transfer learning. The proposed method did not do as well as the previous methods \citep{konecny-2014,wan-2013,Feng_jmlr2014} on the CDG 2011 dataset, but nonetheless it contributes novel ideas for transfer learning, that can be useful when the number of training examples per class is limited. This is in contrast to the other papers related to the 2011-2012 challenges, that did not address transfer learning. The paper introduces two mechanisms into the
decision forest framework, in order to transfer knowledge from the
source tasks to a given target task. The first one is mixed
information gain, which is a data-based regularizer. The second
one is label propagation, which infers the manifold structure of
the feature space. The proposed approach show improvements over
traditional decision forests in the ChaLearn Gesture Challenge and on the
MNIST dataset.

\section{Summary of Special Issue Papers Related to 2013 Challenge}



Next we briefly review the contributions of the accepted papers
for the special issue whose methods are applied on the data
provided by 2013 ChaLearn multimodal gesture recognition
challenge. An important difference between this challenge and the previous ChaLearn challenges is the multimodal nature of the data. Thus, a key focus area for methods applied on this data is the problem of fusing information from multiple modalities.

\citep{Wu_jmlr2014} propose a Bayesian Co-Boosting framework for
multimodal gesture recognition. Inspired by boosting learning and
co-training method, the system combines multiple collaboratively
trained weak classifiers, Hidden Markov Models in this case, to
construct the final strong classifier. During each iteration
round, randomly a number of feature subsets are samples and weak
classifiers parameters for each subset are estimated. The optimal
weak classifier and its corresponding feature subset are retained
for strong classifier construction. Authors also define an upper
bound of training error and derive the update rule of instance's
weight, which guarantees the error upper bound to be minimized
through iterations. This methodology won the ChaLearn 2013 ICMI
competition.

\citep{Vassilis2014} present a framework for multimodal gesture
recognition that is based on a multiple hypotheses rescoring
fusion scheme. Authors employ multiple modalities, i.e., visual
cues, such as skeleton data, color and depth images, as well as
audio, and extract feature descriptors of the handsï¿½ movement,
handshape, and audio spectral properties. Using a common hidden
Markov model framework authors build single-stream gesture models
based on which they can generate multiple single stream-based
hypotheses for an unknown gesture sequence. By multimodally
rescoring these hypotheses via constrained decoding and a weighted
combination scheme, authors end up with a multimodally-selected
best hypothesis. This is further refined by means of parallel
fusion of the monomodal gesture models applied at a segmental
level. The proposed methodology is tested on the ChaLearn 2013
ICMI competition data.


\section{Discussion}

We reviewed the gesture recognition topic, defining a taxonomy to
characterize state of the art works on gesture recognition. We
also reviewed the gesture and action recognition challenges
organized by ChaLearn from 2011 to 2015, as well as other
international competitions related to gesture recognition.
Finally, we reviewed the papers submitted to the Special Topic on
Gesture Recognition 2011-2014 we organized at Journal of Machine
Learning Research.

Regarding the ChaLearn gesture recognition challenges, we began
right at the start of the \kinect revolution when inexpensive
infrared cameras providing image depth recordings became
available. We published papers using this technology and other
more conventional methods, including regular video cameras, to
record data, thus providing a good overview of uses of machine
learning and computer vision using multimodal data in this area of
application. Notably, we organized a series of challenges and made
available several datasets we recorded for that purpose, including
tens of thousands of videos, which are available to conduct
further research~\footnote{http://gesture.chalearn.org/}.

Regarding the papers published in the gesture recognition special
topic related to 2011-2012 challenges with the objective of
performing one-shot learning, most of the authors proposed new
multimodal descriptors taking benefit from both RGB and Depth cues
in order to describe human body features, both static and dynamic
ones. As the recognition strategies, common techniques used were
variants of classical well-known Dynamic Time Warping and Hidden
Markov Models. In particular, the most efficient techniques so far
have used sequences of features processed by graphical models of
the HMM/CRF family, similar to techniques used in speech
recognition. Authors also considered a gesture candidate sliding
window and motion-based video-cutting approaches. Last approach
was frequently used since sign language videos included a resting
pose. Also interesting novel classification strategies were
proposed, such as multilayered decomposition, where different
length gesture units are recognize at different levels
(\citep{Feng_jmlr2014}).

Regarding the papers published in the gesture recognition special
topic related to 2013 and 2014 challenges with the objective of
performing user independent multiple gesture recognition from
large volumes of multimodal data (RGB, Depth and audio), different
classifiers for gesture recognition were used by the participants.
In 2013, the preferred one was Hidden Markov Models (used by the
first ranked team of the challenge), followed by Random Forest
(used by the second and third winners). Although several state of
the art learning and testing gesture techniques were applied at
the last stage of the methods of the participants, still the
feature vector descriptions are mainly based on MFCC audio
features and skeleton joint information. The published paper of
the winner to the special topic presents a novel coboosting
strategy, where a set of HMM classifiers and collaboratively
included in a boosting strategy considering random sets of
features (\citep{Wu_jmlr2014}). In 2014, similar descriptors and
classifiers were used, and in particular, three deep learning
architectures were considered, including the method of the winner
team~\citep{natalia2014}.

In the case of the ChaLearn action/interaction challenges
organized in 2014 and 2015 most methods were based on similar
approaches. In particular, alternative representations to
classical BoW were considered, as Fisher Vector and
VLAD~\citep{agregating}. Most methods performed sliding windows and
SVM classification. In addition, to refine the tracking of
interest points, several participants used improved
trajectories~\citep{Wang2013}.

From the review of the gesture recognition topic, the achieved
results in the performed challenges and the rest of papers
published in the gesture recognition Special Topic, one can
observe that still it is possible that progress will also be made
in feature extraction by making better use of the multimodal
development data for better transfer learning. For instance, we
think that structural hand information around hand joint could be
useful to discriminate among gesture categories that may share
similar trajectories of hand/arms. Also recent approaches have
shown that Random Forest and Deep Learning, such as considering
Convolutional Neural Networks, are powerful alternatives to
classical gesture recognition approaches, which still open the
door for future the design of new gesture recognition classifiers.

In the case of action detection or spotting, most of the
methods are still based on sliding-windows approaches, which makes
recognition a time-consuming task. Thus, the research on methods
that can generate gesture/action candidates from data in a
different fashion are still an open issue.



\acks{ This work has been partially supported by ChaLearn
Challenges in Machine Learning \url{http://chalearn.org}, the
Human Pose Recovery and Behavior Analysis
Group\footnote{\scriptsize HuPBA research group:
http://www.maia.ub.es/$\sim$sergio/}, the Pascal2 network of
excellence, NSF grants 1128296, 1059235, 1055062, 1338118,
1035913, 0923494, and Spanish project TIN2013-43478-P. Our
sponsors include Microsoft and Texas Instrument who donated prizes
and provided technical support. The challenges were hosted by
Kaggle.com and Coralab.org who are gratefully acknowledged. We
thank our co-organizers of ChaLearn gesture and action recognition
challenges: Miguel Reyes, Jordi Gonzalez, Xavier Baro, Jamie
Shotton, Victor Ponce, Miguel Angel Bautista, and Hugo Jair
Escalante.}


\bibliography{jmlr}
\bibliographystyle{abbrv}

\end{document}
