Entry Date:
February 22, 2018

Moments in Time Dataset: A Large-Scale Dataset for Recognizing and Understanding Action in Videos

Principal Investigator Aude Oliva

Co-investigators Nick Montfort , Carl Vondrick , Dan Gutfreund


A joint MIT-IBM team set out to build a very large-scale dataset to help AI systems recognize and understand actions in videos. The dataset contains 1 million three-seconds video clips, each annotated with the actions that occur during the clips. The scientists curated data capturing real actions in day-to-day movements, according to the following principles: balance, wide coverage, three-second clips, and dataset diversity.

Moments is a research project in development by the MIT-IBM Watson AI Lab. The project is dedicated to building a very large-scale dataset to help AI systems recognize and understand actions and events in videos.

Today, the dataset includes a collection of one million labeled 3 second videos, involving people, animals, objects or natural phenomena, that capture the gist of a dynamic scene.