We extracted frames from 100DOH videos to build a new 100K frame-level dataset. Primarily, 85% of the frames comes from the subset of 100DOH and the rest 15% comes from VLOG Dataset (a 3x extended and relabeled version). We randomly picked frames and filtered out frames that do not contain hands.

For each frame, we provide annotations for:

• hand bounding box
• hand side: {left (L), right (R)}
• hand contact state: {no contact (N), self-contact (S), other person contact (O), portable object contact (P), stationary object contact (F, short for furniture)}
• object bounding box that the person is contacting irrespective of class name
• association between hand and in-contact object

Before using the dataset or models derived from it, we want our users to be aware that the dataset may not be entirely representative. The dataset is collected from YouTube via keyword searches in English. It covers diverse interaction activities, but it might not representative of all demographics and due to its sources, it is likely there are underlying biases and spurious correlations (e.g., certain people doing certain activities). Like all data, one should be cautious and skeptical.