The exploration of large video data is a task which is now possible because of the advances made on object detection and tracking. Data mining techniques such as clustering are typically employed. Such techniques have mainly been applied for segmentation/indexation of video but knowledge extraction of the activity contained in the video has been only partially addressed. In this paper we present how video information is processed with the ultimate aim to achieve knowledge discovery of people activity in the video. First, objects of interest are detected in real time. Then, in an off-line process, we aim to perform knowledge discovery at two stages: 1) finding the main trajectory patterns of people in the video. 2) finding patterns of interaction between people and contextual objects in the scene. An agglomerative hierarchical clustering is employed at each stage. We present results obtained on real videos of the Torino metro (Italy).