This paper presents a data-oriented tracking fr a mework which aims to recover the spatio-temporal trajectories for an unknown number of interacting objects appearing and disappearing at arbitrary times. Data association is performed at three-levels of a hierarchy: (i)
rst, trajectory segments and an associated quality measure are generated by a local analysis of the space-time distribution of observations; (ii) a conservatively constrained association step links nearby consistent segments into intermediate trajectory fragments; and (iii) a last as- sociation step taking into account all available data (observations, trajectory fragments) generates the
nal trajectory estimates. The association step relies on the Hungarian algorithm and it also con- siders detection responses below the detection threshold as evidence associated with high ambiguity. We demonstrate the feasibility of the proposed approach applied to the pedestrian tracking task on two challenging datasets.