Detection of moving objects in a video sequence is a difficult task and robust moving object detection in video frames for video surveillance applications is a challenging problem. Object detection is a fundamental step for automated video analysis in many vision applications. Object detection in a video is usually performed by object detectors or background subtraction techniques. Frequently, an object detector requires manual labeling, while background subtraction needs a training sequence. To automate the analysis, object detection without a separate training phase becomes a critical task. This paper presents a survey of various techniques related to moving object detection and discussed the optimization process that can lead to improved object detection and the speed of formulating the low rank model for detected object.