IJCATR Volume 7 Issue 8

Web Scraping for Estimating new Record from Source Site

Warna Agung Cahyono , Wijono , Herman Tolle
10.7753/IJCATR0708.1009
keywords : Web Scrapping, Exponential Smoothing, MDR, STM

PDF
Study in the Competitive field of Intelligent, and studies in the field of Web Scraping, have a symbiotic relationship mutualism. In the information age today, the website serves as a main source. The research focus is on how to get data from websites and how to slow down the intensity of the download. The problem that arises is the website sources are autonomous so that vulnerable changes the structure of the content at any time. The next problem is the system intrusion detection snort installed on the server to detect bot crawler. So the researchers propose the use of the methods of Mining Data Records and the method of Exponential Smoothing so that adaptive to changes in the structure of the content and do a browse or fetch automatically follow the pattern of the occurrences of the news. The results of the tests, with the threshold 0.3 for MDR and similarity threshold score 0.65 for STM, using recall and precision values produce f-measure average 92.6%. While the results of the tests of the exponential estimation smoothing using ? = 0.5 produces MAE 18.2 datarecord duplicate. It slowed down to 3.6 datarecord from 21.8 datarecord results schedule download/fetch fix in an average time of occurrence news.
@artical{w782018ijcatr07081010,
Title = "Web Scraping for Estimating new Record from Source Site",
Journal ="International Journal of Computer Applications Technology and Research(IJCATR)",
Volume = "7",
Issue ="8",
Pages ="292 - 369",
Year = "2018",
Authors ="Warna Agung Cahyono , Wijono , Herman Tolle"}