Crawler detection: A Bayesian approach

Athena Stassopoulou, Marios D. Dikaiakos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Citations (Scopus)

Abstract

In this paper, we introduce a probabilistic modeling approach for addressing the problem of Web robot detection from Web-server access logs. More specifically, we construct a Bayesian network that classifies automatically access-log sessions as being crawler- or human-induced, by combining various pieces of evidence proven to characterize crawler and human behavior. Our approach uses machine learning techniques to determine the parameters of the probabilistic model. We apply our method to real Web-server logs and obtain results that demonstrate the robustness and effectiveness of probabilistic reasoning for crawler detection.

Original languageEnglish
Title of host publicationInternational Conference on Internet Surveillance and Protection, ICISP'06
DOIs
Publication statusPublished - 2006
EventInternational Conference on Internet Surveillance and Protection, ICISP'06 - Cote d'Azur, France
Duration: 26 Aug 200628 Aug 2006

Other

OtherInternational Conference on Internet Surveillance and Protection, ICISP'06
CountryFrance
CityCote d'Azur
Period26/08/0628/08/06

Fingerprint Dive into the research topics of 'Crawler detection: A Bayesian approach'. Together they form a unique fingerprint.

  • Cite this

    Stassopoulou, A., & Dikaiakos, M. D. (2006). Crawler detection: A Bayesian approach. In International Conference on Internet Surveillance and Protection, ICISP'06 [1690400] https://doi.org/10.1109/ICISP.2006.7