Web robot detection: A probabilistic reasoning approach

Athena Stassopoulou, Marios D. Dikaiakos

Research output: Contribution to journalArticlepeer-review

72 Citations (Scopus)

Abstract

In this paper, we introduce a probabilistic modeling approach for addressing the problem of Web robot detection from Web-server access logs. More specifically, we construct a Bayesian network that classifies automatically access log sessions as being crawler- or human-induced, by combining various pieces of evidence proven to characterize crawler and human behavior. Our approach uses an adaptive-threshold technique to extract Web sessions from access logs. Then, we apply machine learning techniques to determine the parameters of the probabilistic model. The resulting classification is based on the maximum posterior probability of all classes given the available evidence. We apply our method to real Web-server logs and obtain results that demonstrate the robustness and effectiveness of probabilistic reasoning for crawler detection.

Original languageEnglish
Pages (from-to)265-278
Number of pages14
JournalComputer Networks
Volume53
Issue number3
DOIs
Publication statusPublished - 27 Feb 2009

Keywords

  • Bayesian classifiers
  • Probabilistic reasoning
  • Web crawler detection

Fingerprint

Dive into the research topics of 'Web robot detection: A probabilistic reasoning approach'. Together they form a unique fingerprint.

Cite this