Real-time web crawler detection

Andoena Balla, Athena Stassopoulou, Marios D. Dikaiakos

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

In this paper we present a methodology for detecting web crawlers in real time. We use decision trees to classify requests in real time, as originating from a crawler or human, while their session is ongoing. For this purpose we used machine learning techniques to identify the most important features that differentiate humans from crawlers. The method was tested in real time with the help of an emulator, using only a small number of requests. Our results demonstrate the effectiveness and applicability of our approach.

Original languageEnglish
Title of host publication2011 18th International Conference on Telecommunications, ICT 2011
Pages428-432
Number of pages5
DOIs
Publication statusPublished - 2011
Event2011 18th International Conference on Telecommunications, ICT 2011 - Ayia Napa, Cyprus
Duration: 8 May 201111 May 2011

Other

Other2011 18th International Conference on Telecommunications, ICT 2011
Country/TerritoryCyprus
CityAyia Napa
Period8/05/1111/05/11

Fingerprint

Dive into the research topics of 'Real-time web crawler detection'. Together they form a unique fingerprint.

Cite this