AUT LibraryAUT
View Item 
  •   Open Theses & Dissertations
  • Masters Theses
  • View Item
  •   Open Theses & Dissertations
  • Masters Theses
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

Data mining log file streams for the detection of anomalies

Green, Brian
Thumbnail
View/Open
Whole thesis (1.587Mb)
Permanent link
http://hdl.handle.net/10292/9214
Metadata
Show full metadata
Abstract
Log files play an important part in the day to day running of many systems and services, allowing administrators and other users to gain insights into operational, performance or even security issues but it is now impractical with the volume of files today to manually examine them.

Existing tools in this space largely work by detecting anomalies from log files that have already been stored or by comparing them against known errors (signatures). By data mining log file streams for the detection of anomalies instead, it will allow administrators to reduce the time required to detect anomalies significantly with no signatures or complex settings needing to be maintained.

This paper presents the experimental work undertaken to define a generic, practical and scalable method for anomaly detection in streaming log files by detecting the change to the mix of log events occurring. This was achieved by following a modified CRISP-DM (Cross Industry Standard Process for Data Mining) methodology enabling a broader more flexible approach to the data mining process.

By taking this approach, a solution was developed that employs common log file features together with a weighted earth mover distance metric. This enabled a framework to be developed that can be broadly applied to many log file types. By setting a simple percentile threshold indicating an acceptable level of change, anomaly detection in streaming log files can be achieved.
Keywords
Data mining; Logs; Streaming; Anomalies; Expermental
Date
2015
Item Type
Thesis
Supervisor(s)
Russel, Pears
Degree Name
Master of Computer and Information Sciences
Publisher
Auckland University of Technology

Contact Us
  • Admin

Hosted by Tuwhera, an initiative of the Auckland University of Technology Library

 

 

Browse

Open Theses & DissertationsTitlesAuthorsDateThesis SupervisorMasters ThesesTitlesAuthorsDateThesis Supervisor

Alternative metrics

 

Statistics

For this itemFor all Open Theses & Dissertations

Share

 
Follow @AUT_SC

Contact Us
  • Admin

Hosted by Tuwhera, an initiative of the Auckland University of Technology Library