Theses and Dissertations

ORCID

https://orcid.org/0009-0007-8973-843X

Advisor

Torri, Stephen

Committee Member

Gudla, Charan

Committee Member

Mitchell, Candice

Committee Member

Newell, Sandra

Committee Member

Trawick, George

Date of Degree

5-15-2026

Original embargo terms

Immediate Worldwide Access

Document Type

Dissertation - Open Access

Major

Computer Science

Degree Name

Doctor of Philosophy (Ph.D.)

College

James Worth Bagley College of Engineering

Department

Department of Computer Science and Engineering

Abstract

Modern Security Operations Centers (SOCs) ingest millions of log entries per day, but manual parsing does not scale to the volume, heterogeneity, and rapid evolution of log formats. This dissertation investigates whether unsupervised clustering can automate the generation of candidate field-extraction templates while remaining deployable in production and feasible under realistic runtime and memory constraints. The central research question asks whether a machine-learning-assisted pipeline that proposes extraction templates for security engineer review—relative to reproducible human-authored baselines such as hand-written regular expressions—can measurably improve rule-set deployability and corpus-scale extraction quality. Improvement is quantified using four applicability metrics: Coverage Rate (CR), Exclusive Match Rate (EMR), Ambiguity Rate (AR), and Regex Utilization Rate (RUR), as well as operational feasibility measures, including end-to-end runtime and peak memory usage. The evaluation dataset is a continuous 24-hour (00:00-23:59 UTC) corpus of endpoint and network telemetry from five enterprise networks operated by separate organizations to reduce bias. The corpus includes Windows event logs, Linux audit logs, and network events, totaling approximately 500 million log entries. To assess production feasibility while preserving the full workflow, production-scale execution runs the pipeline on a representative 1-hour slice of the 24-hour corpus using the same ingestion, normalization, feature construction, clustering, and template-generation stages. The proposed pipeline normalizes semi-structured logs, masks dynamic fields, represents log lines with hybrid TF-IDF feature vectors, performs scalable clustering under scheduler-enforced constraints, and derives regular-expression templates from cluster prototypes for analyst review and refinement. This work contributes a production-scale empirical evaluation framework for automated log template generation in defensive cyber operations, including applicability metrics that operationalize deployability and ambiguity, and an end-to-end reference implementation that quantifies outcomes under realistic computational constraints. Benchmarking on labeled data identified Bisecting K-Means and Mini-Batch K-Means as the most operationally viable clustering approaches, motivating their selection for production-scale testing. In production-scale experiments, the pipeline applies Mini-Batch K-Means to the 1-hour slice. It achieves RUR = 69%, CR = 100%, AR = 3%, and EMR = 97%, with a runtime of 9 hours, 24 minutes, and 5 seconds and peak memory usage of 354.08 GB under the production execution configuration. These results show that ML-assisted template suggestion can achieve near-complete corpus coverage with low ambiguity at enterprise scale, shifting analyst effort from manual rule authoring to review and refinement.

Download

Share

COinS