Vytautas Bunevičius
Contact
Windows Malware Detection: Spotting the Bad Guys
I built a machine learning system to detect malicious Windows PE files, using data from over 23,000 executables. The goal? Catch malware before it can do damage by analyzing file structure and behavioral patterns. With cybersecurity threats growing every day, automated detection tools like this are crucial for keeping systems safe.
Windows Malware Detection Project Header
Project Overview and Methodology

What I Did

  • Feature Engineering: Started with Windows PE file analysis—extracted structural data, entropy calculations, import tables, section headers, you name it. Built features from file headers, text patterns, and behavioral signatures to capture what makes malware tick.
  • Finding the Patterns: Used feature selection techniques to pin down what separates malware from legitimate software. Text-derived features and entropy analysis kept showing up as key indicators.
  • Building the Models: Tried multiple approaches—Random Forest, XGBoost, and Neural Networks. Used Optuna for hyperparameter optimization to squeeze out the best performance, focusing on minimizing false positives while catching real threats.
  • Results: XGBoost came out on top with 97.63% accuracy and 98.72% precision. Translation: it catches malware reliably while keeping false alarms low—critical for production use.

How It Went Down

Started with raw PE file analysis—extracted features from 23,629 Windows executables covering everything from file headers to import tables. Built entropy calculations to spot packed or encrypted malware, analyzed section structures for suspicious patterns, and processed text strings that often reveal malicious intent.

Feature engineering was the secret sauce—combined structural patterns with behavioral indicators to catch sneaky malware. Tested three model types, optimizing for the right balance between catching threats and avoiding false positives. XGBoost won out—it's excellent at handling the complex feature interactions that separate malware from legitimate software.

The Nitty-Gritty

  • Dataset: 23,629 Windows PE files total. Training set of 18,914 files (62% malware, 38% benign), test set of 4,715 files with same distribution. Features included entropy values, section counts, import table analysis, and text pattern extraction.
  • Performance: XGBoost achieved 97.63% accuracy with 98.72% precision and 97.44% recall. False positive rate just 2.06%—crucial for real-world deployment where false alarms cost time and trust.
  • Model Insights: Feature engineering improved performance by 1.96%. Text-derived features were game-changers, and entropy analysis effectively caught packed malware. Hyperparameter optimization with Optuna found the sweet spot for production-ready detection.

What's Next

  • Dynamic analysis features could catch runtime behavior patterns.
  • Ensemble methods might squeeze out even better performance.
  • Real-time deployment would need optimized feature extraction pipelines.

Why It Matters

This system demonstrates how machine learning can automate malware detection at scale. With cyber threats evolving rapidly, automated tools like this help security teams stay ahead of attackers. It's not a silver bullet, but it's a solid foundation for modern cybersecurity defense. Check out the complete analysis and code on my GitHub!

1. Data Loading & Exploratory Analysis
Initial exploration of the Windows PE dataset, examining file distributions, malware vs. benign ratios, and key statistical patterns. This phase reveals the dataset structure and guides our feature engineering approach.
2. Feature Engineering & Selection
Advanced feature extraction from PE file headers, entropy calculations, import table analysis, and text pattern detection. This critical phase transforms raw file data into meaningful predictive features that capture malware behavior signatures.
3. Modeling & Evaluation
Model training and hyperparameter optimization using Random Forest, XGBoost, and Neural Networks. Comprehensive performance evaluation with focus on minimizing false positives while maintaining high detection rates for production deployment.