Comprehensive Suspicious Activity Monitoring Checklist for Linux Systems.

Discover how to enhance the security of your Linux systems with this detailed checklist for monitoring suspicious activities. Learn to identify signs of potential threats such as failed login attempts, unauthorized access, privilege escalations, unusual system behaviors, and more using a variety of command-line tools and log analysis techniques. Stay proactive against cyber threats by implementing these monitoring practices tailored to your environment's needs.

Logs

Suspicious Activity Monitoring Checklist for Linux:

Failed Login Attempts:

  • grep 'sshd.*Failed password' /var/log/auth.log (Excessive failed attempts can indicate brute-force attacks).

Successful Logins from Unusual Locations:

  • grep 'sshd.*Accepted password' /var/log/auth.log (Review the IP addresses and match them against known locations).

Privilege Escalations:

  • grep 'sudo:.*COMMAND=' /var/log/auth.log (Look for unfamiliar commands or patterns).

Unexpected User Creation:

  • grep 'new user' /var/log/auth.log (Look for user creation outside of regular maintenance periods).

Unusual Times of Activity:

  • Analyze /var/log/auth.log for login/logoff events during off-hours.

Denial of Service Indicators:

  • dmesg | grep -i 'out of memory' (System running out of memory due to resource exhaustion).

UFW Firewall Blocks:

  • grep DENY /var/log/ufw.log (Multiple denied connections from a single IP might indicate scanning or attack attempts).

Unexpected System Reboots or Shutdowns:

  • grep -i 'shutting down\\\\|reboot' /var/log/messages (Could indicate power issues, hardware failures, or malicious activities).

Kernel Errors or Warnings:

  • grep -i 'warn\\\\|error' /var/log/kern.log (Hardware issues, software crashes, or potential exploits).

Root Actions:

  • grep 'root' /var/log/auth.log (Actions done by the root user should always be scrutinized).

Changes to Critical System Files:

  • Monitoring tools like aide or tripwire can alert to unexpected changes to system binaries or configuration files.

Unusual Network Connections:

  • Monitoring tools like netstat or ss can be used to check for unfamiliar connections.

Cron Jobs Execution:

  • grep CMD /var/log/cron (Look for unfamiliar cron jobs or those executing at strange intervals).

Unusual Processes:

  • Regularly review running processes using ps or top for anything unfamiliar.

Large Data Transfers:

  • Monitoring network traffic using tools like iftop or nload to detect potential data exfiltration.

Auditd Anomalies:

  • If auditd is set up, you should monitor for unauthorized system calls or actions.

Suspicious Commands in History Files:

  • Checking user's shell history files for malicious or unfamiliar commands.

Unusual File Access Patterns:

  • Monitoring tools like inotify can alert for unexpected file access patterns, especially in critical directories.

Unexpected Package Installations:

  • grep 'install ' /var/log/apt/history.log or grep 'Installed:' /var/log/yum.log (Look for software installations outside of regular maintenance periods).

Unexpected External Storage Devices:

  • Monitor for unexpected mounting of external storage devices.

/var/log/auth.log & /var/log/secure:

Authentication logs, including user logins and authentication mechanisms.

User Authentications & Failures:

  • grep 'sshd.*Failed password' /var/log/auth.log
  • grep 'sshd.*Accepted password' /var/log/auth.log

Privilege Escalations:

  • grep 'sudo:' /var/log/auth.log

Unexpected User Actions:

  • grep 'NEW user' /var/log/auth.log

/var/log/syslog & /var/log/messages:

General system logs, capturing a variety of messages and system activities.


System Errors:

  • grep -i 'error' /var/log/syslog

Cron Activities:

  • grep CRON /var/log/syslog

/var/log/dmesg:

Kernel boot and runtime messages.


Hardware Issues:

  • dmesg | grep -i error

Kernel Messages:

  • dmesg | grep -i warn

/var/log/kern.log:

Kernel-specific logs.

Kernel Errors & Warnings:

  • grep -i 'warn\\\\|error' /var/log/kern.log

/var/log/boot.log:

System boot logs.

Boot Failures:

  • grep -i fail /var/log/boot.log

/var/log/faillog:

Failed user logins.

Failed Logins:

  • faillog -a

/var/log/ufw.log:

UFW firewall logs.

Denied Connections:

  • grep DENY /var/log/ufw.log

/var/log/maillog & /var/log/mail.log:

Mail server logs.


Failed Email Deliveries:

  • grep -i fail /var/log/mail.log

Successful Email Deliveries:

  • grep -i success /var/log/mail.log

/var/log/apt/history.log:

APT package manager logs.


Software Installations:

  • grep 'install ' /var/log/apt/history.log

Software Removals:

  • grep 'remove ' /var/log/apt/history.log

/var/log/yum.log:

For systems using YUM, logs related to package management.

YUM Installations & Updates:

  • grep 'Installed:' /var/log/yum.log

/var/log/cron:

Cron job logs.


Executed Cron Jobs:

  • grep CMD /var/log/cron

Custom Application Logs:

Many applications generate their own logs. For instance, web server logs, database logs, etc. You should monitor these based on the software stack of your system.

Apache Access & Errors (Example for Apache Web Server):

  • grep -i error /var/log/apache2/error.log
  • grep '404' /var/log/apache2/access.log

Database Queries & Errors (Example for MySQL):

  • grep -i error /var/log/mysql/error.log

Additional Considerations:

  • Log Rotation: Regularly rotate logs to ensure that storage does not become an issue and older logs are archived for future reference.


  • Integrity: Use tools like aide or tripwire to ensure logs are not tampered with.


  • Centralized Logging: Consider setting up centralized logging solutions, such as the ELK Stack (Elasticsearch, Logstash, Kibana) or Graylog, to aggregate and analyze logs from multiple sources.


  • Auditd: For systems where deep security monitoring is required, consider setting up auditd for advanced auditing and monitoring of system calls.


This checklist provides a starting point, but always ensure that you tailor it to your environment's needs, and stay updated with best practices and the evolving threat landscape.


/var/log/auth.log or /var/log/secure:

Multiple failed login attempts from the same IP.

  • grep 'Failed password' /var/log/auth.log

/var/log/syslog or /var/log/messages:

Unusual system reboots or shutdowns.

  • grep 'reboot' /var/log/syslog

/var/log/boot.log:

  • No common malicious patterns, but inconsistencies in the boot process could indicate issues.

/var/log/dmesg:

  • Again, no common malicious patterns. Mostly hardware and driver messages.

/var/log/kern.log:

Look for unexpected kernel module insertions.

  • grep 'module verification fail' /var/log/kern.log

/var/log/cron.log:

Unrecognized cron jobs.

  • grep 'CMD' /var/log/cron.log

/var/log/mail.log or /var/log/maillog:

Numerous sent emails (potential spamming or exfiltration).

  • grep 'status=sent' /var/log/mail.log

/var/log/apache2/access.log:

Multiple requests from one IP in short succession (potential scanning or DoS).

  • grep 'GET' /var/log/apache2/access.log | sort | uniq -c | sort -nr

/var/log/apache2/error.log:

PHP or SQL errors indicating potential injection attacks.

  • grep 'PHP Fatal error' /var/log/apache2/error.log

/var/log/mysql.log & /var/log/mysql.err:

Unauthorized access attempts.

  • grep 'Access denied for user' /var/log/mysql.log

/var/log/faillog:

Repeated failed login attempts.

  • cat /var/log/faillog

/var/log/ufw.log:

Multiple blocked requests from the same IP.

  • grep 'BLOCK' /var/log/ufw.log

/var/log/audit/audit.log:

Suspicious SELinux policy violations.

  • grep 'avc: denied' /var/log/audit/audit.log

/var/log/yum.log:

Unexpected packages being installed.

  • grep 'Installed' /var/log/yum.log

/var/log/apt/history.log:

Unexpected packages being installed.

  • grep 'install ' /var/log/apt/history.log

/var/log/btmp:

Repeated failed login attempts.

  • last -f /var/log/btmp

/var/log/wtmp:

Logins at unusual times.

  • last -f /var/log/wtmp

/var/log/journal/:

Suspicious systemd service failures.

  • journalctl -xe | grep 'failed'

SSH Brute Force Attempts:

Multiple failed SSH logins.

  • grep 'Failed password for' /var/log/auth.log

Unexpected Sudo Access:

Unauthorized users attempting to gain superuser privileges.

  • grep 'COMMAND=' /var/log/auth.log

Web Server Directory Traversal:

Potential directory traversal attack.

  • grep '/../' /var/log/apache2/access.log

SQL Injection:

Signs of SQL injection in web server logs.

  • grep "UNION SELECT\\\\|' OR '1'='1" /var/log/apache2/access.log

PHP Errors Indicating Exploits:

PHP errors that might indicate attempts to exploit the server.

  • grep 'PHP Parse error\\\\|PHP Warning' /var/log/apache2/error.log

Unexpected Cron Jobs:

Unknown commands running from cron.

  • grep 'CMD' /var/log/cron.log

Mail Server Spamming:

A large number of emails sent in a short time.

  • grep 'status=sent' /var/log/mail.log | sort | uniq -c | sort -nr

FTP Unauthorized Access:

Failed FTP login attempts.

  • grep 'Login incorrect' /var/log/vsftpd.log

Suspicious Kernel Activities:

Unexpected kernel module loading.

  • grep 'loading out-of-tree module' /var/log/kern.log

Web Server Request Flood:

An unusual number of requests from a single IP.

  • awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -nr

IPTables Block Log:

IP addresses getting blocked frequently.

  • grep 'DROP' /var/log/iptables.log

Unusual Time of Login:

Logins during off hours.

  • last | grep "2:00am\\\\|3:00am"

Mail Server Relay Access:

Unauthorized relay access attempts.

  • grep 'Relay access denied' /var/log/mail.log

Excessive Root Actions:

Multiple activities by the root user.

  • grep 'root' /var/log/auth.log

Nginx 404 Errors:

Excessive 404 errors can indicate scanning activity.

  • grep '404' /var/log/nginx/access.log

Unexpected System Reboots:

Unusual reboots or shutdowns.

  • grep 'reboot\\\\|shutdown' /var/log/syslog

Unusual User Creation:

New users being added.

  • grep 'new user' /var/log/auth.log

Bind DNS Exploits:

Potential DNS server exploit attempts.

  • grep 'unexpected RCODE' /var/log/named.log

Daemon Errors:

Unexpected service/daemon failures.

  • grep 'fail' /var/log/daemon.log

Unusual Time of CRON Execution:

Cron jobs running at odd times.

  • grep '3:00am' /var/log/cron.log

In conclusion, maintaining the security of Linux systems requires vigilance and proactive monitoring of suspicious activities. By following the checklist provided in this blog post, administrators can effectively detect and respond to potential threats, safeguarding their systems against unauthorized access, privilege escalation, and other malicious activities. Regularly reviewing system logs, employing monitoring tools, and staying updated with the latest security best practices are essential steps in ensuring the integrity and confidentiality of Linux environments. With a robust monitoring strategy in place, organizations can better protect their assets and mitigate the risks posed by evolving cyber threats.


Author: RB