Enterprise Networks & Servers: Network Detection of Worms and Viruses

In last month's article we reviewed the ground rules that seem to be changing. We also looked at controlling the rapid spread of viruses and worms, with an access list (ACL) configuration example. Because of space limitations, we deferred discussion of network-based virus/worm detection for this month.

This article delivers on that promise! Note that we are NOT going to talk about Intrusion Detection Systems (IDS), since detecting and responding to events is what they are supposed to do. This article is focused on other approaches. We may come back to IDS's in a future article.

Here's the short list of network methods for detecting a worm or virus attack. The assumption is that the worm is actively trying to propagate itself to other machines. There are other similar methods not on the list; once you've gotten the idea, you can invent your own variations on this theme.

* NBAR discovery

* RMON2 at L7 with RMON probe or NAM

* NetFlow (show commands or w/a NetFlow collection/reporting system)

* Switch MLS statistics

* ACL or VACL

* Sinkhole router

* Packet analyzer software

* Host firewalls

* IDS or PIX reporting (CW V/SMS Monitoring for Security; SIMS)

In the discussions I've seen online, various network administrators have historically been first tipped off by high levels of traffic, broadcasts, etc. These tips came from routine performance reporting they were doing, or occasionally from fortuitous router crashes. When they started digging for the cause, they then often used one of the above techniques.

Lesson Learned #1: Get some performance management in place. It could be Solar Winds Orion, NetScout Performance Reporting, Concord, MRTG, Cricket, or another tool. But get it, use it, and look at the reports! It can help you make your network perform better, and it can tip you off to virus and worm attacks.

We notice that the above detection methods all have something in common. They look at the anomalous traffic, and use that to spot unusual activity. The source of the unusual activity is a compromised or infected host. That's it.

How do you find out what port(s) to watch? Either from reading online about the worm, or from noticing unusual activity on some port(s). Once the worm behavior is characterized, you've got definitive ports. But until then, and until you can get through to the virus vendor Web pages, you may need to proceed on your own. That's where hyperactive ports play a role.

NBAR Discovery

The first item on the above list is NBAR discovery. This tells you about #packets and bytes to various TCP and UDP ports. If you know what's "normal," then this can tip you off to the presence of a virus. NBAR discovery can have performance impact on your router. NBAR discovery only produces highly aggregated data. In particular, it's not going to show who is transmitting to whom. It isn't going to tell you who may have the virus. But NBAR can also be used to rate-limit and control or drop the traffic the worm uses to propagate itself.

Here's a sample. The format shows data in sets of triple lines. The first line of three is the Packet Count, next Byte Count, and the third line is Bits per Second. The following shows the top 5 (plus unknown), you might want the top 10 or 20 in a real network.

The "napster" is in fact from trying to telnet to my PC on port 5555, so you have to be flexible in interpreting the results. I used a program called netcat to send some UDP traffic to port 5678, which is what was showing up as "unknown." After I configured Voice 11(config)#ip nbar port-map custom-01 udp 5678, it showed up as "custom-01" as in the above results. Admittedly, for this to be very useful, you have to know what ports you're interested in, which means you already know something about what's biting you.

If you only want bps, the command would be: show ip nbar proto stats bit-rate top-n 5. We didn't do this above since the bps rates are all 0 (due to using short-lived traffic bursts!).

RMON2

If you have RMON2 probes (or NAM blades) in key parts of the network, they can show the traffic mix for various TCP and UDP ports, similarly to NBAR discovery. (Except that most reporting tools show nice bar charts instead of just numbers). If the probe is collecting source address and destination address and port information, you can then drill down, to find out who is transmitting high volumes to certain ports. We'd love to show you sample screen captures, but our home labs don't have a NAM.

NetFlow

NetFlow can provide similar data. If you know what percentage of your traffic normally goes to various ports, then you can spot unusual levels of traffic to some port(s). This would be from data aggregated by port. You can then examine the unaggregated traffic going to those ports, looking first at source addresses sorted to show high volume to the selected port(s). You can also look for abnormal traffic patterns, such as devices systematically scanning all devices in a subnet. For example, at one customer's site we saw Nachi infected hosts displayed as the source of many flows that are destined to random destinations. By working with the MIS folks, these hosts were quickly isolated.

This technique assumes you've put NetFlow collection and reporting in place ahead of time. See www.cisco.com/warp/public/732/Tech/nmp/netflow/index.shtml for more information. Cisco sells a NetFlow collection tool. See www.cisco.com/en/US/products/sw/netmgtsw/ps1964/index.html. Cisco's strategy is for partners to provide the analysis and reporting. Pete hears that Concord, Info Vista, NetQoS and others have some NetFlow reporting capabilities (but hasn't delved into what they can and cannot do). Some are mentioned in the slightly old document at www.cisco.com/warp/public/cc/pd/ iosw/prodlit/netfl_wp.htm.

See also www.caida.org/tools/measurement/cflowd/ for free tools for working with NetFlow. See also www.caida.org/tools/utilities/flowscan/ and some of the other Caida tools.

Cisco has proposed NetFlow version 9 to the IETF as a way to standardize NetFlow data formats and schemas across vendors to allow network management companies to develop cross-vendor traffic reporting tools. The IPFIX working group is pursuing this, taking into account technologies from other vendors as well. See www.ietf.org/html.charters/ipfixcharter.html.

If you don't have the tools in place, you still may be able to use NetFlow. If you have a lot of activity going on, it's going to show up in the active flow cache.

If you've looked at the above really carefully, you'll notice the DstP (destination port) must be in hex, since 0x0017 is 23, the usual telnet port.

You may well want to be more specific than this. Use the '|' modifier to filter so as to see just what you need to see. This probably does mean looking at the full output at least once. (Big scroll buffer or logging for offline examination in a text editor may help.)

Switch MLS Statistics

If you have Catalyst 6500s running CatOS, consider configuring "set mis flow full." You can then do things like "show mis statistics entry ip dst-port 135" to see what's sending to that port. (This idea is from the Cisco TAC guide on W32.Blaster Worm. See that document for sample output.)

For 6500s with Native IOS, "mis flow ip full" configures it; "show mis ip" and variations display the data.

ACL or VACL

Be careful! Access lists can seriously impact router performance.

Ok, so if you know what ports you care about, that shows if there's any traffic on them.

If you beef up the access list by adding "log" at the end of selected lines, you'll get console or telnet session logging of matches. (Be very careful, doing this for all packets will kill your router.). If you include a rule specifying a UDP or TCP port, then the log entries will also show port numbers. If you have configured syslog logging to Cisco Works (or a Windows or Unix/Linux host running a version of syslogd), you can capture the log messages in a file for further analysis. (With Cisco Works, just check the tail end of .../CSCOpx/log/syslog.log).

You can also use the "log" keyword with VACLs in Catalyst 6500 switches (PFC needed; MSFC not needed). Similar cautions apply.

Bearing in mind our performance cautions, there are a couple of ways to use this trick. One is to live dangerously by applying and then removing the ACL quickly. Make sure you have console logging off before you do this. (Log to buffer or remote syslog). You did say you enjoy playing Russian roulette, didn't you?

Another tip is to restrict what you log. If you know what ports the worm is exploiting, log only those ports. One way to render such logging even safer is to use spare or lab gear, where if you make the router busy it doesn't matter. Enough warnings.

What does this buy us? The big thing to notice here is that we can see the source addresses. So if we're logging a port that only a worm is likely to be using, then the source is an infected computer (unless the worm spoofs source addresses, of course).

Sinkhole Router

One related idea is to create a so-called "sinkhole router." The key assumption is that worms randomly select addresses to try to infect. Some such addresses may well be bogus, as in unassigned or reserved address space. The idea is, therefore, to set up a dedicated (old?) router that is a "magnet" for such packets. This router may well see packets from many if not all of the infected hosts in the network. And if you turn on access list logging on this sinkhole router, it sooner or later spots each infected host for you.

This idea solves two problems. You really don't want to be messing around with access list logging on live routers. And you don't really want to be doing it all over your network. The sinkhole router localizes the work and the risk on a spare router. Much less work, much less risk!

To set up a sinkhole router, create static routes to Null O for bogus destination networks. These can include all private network address space not in use within your network. Then redistribute these static routes into your internal routing protocol. Add your favorite access list with logging, set up syslog logging (or use terminal monitor), and you're gathering info.

A variation on this lets you reduce network-wide routing table impact. I like to have Internet gateway routers configured to originate default. So packets to bogus destinations will normally head for the gateways. Put the prospective sinkhole router on a LAN switch connecting to one if not both gateways. Then put the static routes on the gateways to send packets to bogus destinations to the sinkhole router.

This begs the question of what network destinations are bogus (unassigned). There is a list of registries and other address assignment info at www.cert.mil/techtips/ whois_by_ipaddr.htm. In particular, you may find the following document useful: www.iana.org/assignments/ipv4-address-space. That certainly gets you in the right ballpark. It may not be worth trying to be more precise than that.

For a specific router template with static routes to NULL O for bogus destinations, see also www.cymru.com/Documents/secure-ios-template.html. If being careful, check it against the IANA list as it may be a bit old/out of date.

Packet Analyzer Software

If you have packet analyzer software such as Network Associates' Sniffer, WildPackets' EtherPeek, or Fluke Networks' OptiView Protocol Expert, great. If not, consider the free Ethereal program for Windows or Linux, or tcpdump for Unix/Linux at www.ethereal.com. If you use VNC, you can use these applications for remote packet capture. Pull the capture file back and analyze it at your desk in comfort, (www.tightvnc.com) So using this software, you can hook dedicated or spare computers to switch SPAN ports and be ready to protocol analyze. This can be a great tool for troubleshooting.

The previous paragraph could give you an idea of how to use a packet analyzer in a worm storm. If you know that a certain host is infected, watch its traffic to see what the worm or virus does to spread. You can then set up a capture filter and capture all traffic to the ports used by the worm. Source addresses tell you who else might be infected.

Host Firewalls

OpenBSD systems can filter inbound traffic using the built-in packet filter "pf." If you log the blocked packets, they can easily be viewed by running tcpdump on a pseudo-interface. If you watch this, any unusual traffic to the system or probes by worms or viruses may be visible.

Personal firewalls on some computers are another early warning system. When they start popping up in dialog boxes repeatedly, you know a worm storm is going on.

Loosely fitting under this heading is the Cisco CSA product, www.cisco.com/en/US/products/sw/secursw/ps5057/. This product is a hot seller recently, I hear. The claim is that without any tweaking, it stopped all the recent attacks. One attraction is that it does not use signatures that need constant updates, the way virus scanners do. Since the attacks are now getting faster than the turn-around time on the signatures, that's an attractive feature. The tie-in to the present article: like most personal firewall products, it lets you know when you're under attack.

You might be thinking, "Yes, but those will only report if one system is under attack. What are the odds the randomly generated address will match that one system?" Good point! But consider that you would probably be using tools like this to protect all the servers, and possibly some laptops or desktops. If the alerts go to a central console, now you've got another source of info about what sources might be infected, etc.

IDS or PIX Reporting (CW V/SMS Monitoring for Security; SIMS)

As noted earlier, reporting on things like worm or virus activity is what we'd hope an IDS would do. The PIX and even the security/firewall code in the routers can do some attack detection and alarming. With other vendors' firewalls, YMMV (Your Mileage May Vary). Since this article is long enough already, we won't go into this topic further at this time.

Links

The following are some of the links that I've gleaned the above ideas from. The other document I've referred to which is not shown is the Cisco Networkers 2003 presentation SEC-2004. It's great to see so many creative ideas coming from the minds at Cisco. And since great ideas don't do any good unless you tell someone about them, thanks to the folks that published these items. We're mainly contributing to the effort by summarizing and focusing the discussion and passing on the links.

Combating Internet Worms: An Integrated Security Approach: Cisco Internet seminar on 9/4/03

www.cisco.com/go/semreg/blaster/123542_8

Uses of Network Management for Monitoring the IP Packet Blocks Input Queue PSIRT Advisory www.cisco.com/en/US/products/hw/routers/psl33/products_white_paper09186a00801a5a68 .shtml

Strategies to Protect Against Distributed Denial of Service (DDoS) Attacks www.cisco.com/warp/public/707/newsflash.html

Characterizing and Tracing Packet Floods Using Cisco Routers www.cisco.com/warp/public/707/22.html

Cisco Security Notice: W32.BLASTER Worm Mitigation Recommendations www.cisco.com/warp/public/707/cisco-sn-20030814-blaster.shtml

Cisco Security Notice: Nachi Worm Mitigation Recommendations www.cisco.com/warp/public/707/cisco-sn-20030820-nachi.shtml

Conclusion

Increasing competence at firewalls may shift attacker focus to viruses, worms, Trojan horses, Web sites that implant Trojan horses, and social engineering. Although there seem to be enough sites that still need to tighten up their firewalls or make better use of their DMZs, so that there's still no shortage of easy targets. We'd rather be a hard target. What do you think?

Next month we'll continue the technical discussion by looking at Best Practices for servers, routers and switches. We think there's some new things to be said on that front as well. Yeah, some potential for it to get dry, so we'll try to provide extra lively words to make up for that. Stay tuned!

Peter J. Welcher & Carole Warner Reece

Dr. Peter J. Welcher (CCIE #1773, CCSI #94014, CCIP) is a senior consultant with Chesapeake NetCraftsmen, a high-end consulting firm and Cisco Premier Partner dedicated to quality consulting and knowledge transfer. NetCraftsmen has eight CCIEs with expertise including large network high-availability routing/switching and design, VoIP, QoS, MPLS, IPSec VPN, wireless LAN and bridging, network management, security, IP multicast, and other areas. Pete 's links start at www.netcrqftsmen.net/welcher. New articles will be posted under the Articles link. Questions, suggestions for articles, etc. can be sent to pjw@netcraftsmen.net.

Carole Warner Reece (CCIE #5168) is a senior consultant with Chesapeake NetCraftsmen with over 15 years of experience in internetworking. Carole recently designed challenging new labs for Version 5 of the Cisco CIT Troubleshooting course. She previously developed and worked with other authors to produce networking skills based labs at a wide variety of levels.