Looking for holes

Anyone seeking to deny intruders any opportunity needs to examine their network regularly, looking for security-relevant vulnerabilities. There are plenty of scanners available to do this. Testing shows how effective they are.

1. A good vulnerability scanner does not simply find a lot of vulnerabilities, but it is just as important that the number of false positive messages should be kept as low as possible.
2. For a company that uses a mixture of IT assets, a scanner should produce satisfactory results for every possible operating system and be capable of being integrated into any existing vulnerability management systems.
3. Such scans are a useful way of improving security in the corporate network – however, one cannot rule out the possibility of an attack still occurring.

In the iX test, nine software and hardware vulnerability scanners (appliances) were put under the microscope. The companies NetIQ, Ncircle and Symantec did not provide any test software The test network in which the products were tested comprised 19 systems with operating systems from different manufacturers, dating from 1996 up to the present day, and different hardware. When one scans a network for vulnerabilities and security problems, one does not simply gain information about its status. Rather, the System Administrator uses the data generated to determine what applications such as patch and risk management, problem handling etc. are needed, which is thus critically influenced by the quality of the results. Some of the products provide other vulnerability management functions in addition to scanning, for example, for gathering data on system status, patch management or ticket-based problem management. However, our tests concentrated solely on the primary functions, namely, the detection and description of vulnerabilities. The main action performed by the test candidates was to search over the network for open ports and the services running on them. In addition, assuming that they had this capability, they were allowed to log on to various systems and to carry out local investigations. The test network consisted of an intranet with 19 computers. 13 of these ran on a machine with 3GHz processor and 4GB RAM as a virtual computer. All of them were connected with a switch and had access to the internet via a router with Network Address Translation (NAT). Each of the software scanners was installed in a separate virtual machine. With hindsight, the test environment was not ideal, but all the scanners were on a level footing since the same set of assumptions applied to each of them.

The scan results are also relevant to intrusion prevention and detection systems (IPS and IDS). Moreover, in larger networks several scanners, whose results the Administrator will want to compare, are frequently used. For this reason, it is desirable that the data should be delivered in a standardised format. Unfortunately, each scanner has its own formats and therefore its results require “translation”. In this area, solutions are available from two standardisation bodies for the XML schema. In 2003 OASIS published the Application Vulnerability Description Language (AVDL, www.avdl.org), which has not been developed any further since then, while a year earlier MITRE offered an XML schema for describing vulnerabilities, patches, requirements, data collection and results in the form of the Open Vulnerability and Assessment Language (OVAL, oval.mitre.org). The organisation has already defined the most well-known vulnerability standard for naming and enumerating known vulnerabilities in a uniform way, the Common Vulnerabilities and Exposures (CVE, cve.mitre.org). When it comes to weighting the vulnerabilities, there is a further lack of compatibility: every scanner classifies the risk level (highly critical, medium, low) in its own idiosyncratic way, which not only makes it difficult to assess the vulnerabilities found, but also makes it difficult to compare the scanners. The Administrator should be cautious in handling results which put the lion’s share of the vulnerabilities detected either very high or very low in the scale of threats. It would be more realistic for around 20 percent of vulnerabilities to be classified as highly critical, 50 percent medium and 30 percent less critical. The National Infrastructure Advisory Council (NIAC) has recently been trying to establish standardised measuring units for the degree and urgency of vulnerabilities, with its Common Vulnerability Scoring System (CVSS, www.first. org/cvss/). Every scanner was given two opportunities to show what it could do. The first pass was performed with the standard settings for a quick run, which carries out all the tests that do not hurt the target system. In the second pass, every scanner was initiated manually with all the available options. These included a complete scan of all the ports. In addition, where possible, the scanner was given a password. On the basis of security aspects, on UNIX systems the scanner logs on as a normal user, whereas Windows systems require Administrator access for certain functions such as reading the Registry. The present evaluation is based on the results of the second pass. The number of messages generated is not necessarily an indication of the quality of a good scanner. For example, the Saintbox produces relatively few messages, but these contain a lot of problems and also their CVE numbers, so this product comes out very well when evaluated on the basis of CVE numbers. One would expect the results of the second pass to be more extensive than the first, partly because the scanners were now examining more ports, but mainly because most of them also logged on to the systems and were thus able to generate more results, which were also more precise. If there is no increase, this suggests that little has been carried out in the way of local tests, either because the scanner does not support this feature or because it did not work.

The central aspect of the evaluation of the data gathered in the test is the analysis of the correctness and completeness of the messages. The measure used is the number of CVE numbers found: firstly the correct ones, i.e. all the vulnerabilities found that actually are vulnerabilities, secondly the false messages, that is items that were incorrectly classified as vulnerabilities, and thirdly the missing vulnerabilities that were not detected. The latter is calculated as the difference between the CVE numbers found by a given scanner and the numbers found by all of them together. In this way it is possible with reasonable effort to obtain a useful approximation to the number of missing messages.

Thoroughness and accuracy (Exact) as a percentage: “thorough” refers to the proportion of CVE numbers out of the set of all the correct CVEs found by the relevant scanner, “Exact” refers to the proportion of the CVEs detected which really existed. The higher the two values are, the better the scanner is

In the present test, the output of CVE numbers was an important criterion for assessing the quality of the scanners. Whereas a scanner that does not do this does not necessarily overlook problems, this can still be a disadvantage. For example, consider an old version of a web server that contains many vulnerabilities. Each of them has a CVE number. For the Administrator it is sufficient to receive only a single message which tells him that the version is out of date and poses a risk. On the other hand, for a penetration test or other attempt to accurately assess the degree of risk, it is necessary to know the CVE numbers.

For this reason, scanners that generates many true CVE messages perform well in the comparison stakes. With this method it is not possible to compare scanners that do not use CVE numbers. Of the scanners tested, the only one that fell into this category was the one from GFI. The more CVEs a scanner reports, the higher the risk that this will includes false positives, but the probability that it will report all the existing CVEs is also higher. In the test, a script scrutinised the XML versions of the results reports, looking for CVE numbers. Altogether, 1,745 different CVE/target system combinations were extracted from the 4,414 CVEs reported by the scanners. Each of these combinations stands for one specific problem on one system. With all the combinations it was necessary to assess whether the result was a genuine finding or a false positive. The individual judgements were based on attempts to perform exploits, research in Security focus.com’s vulnerability database and manufacturers’ advisories. In this way a total of 787 true and 509 false CVEs was crystallised. 449 could not be clearly determined as either true or false. The results obtained with the individual scanners were then compared against the approximately 1,300 verified and reliably determinable CVE numbers (see Figure 1). The dark blue bars show the percentage proportion of the total population of correct CVEs that were found by a scanner. The light blue bars show how many of these were true vulnerabilities. To give an example, Nessus reported 608 CVEs out of the 787 possible, and of the CVEs it reported 459 were true. This corresponds to a thoroughness of 58 percent and an accuracy of 75 percent.

Both numbers are important and are also inseparable. A scanner is only useful if it finds a high proportion of the potential vulnerabilities and the number of false positives is as low as possible. Otherwise too many resources are wasted on non-existent problems. The discrepancies in the findings reported are serious for virtually all the scanners. For example, a number of the scanners find a vulnerability for Linux rpc.statd even on the BSD systems, or an NT 4 exploit is found on Windows 2003. Evidently the scanners rarely, if at all, carry out any plausibility checking against facts already known on the relevant systems. The same is true for old tests which there has been no point in running for some years now. Updates of the existing tests are seldom found. Similarly, the function of transferring the results of tests carried out on one system to other systems could also be optimised. One example is passwords: if a scanner finds a valid password on one system, it could be profitable to try this password out on other systems. To determine the suitability of the scanners for heterogeneous IT landscapes, the scan results were considered for the individual operating systems. No scanner detected all the CVE numbers on all the operating systems. Moreover, it turned out that scanners with a high overall hit rate still have weaknesses with individual operating systems. Accordingly, in the table the scan results are broken down by operating system.

1. Beyond
2. Foundstone FS1000
3. ISS Internet Scanner
4. GFI LANguard
5. Nessus
6. Rapid7 NeXpose
7. Qualys Qualys Guard
8. Saintcorporation Saintbox
9. Safety-Lab Shadow Security Scanner

Although the scanner hit rates for finding vulnerabilities are important if they are to be used in the corporate network, the choice of product should not depend solely on this consideration. To allow continuous monitoring of a network, it should be possible to install the scanner permanently. All except two of the products enable one to do this, but two of them require additional software from the manufacturer (see “Distributed scanners” entry in the table). The possibility of carrying out time-controlled scans, offered by all the products, is also important for regular monitoring. If one wants to retain an overview in a large network, one needs a tool that can be easily incorporated into the business processes. This requires that it should be possible to export all the problems into existing solutions as tickets. Some of the products offered this, while all of them include their own ticket administration functionality. Another essential feature is remote management over a central console that is capable of controlling all the scanners in the network. It is not necessary, but it is helpful to assign the systems examined to groups and assess and report the results on the basis of membership of these groups It goes without saying that even after one or more scans have been carried out and all the problems identified have been rectified, a network cannot necessarily be regarded as secure and the possibility of attacks cannot be excluded. For no scanner will find every vulnerability, while not all the vulnerabilities that exist are known, and finally one should never underestimate the creativity of a human attacker. Nevertheless, it is worth using these programs, as the rapid detection and rectification of vulnerabilities will significantly enhance security in the network. And with such tools one can definitely make life more difficult for would-be attackers.

English version: all.pdf

is a Security Consultant at HiSolutions AG in Berlin, is a qualified Certified Information Systems Security Professional (CISSP) and has worked for several years as a penetration tester.