DE102005049561A1 - Automatic recognition of anomalies in wide and local area networks involves filtering out, assessing anomalies using thresholds adapted depending on detection accuracy of real attacks and/or frequency of false alarms - Google Patents

Abstract

The method involves automatic detection of network measurement data in the context of a defined time regime suitable for representing network activities so as to reflect the real world in the network, compressing the data, automatically detecting cluster structures/categories and the related plans during a continuous training and learning process, automatically monitoring the validity of the detected plans and cluster structures/categories and filtering out and assessing anomalies using thresholds adapted depending on detection accuracy of real attacks and/or the frequency of false alarms.

Description

The The present invention relates generally to the field of Network technology and relates to the detection of anomalies in wide area networks and local networks. This includes, in particular, the detection of network attacks by malicious user / attacker understood, for example, try one or more servers by a denial of service attack / Denial of Service Attac / DoS attack to paralyze.

In www.heise.de/ix/ article / 2205/04/107 Article Christoph Puppe, Jörn Maier is executed that the first denial of service attacks that became known are on the websites of Yahoo, CCN eBay Amazon and other service providers were made. On 09.02. In 2000, "Mafiaboy" began its attacks and cared for a week long for massive disruptions. On the 10th of March 200 collapsed investor confidence in the Internet shares and on the stock exchange $ 2.7 trillion was destroyed.

The Consequences of such an attack are next to an immense material Loss also always a loss of confidence of customers and investors into the attacked company. Security issues of computer systems are therefore, in particular from the o.a. Reasons, in recent times always become more important. This is due to the fact that on the one hand, the number the attacks on networks, and especially on servers, is constantly increasing, and that, on the other hand, more and more computers are using both wide area networks (WAN) as well as local area networks (LAN).

primitive DoS attacks encumber the services of a server, for example HTTP, with a larger number of requests as this can handle. As a result, the affected Server stop processing or answer the regular requests so slowly that the answer is canceled.

Yet more effective in its effect is an attack based on exploit a program error to trigger an error function in the server software, which then results in the crash of the server. Such an attack However, this can be the basis by improving the software programs or by avoiding program errors.

He follows the attack is coordinated by a larger number of other systems, one speaks of a Distributet Denial of Service attack, DDoS attack. The strategy of these attacks is to work with backdoor programs spread alone on other machines in the network, the attacker more hosts to run his attacks too put.

A special form of attack is the distributed reflected denial of service attack DRDoS attack. Addressed at this type of attack the attacker does not send his data packets directly to the victim, but instead at regular working internet services. He carries however as sender address the victim's address (IP spoofing). The answers to this Inquire then for the victim is the actual DoS attack. The origin of the attack is for the victim by this procedure practically no longer be determined.

It In principle, this is still the susceptibility before DoS attacks one of the most serious vulnerability of the Internet is.

The Internet vulnerabilities that allow DoS attacks are inherent in the system and are in particular justified in that it is not provided on the Internet, each communication partner clearly to identify a centralized management of data transfers too offer or each participant a share of the existing bandwidth to guarantee.

For the recognition There are two different solutions to security problems in a computer network. Of the first solution is called a knowledge-based process. The knowledge-based Process approach is based on the assumption that the system all potential Attack types are known. For every attack has a defined signature. A system for monitoring Traffic searches for these signatures and identifies them she.

In US Patent 5,278,901 will be such a signature based System described.

The advantage of this method is that very rarely a false alarm is triggered, ie the error rate is relatively low. The main disadvantage of this approach is seen in the fact that systemically only the attacks can be detected, the signatures are already stored. Currently, however, there are so many attacks that the number of different signatures is increasing very fast. Many signatures are also structured in such a way that they are difficult to represent using an algorithm. Two examples of products based on the knowledge-based approach are NetRanger from Cisco Systems (see www.alliancedaracom.com/manifacturers/cisco-systems/security_vpn/ids.asp) and RealSecure from Internet Security Systems Inc. (see www.rio co.co.uk/products/intrusion_detektion/Datasheet_RealSecure.pdf)

Of the second approach is called a behavioral approach. He goes from the Assume that the "behavior" of a system changes when an attack on the system takes place. In the behavior-based Approach is for the system defines a normal profile. Starting from the normal profile is searched for deviations.

Two examples for new, just announced Products based on the behavioral approach are Cisco Guard XT 5650 (see www.cisco.com/en/US/products/ps5894) or Cisco Traffic Anomaly Detector XT 5600 (see www.cisco.com/en/US/products/ps5892).

For the realization this behavior-based approach can various methods, e.g. Statistical methods, rule-based systems and neural networks are used. A kind of artificial neural networks, for example, make self-organizing maps (Self-Organizing Maps / SOMs or Self-Organizing Feature Maps), which according to Teuvo Kohonen are also called Kohonen nets.

through the above method can different monitoring goals, such as the users of the system, the performance parameters of the computer network, the CPU clocks, etc. are defined. The main advantage compared to the knowledge-based approach is to look at the behavioral-based approach even unknown types of attack can be detected. It is omitted in this approach thus forcing a database permanently with the well-known signatures to update.

Of the Disadvantage of this approach This is because deviations occur even without an attack can, for example, through changes the user activities, new software, new machines and new users are triggered.

The Difficulty in this approach exists therefore, normal user behavior of a malicious To distinguish attack.

Of the Behavior-based approach can therefore be due to the applied solution principle lead to a very high number of false alarms, which is the procedure for the user make it unattractive.

• a) auf einem Trainingsmodus, dem eine Tabelle charakteristischer, den Prozess darstellender Muster zugrunde liegt, die das normale Verhalten eines Modellprozesses in dem Rechner definieren, durch Ausführen der Schritte – Erstellen einer ersten Ereignissequenz durch Filtern eines ersten durch den Modellprozess generierten Ereignisdatenstroms – Verwenden des Teiresias-Algorithmus zum Extrahieren von Ereignissequenzmustern, aus der ersten Ereignissequenz, wobei die Muster den Modellprozess darstellen, – Speichern der den Prozess darstellenden Muster und
• b) auf einem Betriebsmodus, in dem aus einem realen Prozess charakteristische Muster durch Ausführen der folgenden Schritte extrahiert werden; – Erstellen einer zweiten Ereignissequenz durch Filtern eines durch den realen Prozess generierten Ereignisdatenstromes, – Vergleichen der ersten Ereignissequenz mit den gespeicherten, den Prozess darstellenden Mustern und – Anzeigen des Ergebnisses des Vergleichsschrittes.

Another solution based on the behavioral approach is out DE 698 17 176 T2 known. This solution is a method and an apparatus for intrusion detection, primarily in a computer system based on event patterns and in particular the detection of deviations from a "normal" process behavior and thus the detection of attacks against this process

• a) on a training mode based on a table of characteristic patterns representing the process, defining the normal behavior of a model process in the computer by performing the steps of - creating a first event sequence by filtering a first event data stream generated by the model process - using Teiresias algorithm for extracting event sequence patterns, from the first event sequence, wherein the patterns represent the model process, - storing the pattern representing the process and
• b) an operation mode in which patterns characteristic of a real process are extracted by performing the following steps; - creating a second event sequence by filtering an event data stream generated by the real process, - comparing the first event sequence with the stored process representative patterns, and - displaying the result of the comparing step.

In the Teiresias algorithm, see also US 6,108,666A and DE 698 17 176T2 is a search algorithm developed by IBM in 1996 and originally intended to search genetic sequence data for recurring genetic patterns. In the future, this algorithm will be used as part of an anti-spam filter.

Farther Methods are known that are based on the automatic formation of hypotheses about intentions from observed behavior (plan recognition method). Under intentions are such goals / plans of an actor or a Group of actors understood by him or by them within the next Time to be striven for and reachable to him also in this period appear. Not the term of intention therefore includes longer-term ones Goals and goals over which Implementation of the actor has not thought (see www.dfki.uni-sb.de/vitra/papers/sport89/node9.html).

Essentially, the existing methods are based on data and measurements Computer systems - so-called hosts - are performed. These approaches are only of limited suitability for direct deployment in the network infrastructures themselves because data must be aggregated and analyzed at a higher aggregation level. This requires much more complex forms of recognition mechanisms, geared to the particular situation in the networks.

aim The present invention is based on the behavior-based approach Denial-of-service attacks in IP-based networks almost in Real-time detection, with different requirements an automatic detection of denial-of-service attacks in Are to be reconciled.

outgoing from the behavior-based approach the invention is based on the introduction of a continuous Learning process with the "normal Behavior of Network Resources to understand. This is achieved by a method in which two known per se Solution approaches for the first time in the context of a new independent solution together become.

First network metrics are collected at continuous intervals, which are suitable, continuously a symbolic representation of in the to be monitored Network to generate network activity. For the symbolic Presentation of network activities the acquired network data are collected by means of techniques of "unmonitored Learning "compressed.

Under Techniques of the unsupervised Learning here becomes machine learning without classes known in advance Understood. examples for such unsupervised machine learning is automatic grouping (clustering) or compressing data for dimensionality reduction. outgoing from those according to the principles of the thus understood "unsupervised machine learning " and compressed information about the network activity or Network status becomes these compressed network metrics symbolically represented by cluster structures.

Secondly, plan detection techniques known per se are applied to the network data represented in symbolic form, for example by SOMs. This is intended to recognize and model the rules underlying the learning behavior. On the basis of this approach, deviations from "normal network behavior can be detected almost in real time in the future. An example of collecting network metrics in clustered structures and plan databases is in 1 shown.

To the There are basically two different ways of collecting network measurement data Types of measurement.

• – Konnektivität (connectivity)
• – Einfache Verzögerung (one-way-delay) und Umlaufverlust (one-way-loss)
• – Netzlaufzeit (round-trip-delay)
• – Laufzeitvariation (delay variation)
• – Verlustmuster (loss patterns)
• – Umordnung der Pakete (packet reordering)
• – Massentransportkapazität (bulk transport capacity) und
• – Link-Bandbreitenkapazität (link bandwidth capacity)

In passive measurement, network measurement data is collected by sampling and processing data packets from user traffic. The active measurement is done by sampling and processing test packages, which are additionally injected into the network for measurement purposes. To determine a reduction in network performance, the following measured variables are determined, for example:

• - connectivity
• - One-way-delay and one-way-loss
• - Network runtime (round trip delay)
• - runtime variation (delay variation)
• - loss patterns
• - reordering of the packets (packet reordering)
• - Mass transport capacity (bulk transport capacity) and
• - Link bandwidth capacity

Network measurement data, obtained by the methods described above arranged in predefined categories, or with the help of "unmonitored machine learning "in Divided into groups / clusters. This is not the case to the assignment to predefined categories, but to "learned" categories, the Automatically due to different network activities as observation focal points have proven.

After that The recorded and assigned data are used as training material for recognizing plans used in a special type of traffic or log data. For example, this may be the average throughput of edge routers in a network segment or even the average response time of web servers to act on http requests in a hosting environment. The condition is that the elected streams Reproduce aspects of the real world and are therefore suitable, all related network activities represent, being enough Must have measurements to detect sufficiently stable cluster structures.

The period over which a cluster structure must remain stable depends on the type of application, ie whether performance degradation of routers based on clusters is to be observed in traffic-peak scenarios, or whether it depends on click and surf behavior of Internet users arrives.

For example becomes a self-organizing card after a negligible training time determined from the measurement results, a first cluster structure C1. The cluster structure remains over stable for a certain period of time and then begin to disintegrate or to move into another cluster structure C2, which in turn after a time decays.

Remarkable in this process is the following:
Unmonitored machine learning usually requires that the observed domain has some stability. This supports the further use of the identified clusters as input to scheduling algorithms based thereon, which search the identified clusters for recurring patterns. However, this happens on an abstract level within the existing cluster structures.

Becomes the observed world unstable, the entire training cycle begins again. In an IP-based data network, clustering may occur All of which makes it unstable, that in the real world a bigger change entering through the changing Measurement results is manifested.

in the example of a router or server load described above can such a change already be caused by a popular television program is broadcast and thereby larger user groups switch from Internet to TV at the same time. That from this process resulting result would be a sudden change the traffic or load patterns, depending on the geographical distribution the user or their individual preferences when using the internet have an impact on the respective cluster structure. On the other hand a stability to watch when big Coherent user groups Show behavior, for example, in typical Internet times, such as B. at 22.00 clock or in the late night, can be detected.

in the With regard to the architecture according to the invention Periods of stable cluster structures suggest that no major changes occurred are. In contrast, instability, as already described, can natural Causes or malicious Behavior, such as a distributed denial-of-service attack; DDoS attack. Deliberate flooding from network segments or bulk requests to destination servers trigger a short-term change of the cluster structure unless they coincide with a natural one stability change of the cluster structure coincide.

There the underlying real world by measurements in an IP based Network is imaged, follows typical patterns, for example, if entire user groups during a definable period of time to go to television, let yourself the transition Explain the cluster structure as plan-based behavior. Under A plan is understood here to be rule-based behavior, which caused by a group of autonomous agents whose behavior underlying well-defined goals. However, these goals will be not disclosed. Furthermore, it is assumed that the exact Definition of the agents is also unknown. Only the existence of these agents and the fact that the behavior of the agents as can be considered plan-based, is to be regarded as secured.

following will be an example of plan-based behavior of user groups based on the everyday life of Students described.

Consider, as generally in 2 Pictured, the everyday life of young students with a fondness for evening movie or club visits and nocturnal surfing on the Internet. The real world underlying this example is a large university city, such as Berlin. The real world underlying this scenario is captured by the already described traffic measurements in selected edge routers and observable http requests. The measurements take place, for example, at a distance of 100 milliseconds and can comprise about 200 values per measurement run.

To a time of intense internet surfing by a variety of students, which falls in a period of about 22:00 clock to 04:00 clock, go most of the students attend between 04:00 and 05:00 Rest to find some sleep before the next lecture at 09:00 starts.

If a significant number of students in the selected city follow this pattern of behavior, a change in the cluster structure can be observed after retraining the SOMs. However, since the clusters themselves do not yet provide information about the plans that underlie the changes, in the second shift the underlying plans are determined using analysis techniques. This is preferably done in two phases. In the first phase, the cluster structures are considered on a symbolic level and a long training phase is begun in order to recognize the plans hidden in the symbolic representations. In the second Step then monitors the validity of the identified plans and defines an anomaly as an unexpected and nonconforming transition in clustering, which can not be explained by the measures used to quantify distances between previous and present states or plans, based on the respective plan or cluster knowledge , First, preset thresholds are used to evaluate filtered-out anomalies, which are automatically adjusted depending on the detection accuracy of real attacks or the frequency of false alarms.

The supervision of validity the recognized plans and the definition of an anomaly as a transition in clustering takes the form of a compromise between accuracy and Speed. A key aspect for one working solution is to be observed, based on the fact that the selected tools, such. B. self-organizing maps, about an inherent one Have flexibility that the observed real world or the observed network components must be adjusted. At low granularity of network measurements or Cluster numbers may be the meaningfulness of the model used low. This leaves for example, with increasing DoS attacks in a network segment illustrate. The impact of a DoS attack on a network segment is initially weak and with rough models that are on a low Number of network measurements and a small number of detected ones Clusters are based, not representable. Within a short period of time, measured from the beginning of the DoS attack, deviations, which close to the DoS attack let, only with finely granular models recognize that on a substantial larger number network measurements and a larger number of detected clusters based. On the other hand, increases with the degree of accuracy selected Cluster or plan models the amount of needed Computing resources required to fulfill the real-time requirements are needed.

It is therefore a compromise between detection accuracy and computing time to find.

3 Graphically represents a compromise between detection accuracy and computation time based on a diagram.

The Develop a working model that is capable of DoS attacks in a commercial IP-based Recognizing networks requires the use of algorithms, such as SOM calculations and plan recognition algorithms, an accurate understanding of power requests to the network. From the bottom-up point of view, public accessible Test data from already referenced projects are used to check the speed of existing applications. at a top-down approach On the other hand, starting from a reference network, they are modeled for parts a communication network can serve, using the theoretical valid scalability considerations Requirements for computing power derived to a commercial application, to determine the required minimum power.

critical is that DoS attacks and other network problems with the invention can only be successfully recognized if a certain minimum accuracy is complied with.

Claims (2)

Method for the automatic detection of anomalies in wide area networks (WAN) and local area networks (LAN) and in particular from denial-of-service attacks based on behavioral-based Approach, marked through the steps - automatic Acquiring network measurement data within a given time regime, which are suitable for mapping network activities in such a way that they reflect the real world in the network, - compress the network metrics using techniques of unsupervised Learning (dimensionality reduction), - automatic detection of Cluster structures / categories from the continuously collected compressed Network metrics using methods of unsupervised learning during one permanent continuous training and learning process, - automatic determine the underlying cluster structures / categories Plans by Algorithms for plan recognition during a permanent one continuous training and learning process, - automatic monitoring of validity the recognized plans and cluster structures / categories and filtering out anomalies in the sense of an unexpected and non-conformist transition in at least one cluster structure / category that does not conform to the assign previously determined plan or cluster knowledge, - automatic evaluate the filtered out anomalies based on pre-set Thresholds depending on the recognition accuracy of real attacks or the frequency false alarms are adjusted automatically. A method according to claim 1, characterized in that the network measurement data - connectivity, - simple delay (one-way-delay) and loss of circulation (one-way-loss), - network runtime (round-trip-delay), - delay variation, - loss patterns, - packet reordering, - mass transport capacity and link bandwidth capacity for the continuous continuous training process.