CA2767504A1 - A method of constructing a mixture model - Google Patents

Abstract

A method of constructing a general mixture model (100) of a dataset includes partitioning the dataset into at least two subsets (104) according to predefined criteria (108), generating a subset mixture model for each of the at least two subsets (110), and then combining the mixture models from each subset to generate a general mixture model (120).

Description

248634-3, A METHOD OF CONSTRUCTING A MIXTURE MODEL
BACKGROUND OF THE INVENTION

Data mining is a technology used to extract information and value from data.
Data mining algorithms are used in many applications such as predicting shoppers' spending habits for targeted marketing, detecting credit card fraudulent transactions, predicting a customer's navigation path through a website, failure detection in machines, etc. Data mining uses a broad range of algorithms that have been developed over many years by the Artificial Intelligence (Al) and statistical modeling communities. There are many different classes of algorithms but they all share some common features such as (a) a model that represents (either implicitly or explicitly) knowledge of the data domain, (b) a model building or learning phase that uses training data to construct a model, and (3) an inference facility that takes new data and applies a model to the data to make predictions.
A known example is a linear regression model where a first variable is predicted from a second variable by weighting the value of the second variable and summing the weighted value with a constant value. The weight and constant values are parameters of the model.
Mixture models are commonly used models for data mining applications within the academic research community as describe by G McLachlan and D Peel in Finite Mixture Models, John Wiley & Sons, (2000). There are variations on the class of mixture model such a Mixtures of Experts and Hierarchical Mixtures of Experts. There are also well documented algorithms for building mixture models. One example is Expectation Maximization (EM). Such mixture models are generally constructed by identifying clusters or components in the data and fitting appropriate mathematical functions to each of the clusters.

BRIEF DESCRIPTION OF THE INVENTION

In one aspect, a method of generating a general mixture model of a dataset stored in a non-transitory medium comprises the steps of providing subset criteria for defining subsets of the dataset, partitioning in a processor the dataset into at least two subsets based on the subset criteria, generating a subset mixture model for each of the at least two subsets, and combining the subset mixture model for each of the at least two subsets into a general mixture model.

BRIEF DESCRIPTION OF THE DRAWINGS
In the drawings:

FIG. 1 is a flow chart depicting a method of generating a general mixture model according to one embodiment of the present invention.

FIG. 2 is a flow chart depicting a method of filtering components from subset mixture models as part of the method depicted in FIG. 1.

FIG. 3 is a chart depicting an example of filtering of a dataset according to the method of generating a general mixture model of FIG. 1.

FIG. 4 is a chart depicting a subset mixture model of a first subset.
FIG. 5 is a chart depicting a subset mixture model of a second subset.

FIG. 6 is a chart depicting a general mixture model of constructed by the method disclosed in FIG. 1.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the technology described herein. It will be evident to one skilled in the art, however, that the exemplary embodiments may be practiced without these specific details. In other instances, structures and device are shown in diagram form in order to facilitate description of the exemplary embodiments.

The exemplary embodiments are described below with reference to the drawings.
These drawings illustrate certain details of specific embodiments that implement the module, method, and computer program product described herein. However, the drawings should not be construed as imposing any limitations that may be present in the drawings. The method and computer program product may be provided on any machine-readable media for accomplishing their operations. The embodiments may be implemented using an existing computer processor, or by a special purpose computer processor incorporated for this or another purpose, or by a hardwired system.

As noted above, embodiments described herein include a computer program product comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media, which can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of machine-executable instructions or data structures and that can be accessed by a general purpose or special purpose computer or other machine with a processor. When information is transferred or provided over a network or another communication connection (either hardwired, wireless, or a combination of hardwired or wireless) to a machine, the machine properly views the connection as a machine-readable medium. Thus, any such a connection is properly termed a machine-readable medium. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions comprise, for example, instructions and data, which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

Embodiments will be described in the general context of method steps that may be implemented in one embodiment by a program product including machine-executable instructions, such as program code, for example, in the form of program modules executed by machines in networked environments. Generally, program modules include routines, programs, objects, components, data structures, etc. that have the technical effect of performing particular tasks or implement particular abstract data types.
Machine-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the method disclosed herein.
The particular sequence of such executable instructions or associated data structures represent examples of corresponding acts for implementing the functions described in such steps.

Embodiments may be practiced in a networked environment using logical connections to one or more remote computers having processors. Logical connections may include a local area network (LAN) and a wide area network (WAN) that are presented here by way of example and not limitation. Such networking environments are commonplace in office-wide or enterprise-wide computer networks, intranets and the internet and may use a wide variety of different communication protocols. Those skilled in the art will appreciate that such network computing environments will typically encompass many types of computer system configuration, including personal computers, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like.

Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination of hardwired or wireless links) through a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

An exemplary system for implementing the overall or portions of the exemplary embodiments might include a general purpose computing device in the form of a computer, including a processing unit, a system memory, and a system bus, that couples various system components including the system memory to the processing unit.
The system memory may include read only memory (ROM) and random access memory (RAM). The computer may also include a magnetic hard disk drive for reading from and writing to a magnetic hard disk, a magnetic disk drive for reading from or writing to a removable magnetic disk, and an optical disk drive for reading from or writing to a removable optical disk such as a CD-ROM or other optical media. The drives and their associated machine-readable media provide nonvolatile storage of machine-executable instructions, data structures, program modules and other data for the computer.

Technical effects of the method disclosed in the embodiments include more efficiently providing accurate models for mining complex data sets for predictive patterns. The method introduces a high degree of flexibility for exploring data from different perspectives using essentially a single algorithm that is tasked to solve different problems. Consequently, the technical effect includes more efficient data exploration, anomaly detection, regression for predicting values and replacing missing data, and segmentation of data. Examples of how such data can be efficiently explored using the disclosed method include targeted marketing based on customers' buying habits, reducing credit risk by identifying risky credit applicants, and predictive maintenance from understanding an aircraft's state of health.

The present invention is related to generating a general mixture model of a dataset. More particularly, the dataset is partitioned into two or more subsets, a subset mixture model is generated for each subset, and then the subset mixture models are combined to generate the general mixture model of the dataset.

Referring now to FIG. 1, the method of generating a general mixture model 100 is disclosed. First a dataset contained in a database 102 along with subset criteria 108 are provided for generating subsets with a subset identification 104. The database with the constituent dataset can be stored in an electronic memory. The dataset can contain multiple dimensions or parameters with each dimension having one or more values associated with it. The values can be either discrete values or continuous values. For example, a dataset can comprise a dimension of gas turbine engine with discrete values of CFM56, CF6, CF34, GE90, and GEnx. The discrete values represent various models of gas turbine engines manufactured and sold by General Electric Corporation. The dataset can further comprise another dimension titled air frame with discrete values of B737-700, B737700ER, B747-8, B777-200LR, B777-300ER, and B787, representing various airframes on which the gas turbine engines of the gas turbine engine dimension of the dataset can be mounted. Continuing with this example, the dataset may further comprise a dimension titled thrust with continuous values, such as values in the range of 18,000 pounds-force to 115,000 pounds force (80 kN - 512 kN).

The subset criteria 108 can be one or more values of one or more dimensions of the dataset that can be used to filter the dataset. The subset criteria can be stored in a relational database or designated by any other known method. Generally, the subset criteria 108 is formulated by the user of the dataset, based on what the user wants to learn from the dataset. The subset criteria 108 can contain any number of individual criteria for filtering and partitioning the data in the dataset. Continuing with the example above, subset criteria 108 may comprise three different elements such as GE90 engines mounted on B747-8, GEnx engine mounted on a B777-300ER, and a GEnx mounted on B787.
Although this is an example of a two dimensional subset criteria with three elements, the subset criteria may include any number of dimensions up to the number of dimensions in the dataset and may contain any number of elements.

Generating the subsets and subset identification 104 comprises filtering through the dataset and identifying each element within each of the subsets. The number of subsets is equivalent to the number of elements in the selection criteria. The filtering process may be accomplished by a computer software element running on a processor with access to the electronic memory containing the database 102. After or contemporaneous with the filtering, each of the subsets is assigned a subset identifier to distinguish the subset and its constituent elements from each of the other subsets and their constituent elements. The subset identifier can be a text string or any other known method of identifying the subsets generated at 104.

It is next assessed if there is at least one subset at 106. If there is not at least one subset, then the method 100 returns to 108 to accept new subset criteria that produce at least one subset. If there is at least one subset, then the method 100 generates a mixture model for each of the subsets at 110. The generation of mixture models is also commonly referred to as training in the field of data mining. The mixture model for each of the subsets can be generated by any known method and as any known type of mixture model, a non-limiting example being a Gaussian Mixture Model trained using expectation maximization (EM). The process of generating a mixture model for each subset results in a mathematical functional that represents the subset density. In the example of modeling continuous random vectors, the mathematical functional representation of each of the subsets is a scaled summation of probability density functions (pdf). Each of the pdf corresponds to a component or cluster of data elements within the subset for which the mixture model is being generated. In other words, the method of generating a mixture model of each of the subsets 110 is conducted by a software element running on a processor, where the software element considers all data elements within the subset, clusters the data elements into one or more components, fits a pdf to each of components, and ascribes a scaling factor to each of the components to generate a mathematical functional representation of the data. A non-limiting example of a mixture model is a Gaussian or Normal distribution mixture model of the form:

K
p(X)-I'TkN(XPk'Ykk=]
where p(X) is a mathematical functional representation of the subset, Xis a multidimensional vector representation of the variables, k is an index referring to each of the components in the subset, K is the total number of components in the subset, lTk is a scalar scaling factor corresponding to cluster k with the sum of all irk for all K
clusters equaling 1, N(X uk , yk) is a normal probability density function of vector X for a component mean Pk and covariance Ek.

If the vector X is of a single dimension, then Ek is the variance of X and if X has two or greater dimensions, then Ek is a covariance matrix of X.

After the mixture models are generated for each subset at 110 it is determined if there are at least two subsets at 112. If there are not at least two subsets, then the single subset mixture model generated at 110 is the general mixture model. If, however, it is determined that there are at least two subsets at 112, then it is next determined if filtering of the model components is desired at 116. If filtering is desired at 116, then one or more components are removed from the model at 118. The filtering method of 118 is described in greater detail in conjunction with FIG. 2. Once the filtering is done at 118 or if filtering was not desired at 116, then the method 100 proceeds to 120 where the subset models are combined.

Combining subset models at 120 can comprise concatenating the mixture models generated for each of the subsets to generate a combined model. Alternatively, the combining subset models can comprise independently scaling each of the mixture models of the individual subsets prior to concatenating each of the mixture models to generate a combined model.

At 122, it is determined if simplification of the model is desired. If simplification is not desired at 122, then the combined subset model is the general model at 124. If simplification is desired at 122, then a simplification of the combined model is performed at 126 and the simplified combined model is considered the general model at 128. The simplification 126 can comprise combining one or more clusters from two or more different subsets. The simplification 126 can further comprise removing one or more components from the combined mixture models of the subsets.

Referring now to FIG. 2 the method of filtering the components of the individual subset mixture models at 118, prior to combining the subset mixture models, is described. First, a completed list for tabulating each component and associated distances to other components is cleared at 140. Next, all of the components from all of the subsets are received by a processor and associated electronic memory at 142. A component from all of the components is selected at 144 and the distance of the selected component to all other components in other subsets is determined at 146. In other words, the selected component is compared to all other components with a subset identifier that is different from the subset identifier of the selected component. The distance can be computed by any known method including, but not limited to, the Kullback Leibler divergence. The component and the associated distances to all the other components of other subsets are tabulated and appended to the completed list at 148. In other words, the completed list contains the distance from the component to all components of the other subsets. At 150, it is determined if the selected component is the last component. If it is not, then the method 118 returns to 144 to select the next component. If, however, at 150 it is determined that the selected component is the last component, then the completed list is updated for all of the components of all of the subsets and the method proceeds to 152, where the completed list is sorted in descending order of the distances calculated at 146.
At 154, the top component on the completed list, or the component that has the greatest distance to all the other components of all the other subsets, is removed or filtered out.
At 156, it is determined if filtering criteria have been satisfied. The filtering criteria, for example, can be a predetermined total number of components to be filtered.
Alternatively, the filtering criteria can be the filtering of a predetermined percentage of the total number of components. If the filtering criteria are met at 156, then the final component set is identified at 160. If, however, the filtering criteria are not met at 156, then it is determined at 158 if iterative filtering is desired. The desire for iterative filtering can be set by the user of the method 118. If iterative filtering is not desired at 158, then the method returns to 154 to remove from the remaining components, the component with the greatest distance to all other components from other subsets. At 158, if it is determined that iterative filtering is desired, then the method 118 returns to 140.
Iterative filtering means that the method 118 recalculates the distances for each component to every other component and generates a new completed list by executing 140 through 152 every time a component is removed from the mixture model. The distances between components can change and, therefore, the relative order of the components on the completed list can change as components are removed from the mixture model. Therefore, by executing iterative filtering, one can ensure with greater confidence that the component being removed is the component with the greatest distance to the components from every other subset. However, in some cases, one may not want to execute iterative filtering, because iterative filtering is more computationally intensive and, therefore, more time consuming. In other words, when executing the filtering method 118 disclosed herein, one may assess the trade-off between filtering performance and time required to filter to determine if iterative filtering is desired at 158.

FIGS. 3-6 depict an example of executing the foregoing method 100 of generating a general mixture model. In FIG. 3, data 180 and 190 from a dataset is plotted against a variable xi. The data is further partitioned into a first subset 180 depicted as open circles on the graph and a second subset 190 depicted as closed triangles on the graph according to the procedures described in conjunction with 104 of method 100. Although the method 100 can be applied to multivariate analysis with many subsets, a single variable data dependency with only two subsets is depicted in this example for simplicity in visualizing the method 100.

FIGS. 4 and 5 depict the generation of a mixture model as at step 110 for the first subset 180 and second subset 190, respectively. In the case of the first subset 180, three components are identified and each is fit to a scaled Gaussian distribution G1, G2, and G3 with means I, t2, and 13, respectively. In the case of the second subset 190, two components are identified and each is fit to a scaled Gaussian distribution G4 and G5 with means .I4 and s, respectively. Thus, the mixture model of the first subset 180 is represented by the envelope of the scaled fitting function of the constituent components G1, G2, and G3. Similarly, the mixture model of the second subset 190 is represented by the envelope of the scaled fitting function of the constituent components G4 and G5. In FIG. 6, the combined constituent scaled fitting functions of the general mixture model are depicted, as at step 120 of the method 100, after filtering. In this example, it can be seen that in the filtering step 118, it was found that the component with fitting function G3 was at a distance from the components of the other subset G4 and G5 that exceeded some predetermined value (not shown), and therefore the component G3 was removed from the general mixture model of FIG. 6.

This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to make and use the invention. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.

Claims (10)

1. A method of generating a general mixture model (100) of a dataset stored in a non-transitory medium (102) comprising the steps of:
providing subset criteria (108) for defining subsets of the dataset;
partitioning in a processor the dataset into at least two subsets based on the subset criteria (108);
generating a subset mixture model (110) for each of the at least two subsets;
and combining the subset mixture model for each of the at least two subsets into the general mixture model (120).

2. The method of claim 1 wherein the subset criteria include one of being defined in a relational database and filtering the dataset by at least one dimension.

3. The method of claims 1 or 2 wherein the generating step includes at least one of identifying at least one component of a subset (104), fitting a function to at least one component of a subset, scaling fitting functions by a scaling factor, and summing scaled fitting functions.

4. The method of claim 3 wherein the function is a probability density function.

5. The method of claim 4 wherein the probability density function is a normal distribution function.

6. The method of claim 3 wherein the scaling factor is a scalar value.

7. The method of claim 4 wherein the sum of all of the scaling factors corresponding to each of the fitting functions of a subset is 1.

8. The method of claims 1 or 2 wherein the combining step (120) comprises concatenating the subset mixture models for each of the at least one subset, independently scaling the subset mixture models for each of the at least one subset and then concatenating the scaled subset mixture models, and removing one or more component functions prior to combining the subset mixture models (150).

9. The method of claim 8 wherein removing of one or more component functions (150) prior to combining the subset mixture models comprises selecting a component and determining the distance between the selected component and all of the components from subsets other than the subset corresponding to the selected component (144).

10. The method of claim 9 wherein the removing of one or more component functions prior to combining the subset mixture models (150) further comprises removing the component with the greatest distance.