WO2004097677A1 - Automatic data consolidation - Google Patents

Automatic data consolidation Download PDF

Info

Publication number
WO2004097677A1
WO2004097677A1 PCT/IB2004/000669 IB2004000669W WO2004097677A1 WO 2004097677 A1 WO2004097677 A1 WO 2004097677A1 IB 2004000669 W IB2004000669 W IB 2004000669W WO 2004097677 A1 WO2004097677 A1 WO 2004097677A1
Authority
WO
WIPO (PCT)
Prior art keywords
distributed databases
request
data
access information
information
Prior art date
Application number
PCT/IB2004/000669
Other languages
French (fr)
Inventor
Stefan G. Hild
René A. PAWLITZEK
Walid Rjaibi
Markus Stolze
Original Assignee
International Business Machines Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corporation filed Critical International Business Machines Corporation
Priority to EP04717176A priority Critical patent/EP1618495A1/en
Publication of WO2004097677A1 publication Critical patent/WO2004097677A1/en
Priority to US11/227,875 priority patent/US7401101B2/en
Priority to US12/121,002 priority patent/US7930312B2/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/254Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99944Object-oriented database structure
    • Y10S707/99945Object-oriented database structure processing
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99948Application of database or data structure, e.g. distributed, multimedia, or image
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99951File or database maintenance
    • Y10S707/99952Coherency, e.g. same view to multiple users
    • Y10S707/99953Recoverability

Definitions

  • the present invention is related to a method, apparatus, and system for consolidating data from distributed databases into a central database.
  • the data is fragmented within the organization or company, with different database systems being utilized in different departments, all of which often maintain essentially the same data in multiple formats using different database table designs.
  • the international publication WO 99/52047 A 1 relates to a method and system for migrating data from one or more ASCII files and/or from one or more relational databases to one or more relational database tables without the need to write code.
  • This allows the user to define mapping templates and conditionals to assist in translating and transforming data values.
  • the method also enforces referential integrity, data dependencies, order of operations, and uniqueness constraints using a predefined set of migration rules templates that are based on the principles of relational design.
  • the method uses these mapping and migration rules templates to generate instructions for updating or populating relational database destination tables.
  • the instructions control the data transfer, data translation, data transformation, data validation, foreign key insertion, and the addition of required codes and flags in the destination tables.
  • a migration engine of the system includes a data map architect and an update processor which spawns the templates and migrates the data dynamically, utilizing the data definitions for the destination tables.
  • This prior art has the drawback that it is limited to a particular type of database systems, in which Oracle ® Application tables are implemented. This requires users to manually define rules for the migration or at least interactively, i.e. by user interaction. (Oracle is a trademark of Oracle Corporation). This prior art does not show provisions which allow to discover databases across an organization or augment a consolidation process with actual access patterns.
  • sensors also referred to as interceptors or request detectors
  • An inference engine that analyses the access patterns and the data formats contained in the individual databases.
  • a central database also referred to as central data warehouse or centralized repository, aggregates the data from the individual data sources with the view to replace those eventually.
  • the sensors or request detector are attached to infrastructure elements, typically software drivers that manage the database access of users and/or applications, and record the requests that are being submitted by users and applications.
  • a typical example of such sensor is a modified ODBC (JDBC) driver.
  • JDBC drivers are Java code that are frequently used today to access databases from Java programs.
  • JDBC driver By modifying the JDBC driver the sensor logic can record all data requests that are initiated from programs or users to databases, and also which database is being addressed.
  • Other examples can be derived by modifying the database itself. All data access is logged and transmitted to the inference engine, either in real time or in batch mode.
  • the inference engine analysis the data access recorded by the sensors or request detectors to identify i) the database engines used which are distributed databases within a network or networks and ii) the data schemes employed; for example, the inference engine learns what the format of the data tables is in the various database engines, what primary keys and foreign keys are employed, and what type of data is contained within those databases tables. Further, the inference engine can perform a correlation iii) between different databases; for example, the inference engine should correlate columns from different databases even though they may not be named the same.
  • the engine Based on the results of the inference the engine generates a new data schema, generates an instance of that schema on the central data warehouse, i.e. the central database. Over time the inference engine then copies existing data from the individual distributed databases that have been discovered into this new central database. When completed, the inference engine may issue an order to redirect calls to the individual databases to the central data warehouse. This can be done by advising the request detectors to intercept the individual data access calls and redirecting them to the central database.
  • the central data warehouse or cental database is a database engine, e.g. an IBM DB/2. For increased availability a cluster may be utilized.
  • a method for consolidating data automatically from distributed databases into a central database comprises the steps of receiving access information comprising request information to the distributed databases, analyzing the received access information, and aggregating into the central database the data content of the distributed databases in dependence on the analyzed access information. This allows a simple automatic migration of redundant data distributed over several databases.
  • the method can further comprise the steps of filtering the request information to the respective distributed databases from data traffic and forwarding the filtered request information within the access information to an inference engine. All the collected request information can be analyzed in one place, i.e. the information from the various databases can be compared and possible consolidations can be investigated.
  • the analyzing step can comprise the usage of log-file information. This is simple to perform and does not require any change in the infrastructure, but may not yield access data at the same level of detail as a sensor or request detector would detect.
  • a request detector for supporting data consolidation from distributed databases into a central database.
  • the request detector can comprise a detecting means for detecting request information to the distributed databases, a transforming means that derives access information from the detected request information, and a providing means that sends the access information to an inference engine.
  • the request detector can be provided at each of the distributed databases to be consolidated, preferably in form of a modified ODBC (JDBC) driver. The request detector may even be integrated into each of the distributed databases to be consolidated.
  • JDBC modified ODBC
  • the request detector can comprise redirecting means for redirecting a request to an individual database to the central database. This has the advantage that the request is forwarded directly to the consolidated central database and the user may get more information than it is provided by the individual database.
  • an inference engine for controlling data consolidation from distributed databases into a central database.
  • the inference engine can comprise means for analyzing of access information that is received from distributed databases and comprises request information to the respective distributed databases.
  • the inference engine can comprise a correlation means for correlating columns and/or rows between different distributed databases, but also fields, records, and/or data structures can be correlated. This leads to a new schema that then can be used by the consolidated central database.
  • the inference engine allows a simple migration of the data. Equivalent information or data is brought together and stored on one place. This helps to avoid doubles in distributed systems.
  • a system for consolidating data from distributed databases into a central database comprises a request detector at each of the distributed databases to be consolidated for providing access information comprising request information to the distributed databases, an inference engine for analyzing the received access information, and a central database into which the data content of the distributed databases is aggregated in dependence on the analyzed access information.
  • FIG. 1 shows a schematic illustration of a distributed database structure.
  • FIG. 2 shows a schematic illustration of database structure according to the present invention.
  • FIG. 3 a shows a schematic illustration of a request and access information flow.
  • FIG. 3 b shows a schematic illustration of a redirect flow.
  • FIG. 4 shows schematic illustration of a consolidation of two databases into a central database.
  • Fig. 1 shows a schematic illustration of a distributed database structure 1 with distributed databases 11, 12, 13, 14, 15.
  • the databases also labeled with DBx, are connected and accessible via a network 6.
  • the user 10, i.e. a user's computer sends a first request i to the first database 11, a second request r 2 to the second database 12, and third request r to the third database 14.
  • the user 10 receives the respective responses from the distributed databases 11, 12, 14 which then can be evaluated.
  • Fig. 2 shows a schematic illustration of a modified database structure 2 according to the present invention.
  • the database structure further comprises request detectors 21, 22, 23, 24, 25 attached to each of the distributed databases 11, 12, 13, 14, 15, respectively.
  • a central data unit 30 is provided and connected to the network 6 .
  • This unit comprises an inference engine 32 and a central database 34 which are connected to each other and to the network 6, and thus to the distributed databases 11, 12, 13, 14, 15.
  • the user 10 sends the same requests ri, r 2 , r 3 to the respective distributed databases 11, 12, 14 as described with reference to Fig. 1.
  • Each of the requests r,, r 2 , r 3 to the respective distributed databases 11, 12, 14 is now detected by the respective request detectors 21, 22, 24.
  • the detectors 21, 22, 23, 24, 25 comprise of detecting means (not shown) for detecting such requests ri, r 2 , r 3 to the distributed databases 11, 12, 14.
  • access information ai, a 2 , a 3 is derived indicating, e.g. a database address, inquiry details, etc..
  • the access information ai, a 2 , a 3 is then sent to the inference engine 32 as indicated in Fig. 2.
  • the distributed databases 11, 12, 13, 14, 15 can be removed, since the content is then consolidated and stored in the cental database 34. This allows more redundancy in storage and better archiving possibilities.
  • Fig. 3a shows a schematic illustration of a flow of a request r and access information a for current distributed database structures in order to establish the central database 34.
  • the request r is sent by the user 10 to one distributed database lx.
  • the request detector 2x attached to the distributed database lx receives this request r, transforms it to access information a and sends this access information a for purposes of analysis to the inference engine 32.
  • Fig. 3b shows a schematic illustration of a redirect flow. This is performed when the data consolidation has been performed successfully. Then, the request detector 2x is informed by the inference engine 32 about redirecting and any request r sent by the user 10 is redirected as a redirect request RR by the request detector 2x to the central database 34. The request inquiry is then answered by the consolidated central database 34.
  • Fig. 4 shows a schematic illustration of a consolidation of two distributed databases 11, 12 into a central database 34.
  • the figure shows on the left hand side the content of the first database 11 and the content of the second database 12.
  • the first database 11 stores "Names", like Joe, Bob and Alice, and their respective "Age", 28, 40, and 18.
  • the second database 12 stores also "Names", here Joe and Bob, but is stores further the "Place of birth” instead of the Ages.
  • a data consolidation of the data is therefore desired saving infrastructure and maintenance costs.
  • the inference engine 32 analyzes the available data, compares it, and performs a correlation.
  • This correlation correlates columns and/or rows between different distributed databases to find similarities or matches.
  • the data fields "Name” appear in both distributed databases 11, 12.
  • the distributed databases 11, 12 are candidates to merge their content into the central database 34.
  • the inference engine 32 generates the new data schema and provides it to the central database 34.
  • the central database 34 is filled with the content of the distributed databases 11, 12 to have the fields "Name", "Age”, and "Place of birth” with the records for 'Joe, 28, Bern'; 'Bob 40, Zurich'; and 'Alice, 18'.
  • Computer program means or computer program in the present context mean any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following a) conversion to another language, code or notation; b) reproduction in a different material form.

Abstract

The present invention discloses a method, request detector, inference engine, and system for consolidating data from distributed databases into a central database. The method comprises the steps of receiving access information comprising request information to the distributed databases, analyzing the received access information, and aggregating into the central database the data content of the distributed databases in dependence on the analyzed access information.

Description

AUTOMATIC DATA CONSOLIDATION
TECHNICAL FIELD
The present invention is related to a method, apparatus, and system for consolidating data from distributed databases into a central database.
BACKGROUND OF THE INVENTION
Today many organizations concentrate their IT (information technology) spending on methods and technologies that help them reduce cost by increasing the efficiency and effectiveness of their IT infrastructure. A key pain point faced by many organization and companies is that, in the beginning of the Internet boom, also called the "dot-com" bubble, many organizations had to embrace the Internet overnight, and in doing so established infrastructure elements in an ad-hoc fashion that were not well-designed for scalability or growth. Today, these organizations and companies are faced with infrastructures that are loosely assembled, are costly to maintain, and are difficult and expensive to grow with the business needs. This is evident in both the business processes as well as in the way these organizations manage data.
In many instances, the data is fragmented within the organization or company, with different database systems being utilized in different departments, all of which often maintain essentially the same data in multiple formats using different database table designs. Here, it would be hugely beneficial to maintain all data in a (logically) single place using a standardized schema. Having such centralized data warehouse or database engine would enable quick data analysis for improved customer relationship management, simplify development of new products, and reduce maintenance cost for the IT infrastructure itself while improving the reliability and availability of the entire system.
The international publication WO 99/52047 A 1 relates to a method and system for migrating data from one or more ASCII files and/or from one or more relational databases to one or more relational database tables without the need to write code. This allows the user to define mapping templates and conditionals to assist in translating and transforming data values. The method also enforces referential integrity, data dependencies, order of operations, and uniqueness constraints using a predefined set of migration rules templates that are based on the principles of relational design. The method uses these mapping and migration rules templates to generate instructions for updating or populating relational database destination tables. The instructions control the data transfer, data translation, data transformation, data validation, foreign key insertion, and the addition of required codes and flags in the destination tables. A migration engine of the system includes a data map architect and an update processor which spawns the templates and migrates the data dynamically, utilizing the data definitions for the destination tables.
This prior art has the drawback that it is limited to a particular type of database systems, in which Oracle® Application tables are implemented. This requires users to manually define rules for the migration or at least interactively, i.e. by user interaction. (Oracle is a trademark of Oracle Corporation). This prior art does not show provisions which allow to discover databases across an organization or augment a consolidation process with actual access patterns.
From the above follows that there is still a need in the art for an efficient scheme that allows to consolidate data from distributed databases into a central database.
SUMMARY AND ADVANTAGES OF THE INVENTION
Disclosed is a scheme that performs two basic tasks, in which a) it monitors existing database access patterns in order to derive an overall view of the available data sources within an organization and how they are used, and b) over time it aggregates the data content of the various data sources into a new centralized repository and redirects calls to the remote database servers to this central database.
For that, three infrastructure elements are employed. So-called sensors, also referred to as interceptors or request detectors, monitor any data access across an organization or network. An inference engine that analyses the access patterns and the data formats contained in the individual databases. A central database, also referred to as central data warehouse or centralized repository, aggregates the data from the individual data sources with the view to replace those eventually.
In the following the single infrastructure elements are explained in more detail.
The sensors or request detector are attached to infrastructure elements, typically software drivers that manage the database access of users and/or applications, and record the requests that are being submitted by users and applications. A typical example of such sensor is a modified ODBC (JDBC) driver. For instance, JDBC drivers are Java code that are frequently used today to access databases from Java programs. By modifying the JDBC driver the sensor logic can record all data requests that are initiated from programs or users to databases, and also which database is being addressed. Other examples can be derived by modifying the database itself. All data access is logged and transmitted to the inference engine, either in real time or in batch mode.
The inference engine analysis the data access recorded by the sensors or request detectors to identify i) the database engines used which are distributed databases within a network or networks and ii) the data schemes employed; for example, the inference engine learns what the format of the data tables is in the various database engines, what primary keys and foreign keys are employed, and what type of data is contained within those databases tables. Further, the inference engine can perform a correlation iii) between different databases; for example, the inference engine should correlate columns from different databases even though they may not be named the same.
Based on the results of the inference the engine generates a new data schema, generates an instance of that schema on the central data warehouse, i.e. the central database. Over time the inference engine then copies existing data from the individual distributed databases that have been discovered into this new central database. When completed, the inference engine may issue an order to redirect calls to the individual databases to the central data warehouse. This can be done by advising the request detectors to intercept the individual data access calls and redirecting them to the central database. The central data warehouse or cental database is a database engine, e.g. an IBM DB/2. For increased availability a cluster may be utilized.
In accordance with the present invention, there is provided a method for consolidating data automatically from distributed databases into a central database. The method comprises the steps of receiving access information comprising request information to the distributed databases, analyzing the received access information, and aggregating into the central database the data content of the distributed databases in dependence on the analyzed access information. This allows a simple automatic migration of redundant data distributed over several databases.
The method can further comprise the steps of filtering the request information to the respective distributed databases from data traffic and forwarding the filtered request information within the access information to an inference engine. All the collected request information can be analyzed in one place, i.e. the information from the various databases can be compared and possible consolidations can be investigated.
For the central database a new data schema based on the analyzed access information can be generated. This has the advantage that a consolidated scheme can be used that meets the needs of the various distributed databases.
The analyzing step can comprise the usage of log-file information. This is simple to perform and does not require any change in the infrastructure, but may not yield access data at the same level of detail as a sensor or request detector would detect.
In accordance with another aspect of the present invention, there is provided a request detector for supporting data consolidation from distributed databases into a central database. The request detector can comprise a detecting means for detecting request information to the distributed databases, a transforming means that derives access information from the detected request information, and a providing means that sends the access information to an inference engine. The request detector can be provided at each of the distributed databases to be consolidated, preferably in form of a modified ODBC (JDBC) driver. The request detector may even be integrated into each of the distributed databases to be consolidated.
The request detector can comprise redirecting means for redirecting a request to an individual database to the central database. This has the advantage that the request is forwarded directly to the consolidated central database and the user may get more information than it is provided by the individual database.
In accordance with yet another aspect of the present invention, there is provided an inference engine for controlling data consolidation from distributed databases into a central database. The inference engine can comprise means for analyzing of access information that is received from distributed databases and comprises request information to the respective distributed databases.
The inference engine can comprise a correlation means for correlating columns and/or rows between different distributed databases, but also fields, records, and/or data structures can be correlated. This leads to a new schema that then can be used by the consolidated central database. The inference engine allows a simple migration of the data. Equivalent information or data is brought together and stored on one place. This helps to avoid doubles in distributed systems.
In accordance with a further aspect of the present invention, there is provided a system for consolidating data from distributed databases into a central database. The system comprises a request detector at each of the distributed databases to be consolidated for providing access information comprising request information to the distributed databases, an inference engine for analyzing the received access information, and a central database into which the data content of the distributed databases is aggregated in dependence on the analyzed access information. DESCRIPTION OF THE DRAWINGS
Preferred embodiments of the invention are described in detail below, by way of example only, with reference to the following schematic drawings.
FIG. 1 shows a schematic illustration of a distributed database structure. FIG. 2 shows a schematic illustration of database structure according to the present invention.
FIG. 3 a shows a schematic illustration of a request and access information flow.
FIG. 3 b shows a schematic illustration of a redirect flow.
FIG. 4 shows schematic illustration of a consolidation of two databases into a central database.
The drawings are provided for illustrative purpose only and do not necessarily represent practical examples of the present invention to scale.
DESCRIPTION OF EMBODIMENTS
Fig. 1 shows a schematic illustration of a distributed database structure 1 with distributed databases 11, 12, 13, 14, 15. The databases, also labeled with DBx, are connected and accessible via a network 6. A user 10, also labeled with U, accesses here three distributed databases 11, 12, 14 in order to receive information that is distributed and provided by a first database 11, a second database 12, and a third database 14. In detail, the user 10, i.e. a user's computer, sends a first request i to the first database 11, a second request r2 to the second database 12, and third request r to the third database 14. The user 10 receives the respective responses from the distributed databases 11, 12, 14 which then can be evaluated. However, in the example, three requests ri, r2, r3 are sent to get the desired information. Further, the maintenance of all the distributed databases 10, 11, 12, 13, 14, 15 with overlapping content is not efficient and effective. The same reference signs are used within the description to denote the same or like parts.
Turning to Fig. 2, which shows a schematic illustration of a modified database structure 2 according to the present invention. The database structure further comprises request detectors 21, 22, 23, 24, 25 attached to each of the distributed databases 11, 12, 13, 14, 15, respectively. Further, there is provided and connected to the network 6 a central data unit 30. This unit comprises an inference engine 32 and a central database 34 which are connected to each other and to the network 6, and thus to the distributed databases 11, 12, 13, 14, 15.
It is assumed that the user 10 sends the same requests ri, r2, r3 to the respective distributed databases 11, 12, 14 as described with reference to Fig. 1. Each of the requests r,, r2, r3 to the respective distributed databases 11, 12, 14 is now detected by the respective request detectors 21, 22, 24. For that, the detectors 21, 22, 23, 24, 25 comprise of detecting means (not shown) for detecting such requests ri, r2, r3 to the distributed databases 11, 12, 14. From the requests ri, r2, r3, also referred to as request information, access information ai, a2, a3 is derived indicating, e.g. a database address, inquiry details, etc.. The access information ai, a2, a3 is then sent to the inference engine 32 as indicated in Fig. 2.
The central data unit 30, i.e. the inference engine 32, receives the access information ai, a2, a3 comprising request information ri, r2, r and analyzes the received access information ai, a , a3 by using correlation means for correlating columns and/or rows between different distributed databases 11, 12, 13, 14, 15. In dependence on the analyzed access information, the data content of the distributed databases 11, 12, 13, 14, 15 is aggregated into the central database 34.
After some time, the distributed databases 11, 12, 13, 14, 15 can be removed, since the content is then consolidated and stored in the cental database 34. This allows more redundancy in storage and better archiving possibilities.
For the user 10 and the system it is advantageous that after the consolidation only one request r is to be sent to the central data unit 30. In response to that request r a more complete data set can be provided to the user 10. Fig. 3a shows a schematic illustration of a flow of a request r and access information a for current distributed database structures in order to establish the central database 34. In the example, the request r is sent by the user 10 to one distributed database lx. The request detector 2x attached to the distributed database lx receives this request r, transforms it to access information a and sends this access information a for purposes of analysis to the inference engine 32.
Fig. 3b shows a schematic illustration of a redirect flow. This is performed when the data consolidation has been performed successfully. Then, the request detector 2x is informed by the inference engine 32 about redirecting and any request r sent by the user 10 is redirected as a redirect request RR by the request detector 2x to the central database 34. The request inquiry is then answered by the consolidated central database 34.
Fig. 4 shows a schematic illustration of a consolidation of two distributed databases 11, 12 into a central database 34. The figure shows on the left hand side the content of the first database 11 and the content of the second database 12. The first database 11 stores "Names", like Joe, Bob and Alice, and their respective "Age", 28, 40, and 18. The second database 12 stores also "Names", here Joe and Bob, but is stores further the "Place of birth" instead of the Ages. As can be seen from this simple example, there are some overlapping names, i.e. information, like it is the case in many distributed database structures today. A data consolidation of the data is therefore desired saving infrastructure and maintenance costs. The inference engine 32 analyzes the available data, compares it, and performs a correlation. This correlation correlates columns and/or rows between different distributed databases to find similarities or matches. As indicated in the figure, the data fields "Name" appear in both distributed databases 11, 12. Thus, the distributed databases 11, 12 are candidates to merge their content into the central database 34. The inference engine 32 generates the new data schema and provides it to the central database 34. Finally, the central database 34 is filled with the content of the distributed databases 11, 12 to have the fields "Name", "Age", and "Place of birth" with the records for 'Joe, 28, Bern'; 'Bob 40, Zurich'; and 'Alice, 18'.
Computer program means or computer program in the present context mean any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following a) conversion to another language, code or notation; b) reproduction in a different material form.
Any disclosed embodiment may be combined with one or several of the other embodiments shown and/or described. This is also possible for one or more features of the embodiments.

Claims

1. A method for consolidating data from distributed databases (11, 12, 13, 14, 15) into a central database (34) comprising the steps of:
- receiving access information (ai, a2, a3) comprising request information (ri, r2, r3) to the distributed databases (11, 12, 14),
- analyzing the received access information (ai, a2, a3), and
- aggregating into the central database (34) the data content of the distributed databases (11, 12, 13, 14, 15) in dependence on the analyzed access information (ai, a2, a3).
2. The method according to claim 1 further comprising providing a request detector (21, 22, 23, 24, 25) at each of the distributed databases (11, 12, 13, 14, 15) to be consolidated.
3. The method according to claim 2 further comprising the steps of filtering the request information (ri, r2, r3) to the respective distributed databases (11, 12, 14) from data traffic and forwarding the filtered request information
Figure imgf000012_0001
r2, r3) within the access information (ai, a2, a3) to an inference engine (32).
4. The method according to claim 2 further comprising the step of integrating the request detector (22) into each of the distributed databases (12) to be consolidated.
5. The method according to any preceding claim further comprising generating for the central database (34) a new data schema based on the analyzed access information (ai, a2, a3).
6. The method according to claim 1, wherein the step of analyzing comprises using log-file information.
7. A computer program element comprising program code means for performing a method of any one of the claims 1 to 6 when said program is run on a computer.
8. A computer program product stored on a computer usable medium, comprising computer readable program means for causing a computer to perform a method according to anyone of the preceding claims 1 to 6.
9. A request detector (21, 22, 23, 24, 25) for supporting data consolidation from distributed databases (11, 12, 13, 14, 15) into a central database (34) comprising a detecting means for detecting request information (r,, r2, r3) to the distributed databases (11, 12, 14), a transforming means that derives access information (ai, a2, a3) from the detected request information (τ r2, r3), and a providing means that sends the access information (ai, a2, a3) to an inference engine (32).
10. The request detector (2x) according to claim 9 further comprising redirecting means for redirecting a request (r) to an individual database (lx) to the central database (34).
11. An inference engine (32) for controlling data consolidation from distributed databases (11, 12, 13, 14, 15) into a central database (34) comprising means for analyzing of access information (ai, a , a3) that is received from distributed databases (11, 12, 14) and comprises request information (rt, r2, r3) to the respective distributed databases (11, 12, 14).
12. The inference engine according to claim 11 further comprising correlation means for correlating columns and/or rows between different distributed databases (11, 12, 13, 14, 15).
3. A system for consolidating data from distributed databases (11, 12, 13, 14, 15;lx) into a central database (34) comprising:
- a request detector (21, 22, 23, 24, 25; 2x) at each of the distributed databases (11, 12, 13, 14, 15; lx) to be consolidated for providing access information (ai, a2, a3; a) comprising request information (ri, r2, r3; r) to the distributed databases (11, 12, 14; lx);
- an inference engine (32) for analyzing the received access information (ai, a2, a3; a); and
- a central database (34) into which the data content of the distributed databases (11, 12, 13, 14, 15; lx) is aggregated in dependence on the analyzed access information (ai, a2, a3; a).
* * *
PCT/IB2004/000669 2003-04-28 2004-03-04 Automatic data consolidation WO2004097677A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP04717176A EP1618495A1 (en) 2003-04-28 2004-03-04 Automatic data consolidation
US11/227,875 US7401101B2 (en) 2003-04-28 2005-09-15 Automatic data consolidation
US12/121,002 US7930312B2 (en) 2003-04-28 2008-05-15 System for consolidating data from distributed databases

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP03405295 2003-04-28
EP03405295.1 2003-04-28

Publications (1)

Publication Number Publication Date
WO2004097677A1 true WO2004097677A1 (en) 2004-11-11

Family

ID=33396081

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2004/000669 WO2004097677A1 (en) 2003-04-28 2004-03-04 Automatic data consolidation

Country Status (4)

Country Link
US (2) US7401101B2 (en)
EP (1) EP1618495A1 (en)
CN (1) CN100378731C (en)
WO (1) WO2004097677A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2431257A (en) * 2005-10-17 2007-04-18 Celona Technologies Ltd System and method for accessing data

Families Citing this family (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004097677A1 (en) * 2003-04-28 2004-11-11 International Business Machines Corporation Automatic data consolidation
US7433901B2 (en) * 2004-09-17 2008-10-07 International Business Machines Corporation Apparatus, system, and method for using multiple criteria to determine collocation granularity for a data source
CN101276364B (en) * 2007-03-30 2010-12-22 阿里巴巴集团控股有限公司 Method, system and apparatus for combining distributed computational data
CN101291245B (en) * 2007-04-16 2011-09-28 阿里巴巴集团控股有限公司 Distributed processing method, system and device thereof
US8352716B1 (en) * 2008-01-16 2013-01-08 American Megatrends, Inc. Boot caching for boot acceleration within data storage systems
US8799429B1 (en) 2008-05-06 2014-08-05 American Megatrends, Inc. Boot acceleration by consolidating client-specific boot data in a data storage system
US9009098B1 (en) * 2008-06-30 2015-04-14 Emc Corporation Methods and apparatus for creating a centralized data store
US8285859B2 (en) * 2009-11-20 2012-10-09 Nokia Corporation Method and apparatus for optimizing distribution of information and queries in information spaces
US8606756B2 (en) * 2010-04-09 2013-12-10 Ca, Inc. Distributed system having a shared central database
US20120053957A1 (en) * 2010-05-18 2012-03-01 Lorri Atkins Tool for detecting inconsistencies and omissions in documented justification for medical services
US9280391B2 (en) * 2010-08-23 2016-03-08 AVG Netherlands B.V. Systems and methods for improving performance of computer systems
US9020946B2 (en) * 2010-07-12 2015-04-28 Qvinci Software, Llc System and method for compilation of quickbooks accounts data
CN102446230B (en) * 2010-10-11 2013-03-13 上海华虹Nec电子有限公司 Method for merging GDSII layout data
US9858624B2 (en) 2012-10-04 2018-01-02 Qvinci Software, Llc Methods and apparatus for providing data normalization, scalability and maintainability
EP3014490A4 (en) * 2013-06-24 2017-02-22 K2View Ltd. Cdbms (cloud database management system) distributed logical unit repository
US10657115B2 (en) * 2016-03-31 2020-05-19 Mckesson Corporation Methods and apparatuses for improved data modeling using a relational database management system
US11442953B2 (en) 2016-03-31 2022-09-13 Mckesson Corporation Methods and apparatuses for improved data ingestion using standardized plumbing fields
US11625662B2 (en) 2016-09-22 2023-04-11 Qvinci Software, Llc Methods and apparatus for the manipulating and providing of anonymized data collected from a plurality of sources
US10956386B2 (en) 2018-03-30 2021-03-23 Mckesson Corporation Methods and apparatuses for automated performance tuning of a data modeling platform

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5710917A (en) * 1995-06-07 1998-01-20 International Business Machines Corporation Method for deriving data mappings and data aliases
US6151608A (en) * 1998-04-07 2000-11-21 Crystallize, Inc. Method and system for migrating data
US20020107871A1 (en) * 2001-02-05 2002-08-08 Knowledge Computing Corporation Method and system for database migration and association

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5717604A (en) 1995-05-25 1998-02-10 Wiggins; Christopher Network monitoring system for tracking, billing and recovering licenses
WO1998044442A1 (en) 1997-04-02 1998-10-08 Citibank, N.A. Method and system for standardizing and reconciling invoices from vendors
US6095787A (en) * 1998-10-19 2000-08-01 The Colonel's, Inc. Method of making a skid-resistant bed liner
US6356880B1 (en) 1999-04-27 2002-03-12 Oracle Corporation Methods and systems for dynamic cost allocation through task auto assignment
US6633878B1 (en) * 1999-07-30 2003-10-14 Accenture Llp Initializing an ecommerce database framework
FR2806183B1 (en) * 1999-12-01 2006-09-01 Cartesis S A DEVICE AND METHOD FOR INSTANT CONSOLIDATION, ENRICHMENT AND "REPORTING" OR BACKGROUND OF INFORMATION IN A MULTIDIMENSIONAL DATABASE
US6523173B1 (en) 2000-01-11 2003-02-18 International Business Machines Corporation Method and apparatus for allocating registers during code compilation using different spill strategies to evaluate spill cost
US6615220B1 (en) * 2000-03-14 2003-09-02 Oracle International Corporation Method and mechanism for data consolidation
WO2001092993A2 (en) 2000-06-02 2001-12-06 Vigilant Systems, Inc. System and method for licensing management
US20020069143A1 (en) 2000-12-06 2002-06-06 Cepeda Jose Alberto System and method for allocating operating expenses
US7194439B2 (en) 2001-04-30 2007-03-20 International Business Machines Corporation Method and system for correlating job accounting information with software license information
US6792431B2 (en) * 2001-05-07 2004-09-14 Anadarko Petroleum Corporation Method, system, and product for data integration through a dynamic common model
US6785689B1 (en) * 2001-06-28 2004-08-31 I2 Technologies Us, Inc. Consolidation of multiple source content schemas into a single target content schema
US7177850B2 (en) 2001-10-16 2007-02-13 Infineon Technologies Ag Method and apparatus for determining a portion of total costs of an entity
US20030236721A1 (en) 2002-05-21 2003-12-25 Plumer Edward S. Dynamic cost accounting
US20040088730A1 (en) 2002-11-01 2004-05-06 Srividya Gopalan System and method for maximizing license utilization and minimizing churn rate based on zero-reject policy for video distribution
US20040088176A1 (en) 2002-11-04 2004-05-06 Balaji Rajamani System and method of automated licensing of an appliance or an application
US20040093594A1 (en) 2002-11-13 2004-05-13 Viren Kapadia System and process for developing and maintaining software applications for a business organization
US20040154000A1 (en) 2003-02-03 2004-08-05 Kasra Kasravi System and method for semantic software analysis
WO2004097677A1 (en) * 2003-04-28 2004-11-11 International Business Machines Corporation Automatic data consolidation
US7831457B2 (en) 2003-06-17 2010-11-09 Satyam Computer Services Limited Of Mayfair Center System and method for maximizing software package license utilization
JP4336858B2 (en) 2003-07-02 2009-09-30 日本電気株式会社 Policy processing system, policy processing method, and policy processing program
US20050060662A1 (en) 2003-08-22 2005-03-17 Thomas Soares Process for creating service action data structures
US20050049887A1 (en) 2003-08-29 2005-03-03 Bulleit Douglas A. Methods, systems, and computer program products for allocating costs in using a broadband communication network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5710917A (en) * 1995-06-07 1998-01-20 International Business Machines Corporation Method for deriving data mappings and data aliases
US6151608A (en) * 1998-04-07 2000-11-21 Crystallize, Inc. Method and system for migrating data
US20020107871A1 (en) * 2001-02-05 2002-08-08 Knowledge Computing Corporation Method and system for database migration and association

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2431257A (en) * 2005-10-17 2007-04-18 Celona Technologies Ltd System and method for accessing data

Also Published As

Publication number Publication date
US20060036624A1 (en) 2006-02-16
CN1717686A (en) 2006-01-04
US7930312B2 (en) 2011-04-19
US20080222163A1 (en) 2008-09-11
EP1618495A1 (en) 2006-01-25
US7401101B2 (en) 2008-07-15
CN100378731C (en) 2008-04-02

Similar Documents

Publication Publication Date Title
US7401101B2 (en) Automatic data consolidation
US20220405298A1 (en) Large scale unstructured database systems
Hammer et al. The Stanford data warehousing project
US9158843B1 (en) Addressing mechanism for data at world wide scale
US6681227B1 (en) Database system and a method of data retrieval from the system
JP3851493B2 (en) Database search method, database search system, and computer-readable recording medium recording database search program
US9292575B2 (en) Dynamic data aggregation from a plurality of data sources
CN100478944C (en) Automatic task generator method and system
US20050165865A1 (en) Metadata journal for information technology systems
US20030097359A1 (en) Deduplicaiton system
EP1482418A1 (en) A data processing method and system
EP2182448A1 (en) Federated configuration data management
MXPA01002852A (en) Database synchronization and organization system and method.
US20080027899A1 (en) Systems and Methods for Integrating from Data Sources to Data Target Locations
US20050005237A1 (en) Method for maintaining a centralized, multidimensional master index of documents from independent repositories
US20090183175A1 (en) Systems and methods for receiving and sending messages about changes to data attributes
CN1416078A (en) Data processing method of monitoring system
KR20170096302A (en) System of Parallel Distributed Processing System for Heterogeneous Data Processing
CN111125045B (en) Lightweight ETL processing platform
US20210200731A1 (en) Horizontal skimming of composite datasets
KR20040077535A (en) System and method for generating a request for information about selected objects
JP2009122995A (en) Management system and management method of related process record
JP2002175327A (en) Method for managing database
US8171017B2 (en) Book of business mechanism
JP2001117798A (en) Electronic data integration data management system

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): BW GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
WWE Wipo information: entry into national phase

Ref document number: 2004801568X

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2004717176

Country of ref document: EP

WWP Wipo information: published in national office

Ref document number: 2004717176

Country of ref document: EP

DPEN Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed from 20040101)
WWW Wipo information: withdrawn in national office

Ref document number: 2004717176

Country of ref document: EP