CN102402600B - Asynchronous collecting method for maintaining consistency of database replicas - Google Patents
Asynchronous collecting method for maintaining consistency of database replicas Download PDFInfo
- Publication number
- CN102402600B CN102402600B CN 201110367105 CN201110367105A CN102402600B CN 102402600 B CN102402600 B CN 102402600B CN 201110367105 CN201110367105 CN 201110367105 CN 201110367105 A CN201110367105 A CN 201110367105A CN 102402600 B CN102402600 B CN 102402600B
- Authority
- CN
- China
- Prior art keywords
- data
- duplicate
- complete
- tuple
- local
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses an asynchronous collecting method for maintaining consistency of database replicas, comprising the steps of: 1) dividing a database differential replica into a complete replica type and a partial replica type; dividing the data into splinters according to the application requirement and storing them into two types of data replicas; 2) comparing the database contents to find the differential data in the complete replica and the partial replica, before modifying the local data, renewing the data and synchronizing the data from the complete replica to the partial replica; 3) after modifying the local data, executing a committing operation, and synchronizing the data from the partial replica to the complete replica. The asynchronous collecting method provided by the invention supports multi-database concurrent changes, overcomes the dependence on the dump file and greatly reduces the dependence on the network stability compared with the online synchronous method realized via an application software.
Description
Technical field
The present invention relates to a kind ofly based on collecting thought, the asynchronous method of data consistency between maintenance data base otherness duplicate belongs to electric power system data library management technical field.
Background technology
In the large-scale supervisory system of Fen Saning, model data is distributed in disparate databases more geographically.Common Distribution Strategy has: centralized: data centralization is deposited; Partition type: data disperse to deposit, but can not repeat between data; Redundance type: each node is all deposited complete data duplicate; Hybrid: above-mentioned three kinds mixing, this is the most complicated, flexible way, i.e. the database otherness duplicate that the present invention is suitable for.
Can be divided into two classes for this class otherness duplicate consistency maintaining method: offline synchronization and online synchronous.
Simple offline synchronization method normally exports to dump file with all objects of source database, dump file is imported the consistance of target database with maintenance and source database again.This method for synchronous is extended to indifference opposite sex duplicate with the otherness duplicate according to its extension, and is simple and reliable, but is not suitable for the application of the concurrent modification of a plurality of databases.
Improved offline synchronization method is divided into sliver on demand with all tuples in the source database, to need the tuple place sliver of maintaining coherency to export to dump file in the source database, again dump file is imported to the original corresponding sliver of covering in the target database.This method has remedied the defective that simple offline synchronization is not supported the concurrent modification of multiple database, but conforming maintenance need increase maintenance difficulties by means of copy, transmission and the merging of a plurality of files of operating system aspect between the storehouse.If can not effectively manage to file, very easily cause the version error of file, increase the inconsistent risk of data.
Online is application software synchronously after local data base is carried out the write operation success, directly revises other databases by network mode again; Or send the message packet that comprises synchronizing information to other database servers.The online problem that has well solved the existence of offline synchronization method synchronously, data consistency is safeguarded and is realized by application software system, can satisfy the demand of using data so flexibly.It is comparatively harsh that but online method for synchronous requires network environment, in case the concurrent modification of multiple database has taken place when network congestion, be easy to cause data inconsistent, even network recovery afterwards, also possibly can't return to consistent state, thereby cause nonvolatil error in data.Then carrying out data modification in the inconsistent database, involve key word in the table, index, and trigger all multi-constraint conditions such as relation between table, also may cause the inconsistent expanded range of data.This catastrophic consequence is that system developer and engineering construction person least wish to see.
Summary of the invention
Technical matters to be solved by this invention is that the consistance of realization service data tuple between the otherness duplicate handled the problem of the concurrent modification of multiple database well, and alleviated the degree of dependence of maintenance work to network quality.
For solving the problems of the technologies described above, the invention provides the conforming asynchronous method that collects of a kind of maintenance data base duplicate, it is characterized in that, may further comprise the steps:
1) database otherness duplicate is divided into complete duplicate and local copy two classes, according to the requirement of using data is divided into sliver and leaves in the two class data duplicates;
2) relatively find out variance data in complete duplicate and the local copy by data-base content, before revising local data, carry out and upgrade operation, from complete duplicate to the local copy synchrodata;
3) finish back execution submit operation revising local data, from local copy to complete duplicate synchrodata.
Described complete duplicate comprises the overall situation, and all are centralized, the partition type data of redundance type data and this node; Local copy comprises the partition type data of redundance type data and this node;
Described submit operation is meant to be carried out the data modification of local local copy fully on the duplicate;
Described renewal behaviour is meant the data modification of complete duplicate is carried out on local local copy.
The conforming asynchronous method that collects of aforesaid maintenance data base duplicate is characterized in that: in described step 3), local copy is only revised at local node, after modification is finished local copy is committed to complete duplicate.
The conforming asynchronous method that collects of aforesaid maintenance data base duplicate is characterized in that: in described step 2) in, duplicate is only revised at local node fully, need not submit to after modification is finished.
The conforming asynchronous method that collects of aforesaid maintenance data base duplicate, it is characterized in that: submitting to complete duplicate and when complete duplicate upgrades, carry out data check, if find to exist conflicting data to stop and the current submit operation of rollback or renewal operation, pop-up window notification operator simultaneously.
The conforming asynchronous method that collects of aforesaid maintenance data base duplicate is characterized in that: in described step 2) and step 3) in, optimize the triggering logical relation in the database, make the triggering logical equivalence of insert, update, delete unified.
The conforming asynchronous method that collects of aforesaid maintenance data base duplicate, it is characterized in that: in described step 2) and step 3) in, increasing the attribute that is used for the data of description version change in the relation that produces by using, comprising: tuple creation-time, tuple are created node, tuple modification time, tuple modification node, tuple deleted marker, tuple deletion time and tuple deletion node.
The conforming asynchronous method that collects of aforesaid maintenance data base duplicate, it is characterized in that: non-0 this tuple of expression of tuple deleted marker attribute is deleted, the tuple deleted marker is as the associating major key of relation, if the value of intrafascicular approximately other row of major key is identical, then delete flag adds up since 1.
The conforming asynchronous method that collects of aforesaid maintenance data base duplicate is characterized in that: the data difference opposite sex of relatively more complete copy and local copy when upgrading operation and submit operation, and in conjunction with the processing that conflicts of tuple running time.
The conforming asynchronous method that collects of aforesaid maintenance data base duplicate, it is characterized in that: in described step 3), local copy was updated to earlier before revising local data and the complete the same versions of data of duplicate, and local copy is submitted to complete duplicate after finishing and revising in time.
The present invention proposes the notion of data duplicate grade:, the data duplicate is divided into two kinds of complete duplicate and local copy according to the coverage of data duplicate to total data.Fully comprised the overall situation all are centralized, the partition type data of redundance type data and this node for duplicate; Local copy comprises the partition type data of redundance type data and this node.
Local copy is only revised at local node, after modification is finished local copy is committed to complete duplicate, and other nodes can upgrade the data of having submitted to from complete duplicate.Merge from the redundance type data of local copy submission and the redundance type data the complete duplicate; The partition type data are not submitted to.
Duplicate is only revised at local node fully, after finishing, modification need not submit to, redundance type data wherein can be updated directly into local copy when needed, local copy should be before once revising beginning from complete duplicate new data more, and after finishing modification, in time be committed to complete duplicate.
Submitting to complete duplicate and when complete duplicate upgrades, doing data check, in case find to exist conflicting data just to stop and the current submission of rollback or upgrade operation, pop-up window notification operator simultaneously.
The beneficial effect that the present invention reached:
Than the offline synchronization method, the asynchronous method of collecting that the present invention proposes can be supported the concurrent modification of multiple database, has overcome its dependence to dump file simultaneously.Than the online method for synchronous of realizing by application software, the asynchronous method of collecting that the present invention proposes greatly reduces the dependence to network stabilization, all the time all stablize from requiring network, be reduced to and only require and before revising beginning, upgrade and modification is stablized when finishing the back submission, and can in time detect the inconsistent situation of the data that existed and notify at once to the operator, prevent that effectively the inconsistent scope of data from continuing to enlarge.
Description of drawings
Fig. 1 is the conforming asynchronous process flow diagram that collects method of maintenance data base duplicate of the present invention.
Embodiment
The asynchronous realization that collects method that the present invention proposes mainly comprises 6 steps:
Step 1: the data in the database are cut apart, according to demands of applications, model data is carried out relative degree/tuple level by following 4 kinds, perhaps horizontal/vertical/compound cutting apart:
1. only be stored in the data in the complete duplicate, this part data is referred to as centralized data;
2. only be stored in a certain local copy, this part data is only preserved portion in the overall situation, and this part data is referred to as the partition type data;
3. be stored in complete duplicate and each part local copy, and all require the data that are consistent in all above-mentioned duplicates, this part data is referred to as the redundance type data;
4. be stored in the data in local local copy and the complete duplicate.
First kind of data only has requirements for access at complete duplicate place node, second kind of data only has requirements for access at this local copy place node, so these two kinds of data do not need to carry out consistency maintenance, the third data all have requirements for access at complete duplicate and all local copy place nodes, it promptly may be certain common data commonly used, this is the part the highest to coherence request, the 4th kind of data are common in during the have hub-and-spoke configuration engineering of (being center-minute site type structure) uses, data are in local local copy and in the duplicate requirements for access is arranged fully, do not need visit at other local copy place nodes, but other local nodes are duplicated in this part data commentaries on classics also can not produce any influence the data consistance, so can conclude in the redundance type data, adopt identical disposal route.
Step 2: definition submit operation and renewal operation:
Submit operation: the data modification of local local copy is being carried out on the duplicate fully;
Upgrade operation: the data modification of complete duplicate is carried out on local local copy.
Step 3: the content difference of more local local copy and the complete duplicate of far-end:
Because submission and renewal operation are all carried out on the node of local copy place, so local copy is local, duplicate is a far-end fully.Successively the identical relation in the complete duplicate of local local copy and far-end is compared, 4 kinds of results may occur: " coupling ", " disappearance ", " exceeding ", " difference ", as table 1:
The table as a result that table 1 data-base content relatively may occur
| ? | The local copy Major key | Complete duplicate Major key | The record attribute row |
| Coupling | Exist | Exist | Consistent |
| Disappearance | Exist | There is not (or having deleted) | ? |
| Exceed | There is not (or having deleted) | Exist | ? |
| Difference | Exist | Exist | Inconsistent |
Step 4: the triggering logic between optimum relations
In the database of real application systems, can there be the logical relation of some between relation and the relation, this relation generally realizes by the triggering logic of database.This step is to optimize these to trigger logical relation, and the triggering unified logic of insert, update, delete is got up.For example when update concerns the record of A, can trigger and in concern B, insert new record if a certain Column Properties is put 1, in concerning A, just must detect whether these row are 1 during one of insert so, 1 also in concerning B, insert respective record if be; In like manner, the deletion action for update causes also will take in when carrying out insert and delete, makes being kept perfectly property of logic and the consistance of Insert, update, delete.
Step 5: in relation, increase the control information attribute:
In order to have described the state of tuple in the relation, need additionally add some attributes in the relation that produce by using, these attributes comprise: the tuple creation-time, tuple is created node, the tuple modification time, and tuple is revised node, the tuple deleted marker, tuple deletion time, tuple deletion node.Wherein, it is to carry out in the node at that duplicate place that tuple establishment/modification/deletion node is used for recording operation.Non-0 this tuple of expression of tuple deleted marker attribute is deleted, and the tuple deleted marker is as the associating major key of relation, if the value of intrafascicular approximately other row of major key is identical, then delete flag adds up since 1.For example: alternately carry out repeatedly and insert and sweep equipment number is 100 tuple 3 times, then the delete flag attribute of 3 tuples is respectively 1,2,3.Owing to when carrying out deletion action, being not really tuple to be deleted, be changed to non-0 value but will delete first group mark, so can produce unnecessary tuple in the relation, these tuples can be unified deletion behind once overall consistency maintenance EO.
Step 6: the conflict when submission and renewal is handled:
Carrying out submit operation and upgrading in the process of operation, can skip for the tuple of having mated; For lack, exceed, the situation of difference needs to be treated differently as follows.
During renewal: for " disappearance ", the deletion time of more complete duplicate and the access time in the local copy (creation-time or recently modification time) are if the deletion time gap current time is nearer then delete tuple in the local copy (is non-0 value with the deleted marker position).For " exceeding ", access time in the more complete duplicate and the deletion time in the local copy are if the access time is nearer then increase respective tuple in local copy apart from the current time.For " difference ", in the more complete duplicate and the access time in the local copy, if the access time fully in the duplicate is nearer then with the tuple in the replacement of the tuple in the complete duplicate local copy apart from the current time.If above-mentioned condition does not satisfy then keeps present situation, do not change the data in the local copy.
During submit operation: for " disappearance ", deletion time in the more complete duplicate and the access time in the local copy (creation-time or recently modification time) are if the access time is nearer then adding respective tuple in the duplicate fully apart from the current time.For " exceeding ", access time in the more complete duplicate and the deletion time in the local copy are if the deletion time gap current time is nearer then delete tuple in the complete duplicate.For " difference ", in the more complete duplicate and the access time in the local copy, if the access time in the local copy is nearer then replace tuple in the complete duplicate with the tuple in the local copy apart from the current time.Submit failure to if above-mentioned condition does not satisfy, the versions of data in the pop-up window prompting local copy is too outmoded, need be updated to latest edition and carry out data modification again.
The asynchronous method that collects that the present invention proposes, in the concrete process of implementing, all local copy should be updated to earlier before revising local data and the complete the same versions of data of duplicate, same, after local copy is finished modification, also should submit to complete duplicate in time, can reduce like this and submit unsuccessful occurrence probability to.
Recommended flowsheet when local copy is carried out data modification each time is: carry out and upgrade the data à that versions of data à revises local local copy and submit the data of revising to complete duplicate to; Carrying out data modification at complete duplicate does not need to upgrade and submit to.
In addition to the implementation, all employings are equal to the technical scheme that form obtained of replacement or equivalent transformation, all drop within protection scope of the present invention.
Claims (9)
1. the conforming asynchronous method that collects of maintenance data base duplicate is characterized in that, may further comprise the steps:
1) database otherness duplicate is divided into complete duplicate and local copy two classes, according to the requirement of using data is divided into sliver and leaves in the two class data duplicates;
2) relatively find out variance data in complete duplicate and the local copy by data-base content, before revising local data, carry out and upgrade operation, from complete duplicate to the local copy synchrodata;
3) finish back execution submit operation revising local data, from local copy to complete duplicate synchrodata,
Described complete duplicate comprises the overall situation, and all are centralized, the partition type data of redundance type data and this node; Local copy comprises the partition type data of redundance type data and this node;
Described submit operation is meant to be carried out the data modification of local local copy fully on the duplicate;
Described renewal operation is meant to be carried out the data modification of complete duplicate on local local copy,
Database otherness duplicate is meant the database model data with following three kinds of mode mixed distribution, and three kinds of distribution modes are:
A) centralized: data centralization is deposited;
B) partition type: data disperse to deposit, but can not repeat between data;
C) redundance type: each node is all deposited complete data duplicate.
2. the conforming asynchronous method that collects of maintenance data base duplicate according to claim 1 is characterized in that: in described step 3), local copy is only revised at local node, after modification is finished local copy is committed to complete duplicate.
3. the conforming asynchronous method that collects of maintenance data base duplicate according to claim 1 is characterized in that: in described step 2) in, duplicate is only revised at local node fully, need not submit to after modification is finished.
4. the conforming asynchronous method that collects of maintenance data base duplicate according to claim 1, it is characterized in that: submitting to complete duplicate and when complete duplicate upgrades, carry out data check, if find to exist conflicting data to stop and the current submit operation of rollback or renewal operation, pop-up window notification operator simultaneously.
5. the conforming asynchronous method that collects of maintenance data base duplicate according to claim 1 is characterized in that: in described step 2) and step 3) in, optimize the triggering logical relation in the database, make the triggering unified logic of insert, update, delete.
6. the conforming asynchronous method that collects of maintenance data base duplicate according to claim 1, it is characterized in that: in described step 2) and step 3) in, increasing the attribute that is used for the data of description version change in the relation that produces by using, comprising: tuple creation-time, tuple are created node, tuple modification time, tuple modification node, tuple deleted marker, tuple deletion time and tuple deletion node.
7. the conforming asynchronous method that collects of maintenance data base duplicate according to claim 6, it is characterized in that: non-0 this tuple of expression of tuple deleted marker attribute is deleted, the tuple deleted marker is as the associating major key of relation, if the value of intrafascicular approximately other row of major key is identical, then delete flag adds up since 1.
8. the conforming asynchronous method that collects of maintenance data base duplicate according to claim 1 is characterized in that: the data difference opposite sex of relatively more complete copy and local copy when upgrading operation and submit operation, and in conjunction with the processing that conflicts of tuple running time.
9. the conforming asynchronous method that collects of maintenance data base duplicate according to claim 1, it is characterized in that: in described step 3), local copy was updated to earlier before revising local data and the complete the same versions of data of duplicate, and local copy is submitted to complete duplicate after finishing and revising in time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110367105 CN102402600B (en) | 2011-11-18 | 2011-11-18 | Asynchronous collecting method for maintaining consistency of database replicas |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110367105 CN102402600B (en) | 2011-11-18 | 2011-11-18 | Asynchronous collecting method for maintaining consistency of database replicas |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102402600A CN102402600A (en) | 2012-04-04 |
| CN102402600B true CN102402600B (en) | 2013-07-31 |
Family
ID=45884800
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110367105 Active CN102402600B (en) | 2011-11-18 | 2011-11-18 | Asynchronous collecting method for maintaining consistency of database replicas |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102402600B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020149B (en) * | 2012-11-22 | 2016-01-20 | 用友网络科技股份有限公司 | Share data update apparatus and shared data-updating method |
| CN104216830B (en) * | 2014-09-01 | 2017-05-10 | 广州供电局有限公司 | Method and system for detecting consistency of equipment software |
| CN104298769B (en) * | 2014-10-29 | 2017-06-13 | 北京思特奇信息技术股份有限公司 | Domain variance data synchronization system and method are had between a kind of database |
| CN106559225B (en) * | 2015-09-24 | 2019-08-16 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data calibration |
| CN106055353B (en) * | 2016-05-23 | 2019-03-29 | 北京小米移动软件有限公司 | Data-updating method and device |
| CN107609048B (en) * | 2017-08-18 | 2020-09-04 | 中国人民大学 | Method capable of delaying unique constraint |
| CN108038140A (en) * | 2017-11-23 | 2018-05-15 | 广东岭南通股份有限公司 | Traffic all-purpose card transaction data full dose duplicate checking method and system |
| CN111625601B (en) * | 2020-05-27 | 2023-03-14 | 北京火山引擎科技有限公司 | Data writing method and device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7177866B2 (en) * | 2001-03-16 | 2007-02-13 | Gravic, Inc. | Asynchronous coordinated commit replication and dual write with replication transmission and locking of target database on updates only |
| CN101488148A (en) * | 2009-02-25 | 2009-07-22 | 浪潮集团山东通用软件有限公司 | Data compilation method in multiple databases |
| CN101826073A (en) * | 2009-03-06 | 2010-09-08 | 华为技术有限公司 | Synchronous method, apparatus and system for distributed database |
-
2011
- 2011-11-18 CN CN 201110367105 patent/CN102402600B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7177866B2 (en) * | 2001-03-16 | 2007-02-13 | Gravic, Inc. | Asynchronous coordinated commit replication and dual write with replication transmission and locking of target database on updates only |
| CN101488148A (en) * | 2009-02-25 | 2009-07-22 | 浪潮集团山东通用软件有限公司 | Data compilation method in multiple databases |
| CN101826073A (en) * | 2009-03-06 | 2010-09-08 | 华为技术有限公司 | Synchronous method, apparatus and system for distributed database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102402600A (en) | 2012-04-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102402600B (en) | Asynchronous collecting method for maintaining consistency of database replicas | |
| CN101334797B (en) | Distributed file systems and its data block consistency managing method | |
| CN103455557B (en) | A kind of structural data synchronous method based on daily record | |
| US20110167037A1 (en) | Traversal-free rapid data transfer | |
| CN101577735A (en) | Method, device and system for taking over fault metadata server | |
| EP3575968A1 (en) | Method and device for synchronizing active transaction lists | |
| CN102663076B (en) | Method for processing file data | |
| CN105930498A (en) | Distributed database management method and system | |
| CN108369601A (en) | Promotion attribute in relational structure data | |
| CN106802905B (en) | Collaborative data exchange method of isomorphic PLM system | |
| CN103345502A (en) | Transaction processing method and system of distributed type database | |
| CN105912691A (en) | Power grid model multi-version multi-tenant management system and method based on distributed storage | |
| CN102890678A (en) | Gray-code-based distributed data layout method and query method | |
| WO2021046750A1 (en) | Data redistribution method, device, and system | |
| CN109710586B (en) | A kind of clustered node configuration file synchronous method and device | |
| CN108431807A (en) | The duplication of structured data in partition data memory space | |
| CN112416456A (en) | Configuration file processing method, device, equipment, storage medium and system | |
| WO2023066086A1 (en) | Data processing method, distributed database system, electronic device and storage medium | |
| US7020656B1 (en) | Partition exchange loading technique for fast addition of data to a data warehousing system | |
| CN110727733B (en) | Master-slave server system and data synchronization method | |
| CN110543606B (en) | Method and system for storing genealogy data based on alliance chain | |
| CN101046822A (en) | Centralized management of data nodes | |
| CN114153809A (en) | Parallel real-time incremental statistic method based on database logs | |
| EP2722774A1 (en) | Long term archiving of data in a MES system | |
| CN102193987B (en) | Method and system for increasing node data relationship based on OLTP (online transaction processing) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| ASS | Succession or assignment of patent right |
Owner name: NARI NANJING CONTROL SYSTEM LTD. Effective date: 20141205 |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20141205 Address after: High road high tech Development Zone Nanjing city Jiangsu province 210061 No. 20 Patentee after: NARI Technology Development Co., Ltd. Patentee after: SGCC NARI Nanjing Control System Co., Ltd. Address before: High road high tech Development Zone Nanjing city Jiangsu province 210061 No. 20 Patentee before: NARI Technology Development Co., Ltd. |
|
| CB03 | Change of inventor or designer information |
Inventor after: Chen Tianhao Inventor after: Liang Yi Inventor after: Fang Jun Inventor after: Zhang Saiqiao Inventor after: Shi Qi Inventor after: Liu Jiabao Inventor after: Xu Shiming Inventor after: Li Tianyang Inventor before: Chen Tianhao Inventor before: Liang Yi Inventor before: Fang Jun Inventor before: Zhang Saiqiao Inventor before: Shi Qi Inventor before: Liu Jiabao |
|
| COR | Change of bibliographic data |