WO2012071719A1

WO2012071719A1 - Method, apparatus and system for accessing data in cloud storage

Info

Publication number: WO2012071719A1
Application number: PCT/CN2010/079327
Authority: WO
Inventors: 刘慧�
Original assignee: 北京卓微天成科技咨询有限公司
Priority date: 2010-11-29
Filing date: 2010-12-01
Publication date: 2012-06-07
Also published as: CN102055797A

Abstract

A method, apparatus and system for accessing data in cloud storage are disclosed. Steps of storing data include: creating data check values of original data to be stored in a data transmitting terminal (S101); transmitting the original data and the data check values to different cloud storage data centres for storing in accordance with pre-configured redundancy storing policy (S102). Steps of retrieving data include: based on an access request, accessing the cloud storage data centre storing the original data in accordance with the pre-configured redundancy storing policy (S201); determining whether the original data stored in the cloud storage data centre is available (S202); if no, retrieving the data check value of the original data and data used for creating the data check value from the other cloud storage data centre (S203); recovering the original data according to the data check value and the data used for creating the data check value (S204). This technical solution improves availability and fault tolerance of data in cloud storage.

Description

Method, device and system for data access of cloud storage

Technical field

The present invention relates to the field of cloud storage technologies, and in particular, to a method, device and system for data access of cloud storage.

Background technique

Data has proven to be one of the company's key assets, and the rapid growth of data has made companies face unprecedented challenges. At the same time, the ever-changing world economic situation and the cost pressures of fierce competition have forced companies to consider how to reduce IT costs and respond to the growing storage needs of enterprises.

Existing storage architectures can be divided into two types: one is a proprietary architecture, such as

DAS (direct attached storage, direct attached storage), SAN (Storage Area Network) and network access to the network acces s server (NAS). This kind of storage system is used exclusively by one party, which can provide users with good control, better reliability and performance, but its scalability is poor, and it is not suitable for large-scale deployment; users are also difficult in this mode. Flexible use of storage budgets (requires one-time purchase of storage devices); as storage capacity increases, cost control will also face challenges.

The other is a multi-party shared architecture, the cloud storage architecture. According to the scope of its services, it is divided into private cloud (public cloud) and public cloud (public cloud). The cloud storage architecture is based on network technologies (internet and intranet) that provide storage space for on-demand purchases, leases, and on-demand provisioning services. This service is typically provided by third parties (or third-party departments within the enterprise) including storage devices and specialized Maintenance personnel. Through this storage service, enterprises (or departments within the enterprise) can significantly reduce their internal storage requirements and corresponding management costs to balance the sharply rising storage requirements and enterprise cost pressures. The users of the above mentioned storage may be individuals, enterprises, or even departments or branches within the enterprise. In addition to different service objects and service categories, public cloud storage systems and private cloud storage, and traditional network backup systems in the enterprise, are different in that cloud storage service providers (data centers) are relatively independent, limited to security considerations and data compatibility. Sexual problems, there is little need for information exchange and communication between them.

Public cloud storage services can reduce the cost of enterprise and personal storage, increase the availability of data, and even the data in the cloud storage service provider's data center is lost or tombed. This risk allows businesses or individuals to use public cloud storage services. In particular, key data related to its business data is prohibitive. For example, the cloud storage service provider is bankrupt or other reasons, so that the cloud storage service provided by it is no longer available, which leads to the risk of data being stored on it; or the enterprise chooses some smaller cloud storage service due to the price of the service. Provider, but it is bankrupt due to poor management; or due to irresistible causes, such as earthquakes or major disasters, cloud storage service providers' data center data is lost; or for some reason, such as power outages, cloud storage service providers Data is not available (generally cloud storage service providers promise 99.99% or 99.999% availability of their services); or data center data loss due to human error due to corporate mergers or other reasons; and viruses or hackers The resulting cloud storage data center data is lost or altered by the tomb.

Given the importance of data to the enterprise or due to legal requirements, it is necessary to increase the fault tolerance of public cloud storage system data, especially for companies or individuals.

Traditionally used to improve the availability of public cloud storage services and the fault tolerance of data, usually rely on the enterprise to sign a service agreement (SLA) with the cloud storage service provider and the RAID mechanism on the cloud storage data center (mainly RAI D 2 - RAID6) . For references, see Dav id A. Pa tt er son, Gar th G i bson, and Randy H. Ka tz: A Ca se for Redundant Arrays of Inexpensive Di sks (RAID) . ACM 1988. RAID 2 - RAID6 ensures sufficient storage by storing the checksum value of the data transferred to the storage medium. When the storage medium hardware is damaged, it will not cause data loss.

The RAI D mechanism is used for data protection of the data receiving end for network storage, and data redundancy is created by the data check value at the data receiving end to protect data loss caused by storage hardware damage; the RA ID mechanism is limited to the interconnected, usually in one The storage medium in the data center creates data redundancy; the RAI D protection scope has limitations. The traditional RAI D technology is usually deployed on the storage medium of the cloud storage data center, and can only protect the data loss caused by the physical damage of the storage medium. It can be seen that the RA ID mechanism is mainly used to prevent data loss caused by damage of storage hardware levels such as hard disks, tapes, and the like.

In order to protect against the short-term inaccessibility of data storage services caused by disasters caused by non-storage hardware-level disasters, user data loss and the risk of being modified, it is necessary to create a new public cloud storage service availability and data. The method of fault tolerance.

Summary of the invention

The object of the present invention is to provide a method, device and system for data access of cloud storage, which are to solve the problem that the existing cloud storage method cannot effectively resist the data caused by the short-term unavailability of the cloud storage service caused by the disaster of the non-storage hardware level. Temporarily unable to retrieve, data loss and the risk of being altered by the tomb.

The invention provides a method for data access of cloud storage, the method comprising data storage and data retrieval,

The storing step of the data includes:

Creating a data check value of the original data to be stored at the sending end of the data; the same cloud storage data center storage;

The step of retrieving the data includes:

Accessing the cloud storage data center storing the original data according to the preset redundant storage policy according to the access request;

Determining whether the cloud storage data center storing the original data is available, if not available, Retrieving data check values of the original data and data used to create data check values of the original data from other cloud storage data centers;

The original data is restored based on the data check value of the original data and the data used to create the data check value of the original data.

The present invention provides an apparatus for data access of a cloud storage, the apparatus comprising a storage module of data and a retrieval module of data, the storage module of the data comprising:

a data check value creating unit, configured to create a data check value of the original data to be stored at the sending end of the data;

a data transmission unit, configured to separately transmit the data verification values created by the original data and the data verification value creation unit to different cloud storage data center storage according to a preset redundancy storage policy;

The data retrieval module includes:

The access judging unit is configured to access the cloud storage data center storing the original data according to the preset redundancy storage policy according to the access request, and determine whether the original data stored in the cloud storage data center is available, and send the determination result;

a data retrieval unit, configured to retrieve, according to the determination result that the original data stored in the cloud storage data center is not available, by the access determination unit, to retrieve the data verification value of the original data from other cloud storage data centers and create The data used by the data check value of the original data; the check value and the data used to create the data check value of the original data restore the original data.

The present invention also provides a system for data access of a cloud storage, the system comprising a data storage module, a data retrieval module, and a cloud storage data center, the data storage module comprising:

a data transmission unit, configured to convert the original data and data according to a preset redundant storage policy The data check values created by the check value creation unit are respectively transmitted to different cloud storage data center storages;

The data retrieval module includes:

The invention creates a data check value of the original data to be stored by using a logical operation on the data sending end, and stores the original data to be stored and the data check value according to a preset redundant storage policy in different cloud storage data centers, in the cloud. Storage data center level creates redundancy of data, greatly increasing the availability and fault tolerance of cloud storage data.

DRAWINGS

1 is a flowchart of a method for data access of a cloud storage according to an embodiment of the present invention; FIG. 2 is a schematic diagram of a data redundancy storage method according to an embodiment of the present invention; FIG. 3 is another data provided by an embodiment of the present invention. FIG. 4 is a schematic structural diagram of a device for data access of a cloud storage according to an embodiment of the present invention; and FIG. 5 is a schematic structural diagram of a system for data access of a cloud storage according to an embodiment of the present invention. detailed description

In order to further understand the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Bright.

In the embodiment of the present invention, the data check value of the original data to be stored is created on the data sending end, and the original data and the data check value to be stored are respectively stored in different cloud storage data centers according to the preset redundant storage policies. Store data center level to create redundancy of data.

Referring to FIG. 1, an embodiment of the present invention provides a method for data access of a cloud storage, where the method includes a data storage step S1 00 and a data retrieval step S200;

The data storage step S 1 00 specifically includes:

Step S 1 01: creating a data check value of the original data to be stored at the data sending end; in actual applications, the original data may include part of data of the file or the file; the number of the number of data can be stored in the cloud by the number of data centers Divisible, if it cannot be divisible, it needs to be complemented by other data; the data check value of the original data can be, but is not limited to, created by XOR logical operation, and correspondingly, the original data is restored by XOR logical operation; The data used to store the data check value of the original data includes the original data to be stored and at least one fixed-length binary number, for example: X-OR logical operation can be created using the original data and a binary number 1 001 having 4 bits Data check value, or use the original data to perform XOR logical operations with two binary digits of 1 001 00, 001 000 with 6 bits to create a data check value, etc. In addition, the user can create one or more according to actual needs. Data check value of raw data; input to different cloud storage data centers Storage;

The original data and the data check value are respectively stored in different cloud storage data centers. Specifically, the user can formulate a corresponding redundancy storage policy according to his actual needs, and the original data and the data check value are determined according to the redundancy. Store the policy to store; the created data can be stored in the same cloud storage data center or stored in different cloud storage data centers; usually, the user will store the created data check values in different Cloud storage data center to prevent two or more cloud storage services from being unavailable at the same time;

There are many redundant storage strategies for raw data and its checksums. Here are two examples: 1. Store the checksum values of all the original data to be stored in a cloud storage data center. As shown in Figure 2, if there are four different cloud storage data centers available, store the first part of the original data D1 in the cloud storage. In the cloud storage data center of service provider 1, and so on, the second part of the original data D2 and the third part of the original data D3 are respectively stored in the cloud storage data center of the cloud storage service provider 2, 3, and finally all the original data The check value P1 is stored in the cloud storage data center of the cloud storage service provider 4; similarly, other original data and corresponding data check values are sequentially stored in the above four cloud storage data centers according to the above method;

2. Store all the original data and the corresponding data check values in each cloud storage data center. As shown in Figure 3, if there are four different cloud storage data centers available, the first part of the original data will be The D1 is stored in the cloud storage data center of the cloud storage service provider 1, and the second part of the original data D2 and the third part of the original data D3 are respectively stored in the cloud storage data center of the cloud storage service provider 2, 3, and finally The check value P1 of all the original data is stored in the cloud storage data center of the cloud storage service provider 4; the original data D4, D5, D6 and their check values P2 are respectively stored in the cloud storage data center 1, 2. 4 and 3; similarly, the original data D7, D8, D9 and their check value P3 are respectively stored in the cloud storage data centers 1, 3, 4 and 2; similarly, other original data and corresponding data are corrected. The verification values are sequentially stored in the above four cloud storage data centers according to the above method;

The above only cites two original data and data check value redundancy storage strategies. In the real implementation, users can adopt other redundant data storage strategies according to their actual needs. For example, the original data and data check values are stored alternately.

The data retrieval step S200 specifically includes:

Step S201: accessing the cloud storage data center storing the original data according to the preset redundant storage policy according to the access request;

Step S202: determining whether the original data stored in the cloud storage data center is available, if not, executing step S203, if yes, executing step S205;

Step S203: Retrieving data check values of the original data and data used to create data check values of the original data from other cloud storage data centers; Step S204: Restore the original data according to the data check value and the data used to create the data check value of the original data;

Step S205: Retrieving the original data from the cloud storage data center;

Corresponding to the calculation method used to create the data check value of the original data, in this embodiment, when the original data of the cloud storage data center is unavailable according to the data check value, and the original data cannot be retrieved, the exclusive OR logic is also used. The operation method restores the original data;

The following example shows that if the data redundancy storage strategy adopts the second method described above, refer to Figure 3, assuming that the original data of the cloud storage data center of the second cloud storage service provider is unavailable, it is now necessary to store data from the other three clouds. The other raw data and data check values retrieved by the center are used to restore the original data stored in the second cloud storage data center, for example, the original data D2 needs to be restored, then the original data D1, D3 retrieved from other cloud storage data centers. And the check value P1 of the original data D2, the original data D2 is obtained according to an exclusive OR algorithm, namely: D2 = D1 θ D 3 θ P 1

Similarly, other raw data originally stored in the second cloud storage data center can be restored.

Referring to FIG. 4, an embodiment of the present invention provides an apparatus for data access of a cloud storage, where the apparatus includes: a data storage module 20 and a data retrieval module 30;

The data storage module 20 includes a data check value creation unit 21 and a data transfer unit 22; the data retrieval module 30 includes an access determination unit 33, a data retrieval unit 31, and a data restoration unit 32;

When the data is stored, the data check value creating unit 21 creates the data check value of the original data to be stored at the transmitting end of the data, and the data transfer unit 22 sets the original data and the data check value according to the preset redundant storage policy. The data check values created by the creating unit 21 are respectively transmitted to different cloud storage data centers for storage;

When there is a raw data access request, the access judging unit 33 accesses the cloud storage data center storing the original data according to the preset redundant storage policy according to the access request, and determines whether the original data stored in the cloud storage data center is available; when the cloud storage When the original data of the data center is unavailable, the data retrieval unit 31 retrieves the data of the original data from other cloud storage data centers. The value used and the data used to create the data check value of the original data; the data restore unit 32 uses the data check value of the original data retrieved by the data retrieval unit 31 and the data check value used to create the original data. The data restores the raw data.

Referring to FIG. 5, an embodiment of the present invention further provides a system for data access of a cloud storage, where the system includes a data storage module 20, a data retrieval module 30, and a cloud storage data center 40. The data storage module 20 includes The data check value creation unit 21 and the data transfer unit 22; the data retrieval module 30 includes an access determination unit 33, a data retrieval unit 31, and a data restoration unit 32;

When the data is stored, the data check value creating unit 21 creates the data check value of the original data to be stored at the transmitting end of the data, and the data transfer unit 22 sets the original data and the data check value according to the preset redundant storage policy. The data check values created by the creating unit 21 are respectively transmitted to different cloud storage data centers 40 for storage;

When there is a raw data access request, the access judging unit 33 accesses the cloud storage data center 40 storing the original data according to the preset redundant storage policy according to the access request, and determines whether the original data stored in the cloud storage data center 40 is available; When the original data of the cloud storage data center 40 is unavailable, the data retrieval unit 31 retrieves the data check value of the original data and the data used to create the data check value of the original data from the other cloud storage data center 40; The restoration unit 32 restores the original data based on the data check value of the original data retrieved by the data retrieval unit 31 and the data used to create the data check value of the original data.

In order to illustrate the feasibility of the present invention, a typical example of data check value generation is shown below. This example mainly generates data check values by data bit XOR (X0R) logic operations.

Note a ten b is the exclusive OR of a and b, a, = 0 or 1. The XOR operation follows the following rules:

1. If a and b are the same (that is, both are 0 or the same is 1), then a e b=0; otherwise a θ b=l ;

2. a θ b θ c = a θ (b θ c) = (a θ b) θ c;

3, d = a θ b θ c can be launched a = d @ b @ c When using a public cloud storage service, the data that the user stores in the third-party cloud storage data center, whether it is a file or a data block, physically appears as a string of 0, 1 values.

Assume that there are three existing segments of data to be backed up to the specified cloud storage data center, Dl, D2.

D3, if Dl, D2, and D3 are not equal in length, they need to be equal to 0 to make them equal. There are four different cloud storage data centers available.

Before being transmitted to the designated cloud storage data center, an exclusive OR logical operation is performed between the original data according to the number of available cloud storage data centers, thereby generating a data check value P1 of the original data to be stored.

Data check value PI = D1 θ D2 θ D3.

The above method is recycled until all data check values of the original data to be stored are generated. If the number of the original data to be stored and the corresponding data check value cannot be divisible by the number of the specified cloud storage data center, other data may be used or a certain length of 0, 1 string may be used to complement the generated data to be stored. The data check value of the original data.

The method for generating the data check value is mature. The above is only a typical data check value generation method. The user can also generate two or more data check values as needed and store them in different cloud storage data centers. Prevent raw data from two or more cloud storage data centers from being unavailable at the same time.

After the checksum values of all the original data to be stored are generated, the original data to be stored and the data check value thereof are redundantly stored in different cloud storage data centers according to the number of available cloud storage data centers, thereby achieving resistance. The raw data of any cloud storage data center is not available or the risk of data loss or tomb modification.

When the raw data of the cloud storage data center is temporarily unavailable or unavailable for a long time, the user can use the data check value of the original data to assist in restoring the original data.

The above description only saves a data check value. In the specific implementation, the user can ensure the storage of two or more cloud storage data centers by saving two or more data check values. When the initial data is not available, the original data can still be restored. Just saving multiple copies of the data check value will increase the need for storage space, which will increase the cost. Because the data restoration method is similar, it will not be described again.

The embodiment of the present invention creates the check value of the original data to be stored on the data sending end, and stores the original data to be stored and the check value thereof in different cloud storage data centers according to preset policies, in different cloud storages. Redundancy of data is created between data centers, thereby realizing risks and improving the availability of public cloud storage service data and the fault tolerance of data.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. Within the scope.

Claims

Claim

A method for data access of a cloud storage, the method comprising storing data and retrieving data, wherein:

The storing step of the data includes:

The step of retrieving the data includes:

Determining whether the original data stored in the cloud storage data center is available, and if not available, retrieving the data check value of the original data from other cloud storage data centers and the data check value used to create the original data Data

The method for data access of the cloud storage according to claim 1, wherein the data check value for creating the original data to be stored is a data check value for creating original data by an exclusive OR logical operation; The restoring the original data is to restore the original data by an exclusive OR logical operation.

3. The method of data access for cloud storage according to claim 1, wherein the number is divisible by the number of the cloud storage data centers.

4. The method for data access of a cloud storage according to claim 1, wherein the data used to create the data check value of the original data comprises the original data and at least one fixed length binary. number.

The method for data access of the cloud storage according to claim 1, wherein the data check value is stored in the same cloud storage data center or in a different cloud storage data center.

6. A device for data access of a cloud storage, the device comprising a data storage module and a data retrieval module, wherein

The storage module of the data includes:

The data retrieval module includes:

7. A system for data access of a cloud storage, the system comprising a data storage module, a data retrieval module, and a cloud storage data center, wherein

The storage module of the data includes:

The data retrieval module includes: The access judging unit is configured to access the cloud storage data center storing the original data according to the preset redundancy storage policy according to the access request, and determine whether the original data stored in the cloud storage data center is available, and send the determination result;