WO2017063495A1

WO2017063495A1 - Data migration method and apparatus

Info

Publication number: WO2017063495A1
Application number: PCT/CN2016/099934
Authority: WO
Inventors: 张翼; 秦长鹏; 戴庆军; 舒坦; 刘卯银; 牛克强
Original assignee: 中兴通讯股份有限公司
Priority date: 2015-10-12
Filing date: 2016-09-23
Publication date: 2017-04-20
Also published as: CN106569732A; CN106569732B

Abstract

A data migration method and apparatus, for use in resolving the problem of lowered internal performance of a system due to the great quantity of times of reading hard disks in a conventional data migration process in tiered storage. The data migration method comprises: a high-speed buffering memory Cache module acquires, from a tiered storage module, an address space of to-be-migrated hotspot data; the Cache module writes the to-be-migrated hotspot data according to the address space; and the Cache module notifies the tiered storage module of the address space of the to-be-migrated hotspot data in the Cache module. The method improves performance of a storage system.

Description

Data migration method and device

Technical field

The present invention relates to the field of communications, and in particular, to a data migration method and apparatus.

Background technique

With the advancement of hard disk technology, the types and number of storage media supported by storage systems have increased significantly. Storage media supported by the storage system include: SSD (Solid State Drive), SAS (Serial Attached SCSI) hard disk, and NL (Near Line) SAS hard disk. Storage resource (SSD, SAS, NL_SAS) provisioning is a highly repetitive and time-consuming task. It is difficult to balance the performance requirements and capacity requirements of the storage system through the previous planning, even if it is reached initially. The balance, performance and capacity requirements will also change. The storage system's resource allocation needs constant adjustment. Therefore, the industry storage system generally supports the tiered storage function. The purpose of this function is to intelligently manage data through the active level of statistical data. The storage location improves the overall performance of the storage system and reduces the user cost. The user cost usually includes the cost of equipment procurement, space, energy consumption and management. The automatic allocation of resources through tiered storage can greatly reduce the difficulty of the previous planning, and the later maintenance and expansion becomes easier.

At present, the implementation of tiered storage in the industry is basically carried out through three stages of statistics, analysis, and migration, that is, by periodically counting the frequency of access to a certain area of data, periodically analyzing whether it is hot data, and then through data migration. The means to move from one data space to another. The migration method generally involves the storage system relocating data through the background program without affecting the host IO. For example, the data to be migrated is read from the hard disk and then written to the destination hard disk address to be migrated. Specific to the storage system, by The tiered storage is located under the Cache. The traditional process is shown in Figure 1. The hotspot migration of the tiered storage is the read and write operations of the tiered storage module on the hotspot area of the hard disk. The tiered storage module collects the hotspot area of the hard disk, and the migration process periodically performs data relocation on the hotspot area. In general, the relocation process first reads a piece of data from the disk that needs to be relocated, and the read data is stored through temporary memory resources, and then The tiered storage module writes the data to the destination hard disk address space that needs to be migrated, and releases the temporary memory resources after writing, so that a normal relocation operation is completed, and the relocation process involves one read of the disc and one memory resource application release. And write the disk once. At the same time, the granularity of the hotspot statistics area in the storage system does not match the granularity of the one-time relocation. As shown in Figure 2, the granularity of the hotspot area (usually the upper logical volume address to the hard disk address mapping granularity) is greater than the granularity of one relocation. The logical volume address to hard disk address mapping can be modified only after all data in the hotspot area has been relocated. In this way, when the data of the hotspot area has not been relocated, if there is just a write IO access to the part of the data that has been relocated, the tiered storage processing of the write IO needs to write two copies of the data at the same time, and one copy is written to the original old hard disk. The area needs to be written to the new hard disk area. The purpose of this is to ensure data consistency. This process is the double-write process of the hierarchical storage module. The overview is the traditional data storage process of hierarchical storage. It can be seen that the traditional tiered storage data migration process has more read times on the hard disk, which reduces the internal performance of the storage system.

Summary of the invention

The embodiment of the invention provides a data migration method and device, which are used to solve the problem that the number of hard disk readings is large and the internal performance of the system is reduced during the traditional hierarchical storage data migration process.

According to a first aspect of the present invention, a data migration method is provided, including: a cache cache module obtains an address space of a hotspot data to be migrated from a hierarchical storage module; and a cache module writes a hot spot to be migrated according to the address space. The Cache module notifies the tiered storage module of the address data to be migrated in the address space of the Cache module.

The Cache module writes the hotspot data to be migrated according to the address space, including: when not The Cache module sends a pre-read request to the tiered storage module in the Cache module, and the Cache module receives the hotspot data to be migrated sent by the tiered storage module, and writes the received hotspot data. Into the Cache module.

According to a second aspect of the embodiments of the present invention, a data migration method includes: a hierarchical storage module notifying an address space of a hotspot data to be migrated to a cache cache module; and the hierarchical storage module knowing that the data to be migrated is in the cache module After the address space, the hotspot data is written to the address space of the new medium; the hierarchical storage module modifies the correspondence between the logical volume and the medium mapping table according to the address space of the new medium.

After the tiered storage module learns the address space in the Cache module, the tiered storage module writes the hotspot data to the address space of the new medium, including: when receiving the hotspot data access request forwarded by the Cache module, according to the hotspot data The migration progress determines whether the data to be read is the migrated data, and the judgment result is obtained; the data is read according to the judgment result, and after the data is read, the Cache module is notified that the data is read.

Further, the method further includes: before the Cache module of the hotspot data to be migrated is notified to the Cache module, the tiered storage module performs hotspot data statistics according to the hierarchical storage statistical granularity of the hard disk address space of the logical volume LUN; the tiered storage module The spatial address of the hotspot data to be migrated is analyzed according to the statistical data.

According to a third aspect of the embodiments of the present invention, a data migration apparatus is provided, including: an learning unit, configured to learn, by using a cache cache module, an address space of a hotspot data to be migrated from a hierarchical storage module; The unit is configured to write the hotspot data to be migrated according to the address space by using the Cache module, and the notification unit is configured to notify the tiered storage module of the address space in the Cache module by the Cache module.

The first write unit includes: a pre-read sub-unit, configured to send a pre-read request to the hierarchical storage module when the address space of the hot-spot data to be migrated is not found in the Cache module; Write to the subunit, set to receive the rating through the Cache module The hotspot data to be migrated sent by the storage module, and the received hotspot data is written into the Cache module.

According to a fourth aspect of the present invention, a data migration apparatus includes: a notification unit configured to notify a cache cache module of an address space of hotspot data to be migrated by a hierarchical storage module; and a second write unit After the aging storage module learns the address space of the data to be migrated in the Cache module, the hotspot data is written to the address space of the new medium; the modifying unit is set to the tiered storage module to modify the logic according to the address space of the new medium. The correspondence between the volume and the media mapping table.

The second writing unit includes: a determining subunit configured to, when receiving the hotspot data access request forwarded by the Cache module, determine whether the data to be read is the migrated data according to the migration progress of the hotspot data, The judgment result; the notification subunit is set to read data according to the judgment result, and after the data is read, the Cache module is notified that the data reading is completed.

Further, the device further includes: a statistic unit, configured to: before the Cache module of the address space of the hotspot data to be migrated, the tiered storage module performs hotspot data statistics according to the hierarchical storage statistical granularity of the hard disk address space of the logical volume LUN. The analysis unit is configured to analyze the spatial address of the hotspot data to be migrated according to the statistical data.

The beneficial effects of the present invention are as follows:

The solution provided by the embodiment of the present invention can reduce the disk read operation and the memory usage during the data migration process. The disk disk migration process involves only one write disk operation, which improves the internal performance of the storage system.

DRAWINGS

1 is a flow chart of hotspot data migration of hierarchical storage in the prior art;

2 is a double-write flowchart in a hot-spot data migration process of hierarchical storage in the prior art;

3 is a flowchart of a data migration method according to Embodiment 1 of the present invention;

4 is a basic schematic diagram of a Cache in Embodiment 1 of the present invention;

5 is a flowchart of a data migration method according to Embodiment 2 of the present invention;

6 is a flowchart of data migration in a case where all migrated data is hit in a Cache in Embodiment 3 of the present invention;

FIG. 7 is a flowchart of data migration in the case where the Cache misses all migration data in Embodiment 3 of the present invention; FIG.

8 is a flowchart of data migration in the case where there is a write IO access migration area in a data migration process according to Embodiment 3 of the present invention;

9 is a flowchart of data migration in the case where there is a read IO access migration area in a data migration process according to Embodiment 3 of the present invention;

10 is a structural block diagram of a data migration apparatus according to Embodiment 4 of the present invention;

Figure 11 is a block diagram showing the structure of a data migration apparatus according to Embodiment 5 of the present invention.

detailed description

The present invention provides a data migration method and apparatus for solving the problem that the hard disk is read frequently in the traditional hierarchical storage data migration process, and the internal performance of the system is reduced. The present invention further proceeds to the present invention with reference to the accompanying drawings and embodiments. Detailed description. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Example 1

The present embodiment provides a data migration method, which is applied to a Cache module, which is implemented by a Cahche module. FIG. 3 is a flowchart of a data migration method according to Embodiment 1 of the present invention, as shown in FIG. The method comprises the following steps:

Step 301: The Cache module learns, from the tiered storage module, an address space of the hotspot data to be migrated.

Step 302: The Cache module writes the hotspot data to be migrated according to the address space.

Step 303: The Cache module notifies the hierarchical storage module of the address space in the Cache module.

The step of the Cache module writing the hotspot data to be migrated according to the address space may include:

The Cache module sends a pre-read request to the tiered storage module when the address space of the hotspot data to be migrated is not found in the Cache module. The Cache module receives the hotspot data to be migrated sent by the tiered storage module, and receives the received hotspot data. Hotspot data is written to the Cache module.

As shown in Figure 4, the data space LUN accessed by the host is represented as a tree in the storage system Cache. The function of the Cache is to manage the tree. The access of the host IO is actually the access of the tree. Hit the tree, read ahead, etc. At the same time, Cache also performs operations such as eliminating and dirty data on the tree. Thereby, the feedback of the Cache to the host IO is realized. In addition, once the host IO reaches the storage system Cache phase, it will reside in the Cache for a period of time, thereby achieving the cache hit effect, which is also the meaning of the Cache existence.

In this embodiment, the migration operation of the original hierarchical storage module is performed by the cache module, and is decomposed into the cache module, which is a brush operation, and the flash operation of the cache module triggers the single write of the hierarchical storage migration module to achieve data migration. effect. Based on this, the cache module can use its own existing cache resources. If it hits, it can reduce the disk read operation and a temporary memory occupation. The data in the Cache module is directly brushed to the hierarchical storage module, and the data is written through the hierarchical storage module. To the destination address, only the data is written to the destination address at this time without writing to the original media address. Therefore, reduce one write operation. If it does not hit, the Cache module needs to pre-read the data to the tiered storage module, and then write the data to the destination address by brushing the disk. At this time, like the traditional process, it needs to read the disk once, but the use of temporary memory. Still has the advantage, at the same time, the pre-read page increases the Cache hit rate. In this embodiment, the cooperation between the hierarchical storage module and the cache module completes the entire process of data migration, and uses the cache disk operation to replace the hierarchical storage migration operation, thereby reducing the one-time disk reading and memory occupation of the migration process.

Example 2

The present embodiment provides a data migration method, which is applied to the hierarchical storage module side, and is implemented by a hierarchical storage module. FIG. 5 is a flowchart of a data migration method according to Embodiment 2 of the present invention, as shown in FIG. The method comprises the following steps:

Step 501: The tiered storage module notifies the Cache module of the address space of the hotspot data to be migrated;

Step 502: After the tiered storage module knows the address space of the data to be migrated in the Cache module, the hotspot data is written to the address space of the new medium.

Step 503: The hierarchical storage module modifies a correspondence between the logical volume and the medium mapping table according to the address space of the new medium.

After the tiered storage module learns the address space of the data to be migrated in the Cache module, the tiered storage module writes the hotspot data to the address space of the new media, which may include:

When receiving the hotspot data access request forwarded by the Cache module, determining, according to the migration progress of the hotspot data, whether the data requested to be requested is the migrated data, and obtaining the judgment result; reading the data according to the judgment result, after the data is read, The Cache module is notified that the data has been read.

Further, the foregoing method may further include:

Before the Cache module of the hotspot data to be migrated is sent to the Cache module, the tiered storage module performs hotspot data statistics according to the hierarchical storage statistical granularity of the hard disk address space where the logical volume LUN is located; the tiered storage module analyzes the hotspot to be migrated according to the statistical data. The spatial address of the data.

In this embodiment, the statistics of the hotspots in the tiered storage are the access frequencies of a certain address space on the disk, and in the storage system, the Cache perspective is mainly the Cache brush dirty data for the disk and the Cache transparent write access to the disk. frequency. At this time, if a certain address space on the disk is counted as hot data, it is caused by the Cache brushing the address space, or the host frequently writes the address space of the area. At this time, the Cache may still reside in these Hot data. Once the hotspot data is not eliminated in the Cache, then the hotspot data can be written to the new performance hard disk area through the hierarchical storage function module. However, if the hotspot data is not in the Cache at this time, the Cache is required to pre-read the data that is not hit by the tiered storage module, and then write the new performance hard disk area through the tiered storage function module, and if When the storage system is in the hot spot caused by the write-through state, the data from the pre-reading just meets the cache hit and improves the performance of the system.

The tiered storage module receives the hotspot data swiped under the Cache, and needs to write the Cache data to the new media address area according to the migration progress indicator. Because the hotspot statistics need to be larger than the migration granularity, the hierarchical storage is a mapping table that cannot modify the logical volume address to the disk address before the hotspot area data is migrated. The logical volume to disk address is modified only after the hotspot data is migrated. Mapping table. In this way, when the next IO accesses the logical volume area, the area of the new medium is accessed. For this reason, before the hot data is migrated, if there is a read request to access the hotspot area, if the read request data is in the Cache, the direct access Cache can be returned if the request data is not in the Cache, or partially in the Cache. The Cache is required to send a pre-read request to the tiered storage module. The tiered storage module determines whether to read the old media data or read the new media data by migrating the bitmap information, thereby achieving the purpose of data access.

Example 3

In this embodiment, the Cache and the grading are performed by the migration of the data cache hit, the migration when the migration data cache misses, the write IO access migration area during the migration process, and the read IO access migration area during the migration process. The process of data migration between storage modules is described as follows:

During the migration of data, the data migration process in the case of hitting all migrated data in the Cache:

As shown in FIG. 6, the following specifically includes the following processing:

The hierarchical storage function module stores the hierarchical hard disk address space where the LUN is located according to the hierarchical storage system. Calculate hotspot data statistics by granularity;

The hierarchical storage function module analyzes which address areas are hotspot areas according to the granularity statistics;

The tiered storage function module notifies the Cache module which addresses need to be hotspot migration;

The Cache function module performs a data write operation according to the address space provided by the hierarchical storage. At this time, the write disk address area hits the Cache exactly;

Cache the address space requested by the hierarchical storage module;

The tiered storage writes data to the address space of the new medium according to the write request sent by the Cache;

The tiered storage module modifies the logical volume to media map to the new media address space.

During the migration of data, the data migration process in the case of Cache miss all migration data:

As shown in FIG. 7, the following specifically includes the following processing:

The tiered storage function module performs hotspot data statistics on the hierarchical hard disk address space where the LUN is located according to the hierarchical storage statistical granularity;

The tiered storage function module notifies the Cache which addresses need to be hotspot migration;

The Cache function module performs a data write operation according to the address space provided by the hierarchical storage. At this time, the write disk address area is not in the Cache or partially in the Cache;

The Cache sends a prefetch request to the tiered storage module, and the tiered storage module prefetches the data and returns the Cache;

The Cache continues to deliver the address space required by the tiered storage to the tiered storage module;

During the process of migrating data, there is a case of writing IO access to the migration area during the data migration process. Data migration process:

As shown in FIG. 8, the following specifically includes the following processing:

The Cache function module performs a data write operation according to the address space provided by the hierarchical storage, and at this time, the write disk address area is in the Cache;

The tiered storage module modifies the logical volume to the media mapping table to a new media address space;

If there is a write request to access the hot spot data at this time, at this time, request to access the Cache page, just hit, return directly to the host.

During the data migration process, there is a data migration process in the case of reading the IO access migration area during the data migration process:

As shown in FIG. 9, the following specifically includes the following processing:

The tiered storage writes data to the address space of the new medium according to the write request sent by the Cache. But only part of the data was written, and the entire hotspot area data was not written;

At this point, there is a read request to access the hotspot data, and the area data has been migrated;

If the cache hits, the Cache returns directly. If it does not hit, the Cache sends a request to the tiered storage module.

The tiered storage module determines, according to the migration progress flag, whether the read request needs to read the new data that has been migrated or the old data, and if the migrated data has been written, it is new data; otherwise, the old data;

The tiered storage reads the data to the Cache, and the Cache returns the data to the host to read the request;

Example 4

The embodiment of the present invention provides a data migration device, which is disposed on the Cache side. FIG. 10 is a structural block diagram of a data migration device according to Embodiment 4 of the present invention. As shown in FIG. 10, the device 100 includes the following components:

The learning unit 101 is configured to learn, by using the Cache module, an address space of the hotspot data to be migrated from the hierarchical storage module;

The first writing unit 102 is configured to write the hotspot data to be migrated according to the address space by using the Cache module;

The notification unit 103 is configured to notify, by using the Cache module, the hotspot data to be migrated in the address space in the Cache module to the hierarchical storage module.

The first writing unit 102 may specifically include: a pre-reading sub-unit, configured to send a pre-read to the hierarchical storage module when the address space of the hot-spot data to be migrated is not found in the Cache module. The request is written into the sub-unit, and is configured to receive the hot-spot data to be migrated sent by the hierarchical storage module through the Cache module, and write the received hot-spot data into the Cache module.

Example 5

The embodiment of the present invention provides a data migration device, which is disposed on the side of the hierarchical storage module. FIG. 11 is a structural block diagram of the data migration device according to Embodiment 5 of the present invention. As shown in FIG. component:

The notification unit 111 is configured to notify the Cache module of the address space of the hotspot data to be migrated through the hierarchical storage module;

The second writing unit 112 is configured to write the hotspot data to the address space of the new medium after the address storage space in the Cache module is obtained by the tiered storage module;

The modifying unit 113 is configured to modify the correspondence between the logical volume and the media mapping table according to the address space of the new medium.

The second writing unit 112 may specifically include: a determining subunit, configured to determine, according to the migration progress of the hotspot data, whether the data to be read is the migrated data according to the migration progress of the hotspot data when receiving the hotspot data access request forwarded by the Cache module. The judgment result is obtained; the notification subunit is set to read the data according to the judgment result, and after the data is read, the Cache module is notified that the data is read.

Further, the device 110 may further include: a statistic unit, configured to: before the Cache module of the address space of the hotspot data to be migrated, the tiered storage module performs a hotspot according to the hierarchical storage statistical granularity of the hard disk address space of the logical volume LUN. Data statistics; the analysis unit is configured to analyze the spatial address of the hotspot data to be migrated according to the statistical data.

The solution provided by the embodiment of the present invention reduces the disk read operation and the memory usage during the data migration process. The disk migration process involves only one write operation, and the traditional migration method requires double write when the disk is saved. Data consistency. At the same time, compared with the traditional migration method, in the scenario that the Cache does not hit the migration data in the scenario of the Cache, the pre-read data can improve the system hit rate, thereby improving the system performance to a certain extent.

While the preferred embodiments of the present invention have been disclosed for purposes of illustration, those skilled in the art will recognize that various modifications, additions and substitutions are possible, and the scope of the invention should not be limited to the embodiments described above.

Industrial applicability

As described above, the data migration method and apparatus provided by the embodiments of the present invention have the following beneficial effects: in the process of data migration, one disk read operation and one memory occupation can be reduced, and only one write operation is involved in the process of brush disk migration. Improves the internal performance of the storage system.

Claims

A data migration method, including:

The cache Cache module learns the address space of the hotspot data to be migrated from the hierarchical storage module;

The Cache module writes the hotspot data to be migrated according to the address space;

The Cache module notifies the hierarchical storage module of the address space in the Cache module.
The method of claim 1, wherein the Cache module writes the hotspot data to be migrated according to the address space, including:

The Cache module sends a pre-read request to the hierarchical storage module when the address space of the hot-spot data to be migrated is not found in the Cache module;

The Cache module receives the hotspot data to be migrated sent by the tiered storage module, and writes the received hotspot data to the Cache module.
A data migration method, including:

The tiered storage module notifies the cache Cache module of the address space of the hotspot data to be migrated;

After the tiered storage module learns that the data to be migrated is in the address space in the Cache module, the hotspot data is written to the address space of the new medium;

The hierarchical storage module modifies a correspondence between the logical volume and the medium mapping table according to the address space of the new medium.
The method of claim 3, wherein the tiered storage module writes the hotspot data to an address space of a new medium after learning that the data to be migrated is in an address space in the Cache module, including:

When receiving the hotspot data access request forwarded by the Cache module, determining, according to the migration progress of the hotspot data, whether the data that the request needs to read is the migrated data, and determining result;

The data is read according to the judgment result, and after the data is read, the Cache module is notified that the data is read.
The method of claim 3 or 4, wherein the method further comprises:

Before the Cache module of the hotspot data to be migrated is notified to the Cache module, the tiered storage module performs hotspot data statistics according to the hierarchical storage statistical granularity of the hard disk address space of the logical volume LUN.

The tiered storage module analyzes the spatial address of the hotspot data to be migrated according to the statistical data.
A data migration device comprising:

The learning unit is configured to learn, by using the cache Cache module, an address space of the hotspot data to be migrated from the hierarchical storage module;

a first writing unit, configured to write, by using the Cache module, the hotspot data to be migrated according to the address space;

The notification unit is configured to notify, by the Cache module, the hotspot data to be migrated in the address space in the Cache module to the hierarchical storage module.
The apparatus of claim 6, wherein the first writing unit comprises:

The pre-reading sub-unit is configured to send a pre-read request to the hierarchical storage module when the address space of the hot-spot data to be migrated is not found in the Cache module;

The write subunit is configured to receive the hot spot data to be migrated sent by the hierarchical storage module by using the Cache module, and write the received hot spot data into the Cache module.
A data migration device comprising:

The notification unit is configured to notify the cache Cache module of the address space of the hotspot data to be migrated through the hierarchical storage module;

a second writing unit, configured to: write, by the hierarchical storage module, the hot spot data to an address space of the new medium after learning that the data to be migrated is in an address space in the Cache module;

And a modifying unit, configured to modify, by the hierarchical storage module, a correspondence between the logical volume and the media mapping table according to the address space of the new medium.
The apparatus of claim 8, wherein the second writing unit comprises:

The determining subunit is configured to: when receiving the hotspot data access request forwarded by the Cache module, determine, according to the migration progress of the hotspot data, whether the data that the request needs to read is migrated data, and obtain a determination result;

The notification subunit is configured to read data according to the judgment result, and after the data is read, notify the Cache module that the data reading is completed.
The device according to claim 8 or 9, wherein the device further comprises:

The statistic unit is configured to perform hotspot data statistics on the hard disk address space of the hierarchical level of the logical volume LUN according to the hierarchical storage statistical granularity before the Cache module of the hotspot data to be migrated is notified to the Cache module;

The analyzing unit is configured to analyze, according to the statistical data, a spatial address of the hotspot data to be migrated.