WO2013097119A1

WO2013097119A1 - Method and device for realizing multilevel storage in file system

Info

Publication number: WO2013097119A1
Application number: PCT/CN2011/084830
Authority: WO
Inventors: 陆海翔
Original assignee: 华为技术有限公司
Priority date: 2011-12-28
Filing date: 2011-12-28
Publication date: 2013-07-04
Also published as: CN103548004A; CN103548004B

Abstract

Provided is a method for realizing multilevel storage in a file system. In a file system, a primary storage device and a secondary storage device are used to store files respectively to acquire the value of each file stored in the primary storage device and the secondary storage device; a first file value rank list is established and managed according to the order of the value of each file stored in the primary storage device, and a second file value rank list is established and managed according to the order of the value of each file stored in the secondary storage device; and file transfer is performed in the primary storage device and the secondary storage device according to a preset transfer policy in accordance with the first file value rank list and the second file value rank list. The embodiments of the present invention can store files with high value in a storage medium with higher performance and store files with low value in a slow storage medium, thus making the best of the overall performance of the system.

Description

Method and apparatus for implementing multi-level storage in a file system

Technical field

The present invention relates to the field of electronic technologies, and in particular, to a method and apparatus for implementing multi-level storage in a file system.

Background of the invention

With the development of storage technology, new storage media and storage interfaces are emerging in addition to traditional storage media such as magnetic tapes and mechanical disks. These new storage media and storage interfaces usually mean higher transmission rates, among them, SSD ( Solid State Disk, a solid state drive, is a typical technology.

SSDs, also known as electronic hard disks or solid state electronic disks, consist of a control unit and a solid-state storage unit. The functions, forms, and usage of the interface are the same as those of a magnetic media hard disk. At present, most SSDs are used in notebook computers. In order to match existing notebooks, most of the products have the same size as traditional notebook hard drives, and there are other sizes of products. Since there is no mechanical structure inside the SSD, the reading does not use the motor to rotate, so there is an advantage in shock resistance and noise. In addition, the SSD chip can work in a wide temperature range (-40 to 85 ° C). Therefore, SSD is widely used in military, electric power, medical, aviation, vehicle, industrial control, video surveillance, network monitoring, network terminals, navigation equipment and other fields.

With the maturity of SSD technology, users began to use SSD to build storage systems. For the sake of comprehensive consideration of performance and cost, and to protect existing investments, users have the need to build hybrid storage systems using new storage media and original storage media. . The expensive and high-speed SSD is used as the primary storage system, and the high-capacity, low-cost low-speed mechanical disk is used as the back-end secondary storage system, and finally achieves the overall optimal configuration of high performance, large capacity, and low cost.

In a hybrid storage system, frequently accessed data is generally stored on a fast storage device, and infrequently accessed data is stored on a slow storage device, so that the overall performance of the system can be optimized by using less high-speed storage devices.

A multi-level storage system in the prior art is constructed by adding an SSD to a conventional mechanical HDD (Hard Disk). In the storage system formed by Drive, the hard disk drive forms a hybrid storage pool, and the SSD is used as the second level cache in the system, and the frequently accessed data on the disk is automatically cached to the SSD. When the SSD is full, these are The cached data is eliminated according to the ARC algorithm.

The above-mentioned prior art has the disadvantage that the SSD is used as the secondary cache of the system, which improves the performance of the system as a whole, but the SSD only exists as a cache, and its storage space is wasted. The above multi-level storage system is only suitable for systems with very concentrated hotspots, such as the 1/99 heat model system. However, for systems with low concentration of hotspots, such as the 30/70 model (30% of the data provides 70% of the traffic), 30% of the hottest data needs to be cached in the SSD, then the system needs additional SSD storage. Space accounts for 30% of the total storage capacity. This part of the storage space is not perceived by the application and cannot be reflected in the total storage capacity of the system. This is undoubtedly a huge waste.

Summary of the invention

Embodiments of the present invention provide a method and apparatus for implementing multi-level storage in a file system to implement multi-level storage of file-based values in a file system.

The object of the invention is achieved by the following technical solutions:

A method of implementing multi-level storage in a file system, comprising:

The first storage device and the secondary storage device are respectively used in the file system to store files, and obtain the value of each file stored in the primary storage device and the secondary storage device;

Establishing and managing a first file value sorting list according to an arrangement order of values of the respective files stored in the primary storage device, and establishing and managing the second file according to an arrangement order of values of the respective files stored in the secondary storage device a sorted list of values;

And performing file transfer on the primary storage device and the secondary storage device according to the set migration policy according to the first file value sorting list and the second file value sorting list.

A device for implementing multi-level storage in a file system, comprising:

a file storage module, configured to store a file by using a primary storage device and a secondary storage device respectively in the file system;

a file value processing module, configured to acquire values of respective files stored in the primary storage device and the secondary storage device, and establish and manage the first file according to an arrangement order of values of the files stored in the primary storage device a value sorting list, establishing and managing a second file value sorting list according to an arrangement order of values of the respective files stored in the secondary storage device;

The moving processing module is configured to perform file moving in the primary storage device and the secondary storage device according to the set moving policy according to the first file value sorting list and the second file value sorting list.

It can be seen from the technical solutions provided by the foregoing embodiments of the present invention that the embodiments of the present invention store files by using a primary storage device and a secondary storage device respectively in the file system, and based on the values of the respective files and according to the settings. The moving policy performs file migration in the primary storage device and the secondary storage device, and can store high-value files on a high-performance storage medium (ie, a primary storage device), and store low-value files in a slow On the fast storage medium (ie, the secondary storage device), the system achieves the best overall performance.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a flowchart of a specific process of a method for implementing multi-level storage in a file system according to Embodiment 1 of the present invention;

2 is a schematic diagram of fields in metadata information of a file according to an embodiment of the present invention;

FIG. 3 is a flowchart of a specific process for implementing multi-level storage in a file system when determining the value of a file according to a recent access time of a file according to Embodiment 2 of the present invention;

4 is a specific structural diagram of an apparatus for implementing multi-level storage in a file system according to Embodiment 3 of the present invention.

Mode for carrying out the invention

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Embodiment 1

This embodiment provides a method for implementing multi-level storage in a file system. The specific processing flow of the method is as shown in FIG. 1 and includes the following processing steps:

Step 11: The first-level storage device and the secondary storage device are used to store files in the file system, and the values of the files stored in the primary storage device and the secondary storage device are obtained.

For example, the SSD can be used as a primary storage device, and the mechanical disk is used as a secondary storage device. The first-level storage device and the secondary storage device are used to store files, and constitute a multi-level overall storage space of the file system.

The embodiment of the present invention needs to calculate the value of each file stored in the first-level storage device and the secondary storage device, and the specific calculation method is as follows:

According to the set file value update interval (for example, 30 minutes), according to at least one of the latest access time of the file, the number of accesses in the set time period, the content of the file, and the system configuration information, according to the set algorithm The value of each file stored in the primary storage device and the secondary storage device.

For example, when the value of a file is determined based only on the most recent access time of the file, the value of the file with the most recent access time is the largest.

For another example, when the value of the file is determined according to the recent access time of the file and the number of accesses in the set time period, the latest access time of the file may be multiplied by the corresponding weight, plus the set time period. The number of visits is multiplied by the corresponding weight to calculate the value of each file.

For example, the system administrator can directly set the value of one or more files according to actual needs.

A schematic diagram of the fields in the metadata information of a file provided by this embodiment is shown in FIG. 2. In an actual application, an existing partial field in the metadata information of each file and other optional extension fields may be utilized. The recorded information is used to calculate the value of the file, and the value information of the file, the sorting information, and the storage location information of the file are stored in an optional extension field in the metadata information. For example, after the file is accessed, the system updates the metadata information of the file, and uses the latest access time of the file recorded in the timestamp field in FIG. 2 to calculate the value of the file; or simultaneously utilizes the timestamp field in FIG. The value of the file is calculated by at least one of the recent and historical access time, file length field, and file block number fields of the recorded file. Then, the original value of the file stored in the file value field in FIG. 2 is deleted, the value of the newly calculated file is stored in the file value field, and the sort information field is updated accordingly.

The storage file in the storage location field in FIG. 2 is information stored in the primary storage device or the secondary storage device, and the sorted information field in FIG. 2 stores the first file value sorted list that will be described below or The sorting of the second file value sorted list. The sort information field, the storage location field, and the file value field in FIG. 2 above are extension fields in the metadata information.

Since the first-level storage device is a fast hard disk, and the second-level storage device is a slow-speed hard disk, a file with a relatively high value is stored in the first-level storage device, and a file with a relatively low value is stored in the mechanical hard disk. Ideally, the value of any of the files stored in the primary storage device above is greater than the value of all files stored in the mechanical hard disk.

Step 12: Establish and manage a first file value sorting list according to an arrangement order of values of the files stored in the primary storage device, and establish and manage according to the order of the values of the files stored in the secondary storage device. The second file value sorted list.

The first file value sorted list is created by sorting each file stored in the primary storage device according to the value of the value from large to small. After updating the value of each file stored in the primary storage device according to the set file value update time interval, the first file value sorted list needs to be updated accordingly.

The respective files stored in the secondary storage device are sorted according to the value of the value from the largest to the smallest to establish a second file value sorting list. After updating the value of each file stored in the secondary storage device according to the set file value update interval, the second file value sorted list needs to be updated accordingly.

The identifiers of the respective files are arranged in the first file value sorting list and the second file value sorting list.

Step 13: Perform file transfer on the primary storage device and the secondary storage device according to the set migration policy according to the first file value sorting list and the second file value sorting list.

Presetting a time interval for performing file transfer in the primary storage device and the secondary storage device, and starting timing of the time interval, after the time interval of the time interval ends, at the primary storage device And the file transfer processing is performed in the secondary storage device, and the process of the specific file transfer processing may be:

And if the value of the one or more files in the second file value sorted list exceeds the value of the at least one file in the first file value sorted list, the one or more of the second file value sorted list Moving the file from the secondary storage device to the primary storage device, and then updating the first file value sorted list and the second file value sorted list; or

If the space utilization of the primary storage device exceeds a set threshold, moving one or more files ranked in the first file value sorting list from the primary storage device to the second In the level storage device, until the space utilization rate of the primary storage device does not exceed a set threshold, then the first file value sorted list and the second file value sorted list are updated.

It should be further noted that, if the space utilization rate of the secondary storage device exceeds a set threshold, one or more files ranked in the second file value sorting list are stored from the secondary storage. The device is deleted until the space utilization rate of the secondary storage device does not exceed a set threshold, and then the second file value sorted list is updated.

Embodiment 2

When the value of the file is determined only according to the latest access time of the file, the embodiment provides a specific processing flow of the method for implementing multi-level storage in the file system. As shown in FIG. 3, the following processing steps are included:

Step 31: A file in the file system is accessed, and the file may be a file stored in the primary storage device or the secondary storage device.

Step 32: Since the value of the file is determined only according to the latest access time of the file, the value of the file currently accessed is the highest, and the value of the file is set to the maximum.

Step 33: Determine whether the file is stored in the primary storage device, and if yes, perform step 34; otherwise, perform step 35.

Step 34: Arranging the identifier of the file in the forefront of the first file value sorting list in the primary storage device.

Step 35: Add the file to the moving file list of the secondary storage device, and after the time interval of the preset file moving time ends, the file in the moving list of the secondary storage device is from the second The level storage device is moved to the primary storage device.

Embodiment 3

The embodiment of the invention further provides an apparatus for implementing multi-level storage in a file system, and the specific structure thereof is as shown in FIG. 4, and includes the following modules:

The file storage module 41 is configured to store the file in the file system by using the primary storage device and the secondary storage device respectively. For example, the SSD can be used as the primary storage device, and the mechanical disk is used as the secondary storage device.

The file value processing module 42 is configured to acquire the value of each file stored in the primary storage device and the secondary storage device, and establish and manage the first according to the order of the values of the files stored in the primary storage device. a file value sorting list, establishing and managing a second file value sorting list according to an arrangement order of values of the respective files stored in the secondary storage device;

The moving processing module 43 is configured to perform file moving in the primary storage device and the secondary storage device according to the set moving policy according to the first file value sorting list and the second file value sorting list.

Specifically, the file value processing module 42 may include:

The file value calculation module 421 is configured to determine, according to the set file value update time interval, the one according to at least one of a recent access time of the file, a number of accesses in the set time period, a content of the file, and system configuration information. The value of each file stored in the level storage device and the secondary storage device. For example, after the file is accessed, the system updates the metadata information of the file, and uses the latest access time of the file recorded in the timestamp field in the metadata information of the file to calculate the value of the file; Or, after the file is accessed, the system updates the metadata information of the file, and uses the recent and historical access time of the file recorded in the timestamp field in the metadata information of the file, and the file length field, The value of the file is calculated by at least one of the information recorded in the file block number field.

The file sorting list creating module 422 is configured to sort the files stored in the primary storage device according to the value of the value from the largest to the smallest, and establish a first file value sorting list according to the files stored in the primary storage device. The update result of the value updates the first file value sorted list;

And each of the files stored in the secondary storage device is sorted according to the value of the value from the largest to the smallest to establish a second file value sorting list, according to the update result of the value of each file stored in the secondary storage device. The second file value sorting list is updated.

Specifically, the moving processing module 43 is further configured to preset a time interval for performing file moving in the primary storage device and the secondary storage device, and start timing of the time interval when the time is After the interval has expired,

If the value of one or more files in the second file value sorted list exceeds the value of at least one of the first file value sorted lists, one or more of the second file value sorted list Files are moved from the secondary storage device to the primary storage device;

Updating the first file value sorted list and the second file value sorted list.

Specifically, the moving processing module 43 is further configured to preset a time interval for performing file moving in the primary storage device and the secondary storage device, and start timing of the time interval when the time is After the interval of the interval ends, if the space utilization rate of the primary storage device exceeds a set threshold, one or more files ranked in the first file value sorted list are sorted from the primary storage device. Moving to the secondary storage device until the space utilization of the primary storage device does not exceed a set threshold;

The specific process of implementing the multi-level storage in the file system by using the device in the embodiment of the present invention is similar to the foregoing method embodiment, and details are not described herein again.

One of ordinary skill in the art can understand that all or part of the process of implementing the foregoing embodiments can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, the flow of an embodiment of the methods as described above may be included. The storage medium may be a magnetic disk, an optical disk, or a read-only storage memory (Read-Only) Memory, ROM) or Random Access Memory (RAM).

In summary, the embodiment of the present invention stores files by using a primary storage device and a secondary storage device in a file system, and based on the value of each file and the set migration policy in the primary storage device and File transfer in the secondary storage device can store high-value files on high-performance storage media (ie, primary storage devices) and low-value files on slow storage media (ie, secondary storage devices). In order to achieve the best overall performance of the system.

The embodiment of the invention supports the unified management of the heterogeneous storage medium, and the overall storage space presented by the user is the sum of the storage spaces of the levels; the user does not have to distinguish the storage medium at all levels when using the file, and the file system provides unified management to the user. This eliminates the need for additional tiered storage management software and reduces complexity.

The embodiments of the present invention provide flexible policy configuration, which can flexibly control the frequency cycle of file migration and the algorithm followed by file migration.

The above is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or within the technical scope disclosed by the present invention. Alternatives are intended to be covered by the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

A method for implementing multi-level storage in a file system, comprising:

The first storage device and the secondary storage device are respectively used in the file system to store files, and obtain the value of each file stored in the primary storage device and the secondary storage device;

Establishing and managing a first file value sorting list according to an arrangement order of values of the respective files stored in the primary storage device, and establishing and managing the second file according to an arrangement order of values of the respective files stored in the secondary storage device a sorted list of values;

And performing file transfer on the primary storage device and the secondary storage device according to the set migration policy according to the first file value sorting list and the second file value sorting list.
The method for implementing the multi-level storage in the file system according to claim 1, wherein the obtaining the value of each file stored in the primary storage device and the secondary storage device comprises:

Updating the time interval according to the set file value in the file system, and determining the primary storage device according to at least one of a recent access time of the file, a number of accesses in the set time period, a content of the file, and system configuration information. The value of each file stored in the secondary storage device.
The method for implementing the multi-level storage in the file system according to claim 2, wherein the obtaining the value of each file stored in the primary storage device and the secondary storage device comprises:

After the file is accessed, the system updates the metadata information of the file, and uses the latest access time of the file recorded in the timestamp field in the metadata information of the file to calculate the value of the file;

or,

After the file is accessed, the system updates the metadata information of the file, and uses the recent and historical access time of the file recorded in the timestamp field in the metadata information of the file, and the file length field and the file block. At least one of the information recorded in the digital segment calculates the value of the file.
The method for implementing multi-level storage in a file system according to claim 3, wherein said first file value ranking is established and managed according to an arrangement order of values of respective files stored in said primary storage device a list, the second file value sorting list is established and managed according to an arrangement order of values of the respective files stored in the secondary storage device, including:

And sorting, according to the size of the value, the first file value sorting list according to the value of the value stored in the first-level storage device, and the updating result according to the value of each file stored in the first-level storage device A file value sorting list is updated;

And each of the files stored in the secondary storage device is sorted according to the value of the value from the largest to the smallest to establish a second file value sorting list, according to the update result of the value of each file stored in the secondary storage device. The second file value sorting list is updated.
The method for implementing multi-level storage in a file system according to claim 1, wherein the primary storage device comprises a solid state drive; and the secondary storage device comprises a mechanical disk.
The method for implementing multi-level storage in a file system according to any one of claims 1 to 5, wherein said sorting list according to said first file value sorting list and said second file value are arranged according to The moving strategy performs file migration in the primary storage device and the secondary storage device, including:

Setting a time interval for performing file transfer in the primary storage device and the secondary storage device, and starting timing of the time interval, when the time interval of the time interval ends,

If the value of one or more files in the second file value sorted list exceeds the value of at least one of the first file value sorted lists, one or more of the second file value sorted list Files are moved from the secondary storage device to the primary storage device;

Updating the first file value sorted list and the second file value sorted list.
The method for implementing multi-level storage in a file system according to any one of claims 1 to 6, wherein the sorting list according to the first file value sorting list and the second file value is set according to The moving strategy performs file migration in the primary storage device and the secondary storage device, including:

Setting a time interval for performing file transfer in the primary storage device and the secondary storage device, and starting timing of the time interval, when the time interval of the time interval ends,

If the space utilization of the primary storage device exceeds a set threshold, moving one or more files ranked in the first file value sorting list from the primary storage device to the second In the level storage device, until the space utilization rate of the primary storage device does not exceed a set threshold;

Updating the first file value sorted list and the second file value sorted list.
An apparatus for implementing multi-level storage in a file system, comprising:

a file storage module, configured to store a file by using a primary storage device and a secondary storage device respectively in the file system;

a file value processing module, configured to acquire values of respective files stored in the primary storage device and the secondary storage device, and establish and manage the first file according to an arrangement order of values of the files stored in the primary storage device a value sorting list, establishing and managing a second file value sorting list according to an arrangement order of values of the respective files stored in the secondary storage device;

The moving processing module is configured to perform file moving in the primary storage device and the secondary storage device according to the set moving policy according to the first file value sorting list and the second file value sorting list.
The apparatus for implementing multi-level storage in a file system according to claim 8, wherein the file value processing module comprises:

a file value calculation module, configured to update the time interval according to the set file value, and determine the first level according to at least one of a recent access time of the file, a number of accesses in the set time period, a content of the file, and system configuration information. The value of each file stored in the storage device and the secondary storage device;

a file sorting list establishing module, configured to sort, according to a value of a value, a first file value sorting list according to a value of the value stored in the first-level storage device, according to each file stored in the first-level storage device The update result of the value updates the first file value sorted list;

And each of the files stored in the secondary storage device is sorted according to the value of the value from the largest to the smallest to establish a second file value sorting list, according to the update result of the value of each file stored in the secondary storage device. The second file value sorting list is updated.
The apparatus for implementing multi-level storage in a file system according to claim 9, wherein:

The file value calculation module is further configured to: after the file is accessed, the system updates the metadata information of the file, and uses the latest access of the file recorded in the timestamp field in the metadata information of the file. Time to calculate the value of the document;

or,

After the file is accessed, the system updates the metadata information of the file, and uses the recent and historical access time, the file length field, and the file block number of the file recorded in the timestamp field in the metadata information of the file. At least one of the information recorded in the segment calculates the value of the file.
Apparatus for implementing multi-level storage in a file system according to claim 8 or 9 or 10, wherein:

The moving processing module is further configured to preset a time interval for performing file transfer in the primary storage device and the secondary storage device, and start timing of the time interval, when the time interval ends after that,

If the value of one or more files in the second file value sorted list exceeds the value of at least one of the first file value sorted lists, one or more of the second file value sorted list Files are moved from the secondary storage device to the primary storage device;

Updating the first file value sorted list and the second file value sorted list.
Apparatus for implementing multi-level storage in a file system according to claim 8 or 9 or 10, wherein:

The moving processing module is further configured to preset a time interval for performing file transfer in the primary storage device and the secondary storage device, and start timing of the time interval, when the time interval ends And then, if the space utilization rate of the primary storage device exceeds a set threshold, moving one or more files arranged in the first file value sorting list from the primary storage device to the In the secondary storage device, until the space utilization rate of the primary storage device does not exceed a set threshold;

Updating the first file value sorted list and the second file value sorted list.