CN100544277C - A kind of method and apparatus that improves data-handling efficiency of network management system - Google Patents
A kind of method and apparatus that improves data-handling efficiency of network management system Download PDFInfo
- Publication number
- CN100544277C CN100544277C CNB2006100600720A CN200610060072A CN100544277C CN 100544277 C CN100544277 C CN 100544277C CN B2006100600720 A CNB2006100600720 A CN B2006100600720A CN 200610060072 A CN200610060072 A CN 200610060072A CN 100544277 C CN100544277 C CN 100544277C
- Authority
- CN
- China
- Prior art keywords
- field
- compressible
- management system
- splicing
- network management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Abstract
The invention discloses a kind of method that improves data-handling efficiency of network management system, comprise step to the compression of data database data, it is characterized in that the step of described data compression comprises again: step 1 is divided into compressible field and non-compressible field to the field of database table; Step 2 is adjusted the order of described compressible field and described non-compressible field; Step 3 is spliced described compressible field and is generated the splicing field, and replaces described compressible field; And step 4, described splicing field is compressed processing generate newer field, and replace described splicing field.The compressible field database table that adopts the present invention to propose is spliced the method for afterwards compressing earlier, can save database space, significantly reduces resource cost, effectively improves inquiry and modification speed, and does not influence the retrieval of database data.
Description
Technical field
The present invention relates to the big data quantity data processing technique, particularly relate to a kind of method that improves data-handling efficiency of network management system in the webmaster field.
Background technology
In network management system, the historical data excessive situation that takes up room often appears.For example for a DWDM (Dense Wave Division Multiplexing, dense wave division multipurpose) network management system, every veneer of network element just produced one time 15 minutes historical performances and reports webmaster every 15 minutes, even according to the History Performance Data that keeps 3 months in the engineering minimum requirements database, the historical performance under the situation of 32 ripple full configurations and 100 network elements also has about 8G.
The problem that data volume causes greatly is that communication bandwidth, CPU processing time, memory capacity etc. are all occupied, and the efficiency of management of network management system reduces greatly.
For this reason, demand having proposed a solution urgently, with taking of the database space of effective reduction historical performance.
Summary of the invention
Technical problem to be solved by this invention is to provide a kind of method that improves data-handling efficiency of network management system, and it is excessive to be used for solving prior art big data quantity database table data occupancy space, and resource cost is serious, the problem that inquiry/modification speed is all very slow.
To achieve these goals, the invention provides a kind of method that improves data-handling efficiency of network management system, comprise the step of database data compression, the step of described data compression comprises again:
Step 2 is adjusted the order of described compressible field and described non-compressible field;
The method of described raising data-handling efficiency of network management system, wherein, in the described step 1, as the field of the database table of described compressible field not as the external key of the table outside inquiry restrictive condition, sort criteria, design conditions, branch set condition and the described database table.
The method of described raising data-handling efficiency of network management system, wherein, in the described step 2, described adjustment is in proper order: all described compressible fields are got up continuously and are come the back of all described non-compressible fields.
The method of described raising data-handling efficiency of network management system, wherein, in the described step 3, the number that comprises variable-length field when described compressible field not simultaneously, the method for splicing described compressible field can adopt one of following connecting method, comprising:
Mode one, when not having variable-length field in the described compressible field, direct splicing; Or
Mode two when having only a variable-length field in the described compressible field, is adjusted the order of described compressible field, and this variable-length field is put into the last of described compressible field, splicing then; Or
Mode three is when having two or more variable-length field in the described compressible field, to splicing again behind this variable-length field coding.
The method of described raising data-handling efficiency of network management system wherein, is encoded to this variable-length field by add the byte of the length of one or more these variable-length fields of record in this variable-length field front.
The method of described raising data-handling efficiency of network management system, wherein, in the described step 4, the field type of described newer field is determined according to described splicing field being compressed the compression algorithm that adopts when handling.
The method of described raising data-handling efficiency of network management system, wherein, the field type of described newer field is for being varbinary.
The method of described raising data-handling efficiency of network management system wherein, in the described step 4, comprises that also one adopts the RFC1952 compression algorithm described splicing field to be compressed the step of processing.
A kind of device that improves data-handling efficiency of network management system comprises the unit that database data compresses, and wherein, described device also further comprises:
Be used for the field of database table is divided into the unit of compressible field and non-compressible field;
Be used to adjust the unit of the order of described compressible field and described non-compressible field;
Be used to splice described compressible field and generate the splicing field, and replace the unit of described compressible field;
Be used for that described splicing field is compressed processing and generate newer field, and replace the unit of described splicing field.
The method of described raising data-handling efficiency of network management system, wherein, described network management system is applicable to the dense wave division multipurpose network management system.
Technique effect of the present invention is:
The database space that uses the inventive method can effectively reduce data takies.For example, total field length of table is A, and the total length of compressible field is B, and the length of the field that is spliced into by compressible field after the compression is C, all is under the situation of constant at A, B, C, saves the percentage that database space takies to be:
((B-C)/A)*100％
Can draw from above-mentioned formula, excessive at the database table data capacity, and do not account for table data total capacity ratio when higher as the data capacity of the field of search condition, and the compressible of these fields adopts the inventive method to have following advantage when higher:
(1), saves database space; It is relevant with the compressible ratio of compressible field that saving ratio and compressible field spacing account for total field spacing ratio;
(2), do not influence retrieval; All fields that are used for retrieving remain unchanged, and do not influence original search function.
Describe the present invention below in conjunction with the drawings and specific embodiments, but not as a limitation of the invention.
Description of drawings
Fig. 1 adjusts the compressible field of database table for the present invention and the table after the non-compressible order of the field is provided with schematic diagram;
Fig. 2 is provided with schematic diagram for the present invention to the spliced table of compressible field of database table;
Fig. 3 is provided with schematic diagram for the final table of the present invention after to the splicing field compression of database table;
Fig. 4 splices earlier the method flow diagram that afterwards compresses for the present invention to the compressible field of database table;
Fig. 5 is the embodiment of the History Performance Data aspect of Fig. 1;
Fig. 6 is the embodiment of the History Performance Data aspect of Fig. 2;
Fig. 7 is the embodiment of the History Performance Data aspect of Fig. 3.
Wherein, main Reference numeral is as follows:
Non-compressible field 1
Compressible field 2
The field 3 that compressible field is spliced into
Be spliced the field 4 after the field compression
The number n 1 of non-compressible field 1
The number n 2 of compressible field 2
1001,1002 ... the compressible field 1 of 10n1 right and wrong, non-compressible field 2 ..., non-compressible field n1
2001,2002 ... 20n2 is compressible field 1, compressible field 2 ..., compressible field n2
Embodiment
Figure 1 shows that the present invention adjusts the compressible field of database table and the table after the non-compressible order of the field is provided with schematic diagram.
Database table is reset, adjust the order of all fields, all compressible fields 2 are got up continuously and come the back of all non-compressible fields 1, be the literary name section be set in proper order " non-compressible field 1 (1001); non-compressible field 2 (1002); ...; non-compressible field n1 (10n1); compressible field 1 (2001); compressible field 2 (2002) ..., compressible field n2 (20n2) ", the number of the compressible field 1 of n1 right and wrong wherein, n2 is the number of compressible field 2.
Figure 2 shows that the present invention is provided with schematic diagram to the spliced table of compressible field of database table, and in conjunction with shown in Figure 1.
Compressible field 2 shown in Figure 1 is spliced into a field 3, and replaces compressible field 2; With compressible field 1 (2001), compressible field 2 (2002), ..., compressible field n2 (20n2) is spliced into a field 3, and replace all compressible fields 2, wherein n2 is the number of compressible field 2, that is to say, after the splicing of the compressible field of database table is handled, its literary name section be set to " non-compressible field 1 (1001), non-compressible field 2 (1002); ...; non-compressible field n1 (10n1), the field 3 that compressible field is spliced into ", the number of the compressible field 1 of n1 right and wrong wherein.
Figure 3 shows that the final table after the present invention is to the splicing field compression of database table is provided with schematic diagram; And in conjunction with shown in Figure 2.
Adopt compression algorithm to compress processing with shown in Figure 2 to spliced field 3, and will compress the spliced field 3 of newer field 4 replacements that generates; Database table finally be set to " non-compressible field 1 (1001), non-compressible field 2 (1002) ..., non-compressible field n1 (10n1), the newer field 4 that splicing generates after the field compression ", the number of the compressible field 1 of n1 right and wrong wherein.
The field type of newer field 4 is relevant with compression algorithm, generally is varbinary.
Figure 4 shows that the compressible field of database table is spliced earlier the method flow diagram that afterwards compresses.And in conjunction with Fig. 1,2,3, this method flow specifically comprises the steps:
The database table field can be divided into two kinds of fields: the index field that all table index place fields are formed; And the non-index field of other fields compositions beyond the index field.
Index field is the used field of question blank normally.Index field is added that the field that needs in the non-index field to be used for inquiring about is together as non-compressible field 1; Remaining field beyond the non-compressible field 1 is called compressible field 2.Must satisfy following condition as compressible field 2:
(1), not as inquiry restrictive condition (not appearing in WHERE and the HAVING clause);
(2), not as sort criteria (not appearing in the ORDER BY clause);
(3), not as design conditions (not appearing in the COMPUTE clause);
(4), not as dividing set condition (not appearing in the GROUP BY clause); And
(5), not as the external key of other tables (not appearing in the REFERENCE restriction of other tables).
Above-mentioned WHERE and HAVING clause, ORDER BY clause, COMPUTE clause, GROUP BY clause, REFERENCE are SQL (Structured Query Language, SQL).
Such as, the History Performance Data of network management system, 15 minutes performance data sizes of the history of DWDM network management system are usually more than the 4G byte.If the Field Definition of historical 15 fens clock and watch is as follows:
Time INT
NE ID INT
Groove position SMALLINT
Performance SMALLINT
Performance 2 INT
Then can be time, NE ID, groove position and performance number these four fields as non-compressible field 1, and performance 1, performance 2, performance 3,4 four performance value fields of performance be not as search condition, and when meeting the condition of compressible field, can be used as compressible field 2.
Step 402 is adjusted the order of non-compressible field 1 and compressible field 2;
This step is specially: database table is reset, adjusted the order of all fields, all compressible fields 2 are got up continuously and come the back of all non-compressible fields 1, the literary name section is provided with as shown in Figure 1 in proper order.
For the table of 15 minutes performance datas of history of DWDM network management system as mentioned above, through after the set handling of step 402, four performance value fields are as compressible field 2, and they continuously and come the back of non-compressible field 1.
Step 403 is spliced into a field 3 with compressible field 2, and replaces compressible field 2;
The literary name section is through after passing through this step process again, and table is provided with as shown in Figure 2; And the mode at the difference splicing of the field number that comprises variable-length field/elongated in the compressible field 2 also is divided into following three kinds of situations:
Situation one, if do not have in the compressible field 2 elongated field (varchar, varbinary, text, image), direct splicing then.The purpose of doing like this is to splice field later on through overcompression, and compressible field 2 can be according to the byte length alignment of each field during decompress(ion).
Situation two, (text image), then adjusts the order of field for varchar, varbinary, and elongated field is put into the last of compressible field 2, splicing then if having only an elongated field in the compressible field 2.The purpose of doing like this is to splice field later on through overcompression, and the field of the front fixed length of compressible field 2 can be according to the byte length alignment of each field during decompress(ion), and remaining content is the content of variable-length field.
Situation three is if having two or more elongated field (varchar, varbinary in the compressible field 2, text image), then needs elongated code field, as add the length of one or several this field of byte records in this elongated field front, splicing then.The purpose of doing is like this spliced field through overcompression after being, the field of the fixed length during decompress(ion) in the compressible field 2 can be according to the byte length alignment of each field, and variable-length field is according to the content of coding rule alignment oneself.
Table for 15 minutes performance datas of history of aforesaid DWDM network management system, after the set handling through step 402 and step 403, four performance value fields are as compressible field 2, because of performance number all is to fix 4 bytes, so can direct splicing become a field.The splicing example is as follows:
If the value of performance 1 is 0x00000000, the value of performance 2 is 0x11111111, and the value of performance 3 is 0x22222222, and the value of performance 4 is 0x33333333, and then the content of the field of direct splicing generation is 0x00000000111111112222222233333333.
Step 404 will be spliced field 3 compressions, generate newer field 4, and replace splicing field 3.
Wherein, spliced field 3 is adopted the compression algorithm compression, the field type of the newer field 4 that compression generates is relevant with compression algorithm, generally is varbinary, and the newer field 4 of using compression to generate replaces spliced field 3.Table is final to be provided with as shown in Figure 3.
Table for 15 minutes performance datas of history of aforesaid DWDM network management system, process step 401 is to step 404 set handling, the splicing performance is to fix 16 bytes, experimental results show that: performance classes The data RFC (Request For Comments, Request for Comment) the compressible ratio of 1952 (gzip format) compression algorithm is more than 10:1, the splicing performance is compressed in should be no problem in 4 bytes, and the performance field after the compression can be set to " VARBINARY (4) * ".New table setting can for:
Time INT
NE ID INT
Groove position SMALLINT
Performance SMALLINT
Compression performance VARBINARY (4) *
Wherein 4 expression field lengths among " * " the expression VARBINARY (4) are 4 bytes.
To step 404 processing, the data space ratio before and after realizing improving is more than the 7:4, promptly minimum 42% the space of having saved by step 401.
Figure 5 shows that the embodiment of the History Performance Data aspect of Fig. 1.Shown in Fig. 1,4.
For the table of 15 minutes performance datas of history of the DWDM network management system among Fig. 1, after the set handling through step 402 shown in Figure 4, four performance value fields are as compressible field 2, and they continuously and come the back of non-compressible field 1.
Figure 6 shows that the embodiment of the History Performance Data aspect of Fig. 2.Shown in Fig. 2,4.
Table for 15 minutes performance datas of history of the DWDM network management system among Fig. 2, after the set handling through step 403 shown in Figure 4, four performance value fields are as compressible field 2, because of performance number all is to fix 4 bytes, can direct splicing become a field, become the splicing field.
Figure 7 shows that the embodiment of the History Performance Data aspect of Fig. 3.Shown in Fig. 3,4.
For the table of 15 minutes performance datas of history of the DWDM network management system among Fig. 3, after the set handling through step 404 shown in Figure 4, the splicing field is compressed processing, generate newer field, replace the field of splicing field after as compression.
The compressible field database table that adopts the present invention to propose is spliced the method for afterwards compressing earlier, can save database space, significantly reduces resource cost, effectively improves inquiry and modification speed, and does not influence the retrieval of database data.
Certainly; the present invention also can have other various embodiments; under the situation that does not deviate from spirit of the present invention and essence thereof; those of ordinary skill in the art work as can make various corresponding changes and distortion according to the present invention, but these corresponding changes and distortion all should belong to the protection range of the appended claim of the present invention.
Claims (10)
1, a kind of method that improves data-handling efficiency of network management system comprises the step that database data compresses, and it is characterized in that the step of described data compression comprises again:
Step 1 is divided into compressible field and non-compressible field to the field of database table;
Step 2 is adjusted the order of described compressible field and described non-compressible field;
Step 3 is spliced described compressible field and is generated the splicing field, and replaces described compressible field;
Step 4 is compressed processing to described splicing field and is generated newer field, and replaces described splicing field.
2, the method for raising data-handling efficiency of network management system according to claim 1, it is characterized in that, in the described step 1, as the field of the database table of described compressible field not as the external key of the table outside inquiry restrictive condition, sort criteria, design conditions, branch set condition and the described database table.
3, the method for raising data-handling efficiency of network management system according to claim 1 is characterized in that, in the described step 2, described adjustment is in proper order: all described compressible fields are got up continuously and are come the back of all described non-compressible fields.
4, according to the method for claim 1,2 or 3 described raising data-handling efficiency of network management system, it is characterized in that in the described step 3, the number that comprises variable-length field when described compressible field is not simultaneously, the method of splicing described compressible field can adopt one of following connecting method, comprising:
Mode one, when not having variable-length field in the described compressible field, direct splicing; Or
Mode two when having only a variable-length field in the described compressible field, is adjusted the order of described compressible field, and this variable-length field is put into the last of described compressible field, splicing then; Or
Mode three is when having two or more variable-length field in the described compressible field, to splicing again behind this variable-length field coding.
5, the method for raising data-handling efficiency of network management system according to claim 4, it is characterized in that, when in the described compressible field two or more variable-length field being arranged, this variable-length field is encoded by add the byte of the length of one or more these variable-length fields of record in this variable-length field front.
According to the method for claim 1,2,3 or 5 described raising data-handling efficiency of network management system, it is characterized in that 6, in the described step 4, the field type of described newer field is determined according to described splicing field being compressed the compression algorithm that adopts when handling.
7, the method for raising data-handling efficiency of network management system according to claim 6 is characterized in that, the field type of described newer field is for being varbinary.
8, according to the method for claim 1,2,3 or 5 described raising data-handling efficiency of network management system, it is characterized in that, in the described step 4, comprise that also one adopts the RFC1952 compression algorithm described splicing field to be compressed the step of processing.
According to the method for claim 1,2,3 or 5 described raising data-handling efficiency of network management system, it is characterized in that 9, described network management system is applicable to the dense wave division multipurpose network management system.
10, a kind of device that improves data-handling efficiency of network management system comprises the unit that database data compresses, and it is characterized in that described device also further comprises:
Be used for the field of database table is divided into the unit of compressible field and non-compressible field;
Be used to adjust the unit of the order of described compressible field and described non-compressible field;
Be used to splice described compressible field and generate the splicing field, and replace the unit of described compressible field;
Be used for that described splicing field is compressed processing and generate newer field, and replace the unit of described splicing field.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2006100600720A CN100544277C (en) | 2006-03-25 | 2006-03-25 | A kind of method and apparatus that improves data-handling efficiency of network management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2006100600720A CN100544277C (en) | 2006-03-25 | 2006-03-25 | A kind of method and apparatus that improves data-handling efficiency of network management system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101043353A CN101043353A (en) | 2007-09-26 |
| CN100544277C true CN100544277C (en) | 2009-09-23 |
Family
ID=38808584
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2006100600720A Expired - Fee Related CN100544277C (en) | 2006-03-25 | 2006-03-25 | A kind of method and apparatus that improves data-handling efficiency of network management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100544277C (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102456110A (en) * | 2010-10-26 | 2012-05-16 | 镇江精英软件科技有限公司 | Method used for achieving safe information preservation through combining, encrypting and storing field information in special field |
| CN102298634B (en) * | 2011-09-09 | 2013-05-29 | 厦门市美亚柏科信息股份有限公司 | Structural reorganization method for Sqlite deletion record |
| CN103218364B (en) * | 2012-01-19 | 2016-05-04 | 阿里巴巴集团控股有限公司 | A kind of searching method and system |
| CN103379136B (en) * | 2012-04-17 | 2017-02-22 | 中国移动通信集团公司 | Compression method and decompression method of log acquisition data, compression apparatus and decompression apparatus of log acquisition data |
| CN105812094B (en) * | 2016-03-07 | 2019-08-02 | 电信科学技术研究院 | A kind of method, apparatus of data processing, terminal and access device |
| CN107832345A (en) * | 2017-10-16 | 2018-03-23 | 千寻位置网络有限公司 | The method of base station data unique numberization mark |
-
2006
- 2006-03-25 CN CNB2006100600720A patent/CN100544277C/en not_active Expired - Fee Related
Non-Patent Citations (2)
| Title |
|---|
| 用ODBC API实现数据库大字段连续存取. 戴祖旭.武汉化工学院学报，第26卷第1期. 2004 |
| 用ODBC API实现数据库大字段连续存取. 戴祖旭.武汉化工学院学报，第26卷第1期. 2004 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101043353A (en) | 2007-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100544277C (en) | A kind of method and apparatus that improves data-handling efficiency of network management system | |
| US9483533B2 (en) | Method and apparatus for processing time series data | |
| KR100614677B1 (en) | Method for compressing/decompressing a structured document | |
| US7539685B2 (en) | Index key normalization | |
| US7792877B2 (en) | Scalable minimal perfect hashing | |
| US8933829B2 (en) | Data compression using dictionary encoding | |
| US8239421B1 (en) | Techniques for compression and processing optimizations by using data transformations | |
| CN110518917B (en) | LZW data compression method and system based on Huffman coding | |
| JPH0779262B2 (en) | Encoding method of compressed data | |
| US7647291B2 (en) | B-tree compression using normalized index keys | |
| US20120189201A1 (en) | Efficient histogram storage | |
| Albers et al. | Average case analyses of list update algorithms, with applications to data compression | |
| CN103605730A (en) | XML (extensible markup language) compressing method and device based on flexible-length identification codes | |
| JP5549177B2 (en) | Compression program, method and apparatus, and decompression program, method and apparatus | |
| CN116827354B (en) | File data distributed storage management system | |
| CN111400390B (en) | Data processing method and device | |
| WO2019018030A1 (en) | Structured record compression and retrieval | |
| CN112115302A (en) | Parallel XPath query method of XML document, terminal equipment and storage medium | |
| CN102567515B (en) | Picture description information inserting and reading method and device | |
| CN1129232C (en) | Method for compressing and decompressing data in database | |
| CN112148734B (en) | Transaction data processing method, device, equipment and storage medium based on block chain | |
| CN110263339A (en) | It is a kind of based on Uighur syllable can retrieval type compression method | |
| CN105183750B (en) | Close-coupled XML resolution system | |
| CN114666406B (en) | Electric power Internet of things data compression method and device based on object model | |
| CN115102830B (en) | Log reduction method, device, computer equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20090923 Termination date: 20150325 |
|
| EXPY | Termination of patent right or utility model |