US20160196277A1 - Data record compression with progressive and/or selective decompression - Google Patents
Data record compression with progressive and/or selective decompression Download PDFInfo
- Publication number
- US20160196277A1 US20160196277A1 US15/066,824 US201615066824A US2016196277A1 US 20160196277 A1 US20160196277 A1 US 20160196277A1 US 201615066824 A US201615066824 A US 201615066824A US 2016196277 A1 US2016196277 A1 US 2016196277A1
- Authority
- US
- United States
- Prior art keywords
- data
- fields
- record
- compression
- records
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G06F17/30153—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24561—Intermediate data storage techniques for performance improvement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G06F17/30424—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03H—IMPEDANCE NETWORKS, e.g. RESONANT CIRCUITS; RESONATORS
- H03H7/00—Multiple-port networks comprising only passive electrical elements as network components
- H03H7/30—Time-delay networks
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/3068—Precoding preceding compression, e.g. Burrows-Wheeler transformation
- H03M7/3077—Sorting
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/60—General implementation details not specific to a particular type of compression
- H03M7/6064—Selection of Compressor
- H03M7/6082—Selection strategies
- H03M7/6088—Selection strategies according to the data type
Definitions
- the present disclosure relates in general to data compression and databases, and more specifically to methods of compression for use in in-memory databases as well as document databases.
- a database is an organized collection of information stored as “records” having “fields” of information.
- a restaurant database may have a record for each restaurant in a region, where each record contains fields describing characteristics of the restaurant, such as name, address, type of cuisine, and the like).
- databases may use clusters of computers in order to be able to store and access large amounts of data. This may require a large amount of information storage space.
- compression may be used to reduce the amount of storage space necessary to host the information, but it may increase the computational load significantly as many common compression methods require the entire record or many records to be decompressed every time they are accessed.
- System and method embodiments described herein may implement a combination of suitable data compression processes to each field of database, such that a compressed database record achieves a compression ratio comparable to commercially-accepted ratios, while still allowing decompression of the fields to occur only for the records and fields of interest (i.e., only decompressing data records or fields satisfying a database search query).
- Implementing compression techniques that facilitate selective decompression of records or fields allows for horizontal record-based storage of the compressed data, but also columnar or vertical access to the fields of the data on decompression. This provides the reduced storage benefits of compression, while avoiding much of the compute power and latency associated with decompression when only specific fields are to be decompressed.
- Systems and methods described herein may also implement N-gram compression techniques.
- N-grams are restricted to compressing only one of chains of letters (successive characters of a string), or to chains of words (successive strings in text).
- Conventional N-gram compression is unable to compress chains of letters, individual words, and/or chains of words, within a single implementation of such a compression technique.
- Described herein is the use of N-gram-related compression for columnar compression during record storage, thereby allowing good overall compression, while still providing low-latency access to a single record or a single field within a record, in response to search queries.
- a computer-implemented method comprises determining, by a computer, a compression technique to apply to one or more data elements received in a set of data elements, wherein the computer uses a schema to determine the compression technique to apply to each data element based on a data type of the data element; compressing, by a computer, a data element using the compression technique defined by the schema, wherein the compression technique compresses the data element such that the data element is individually decompressed when returned in response to a search query; storing, by the computer, each compressed data element in a field of a record that stores data of the data type of the data element; associating, by the computer, a field notation in a reference table for each field according to a schema, wherein the representative notation identifies the data type of the field; querying, by the computer, the database for a set of one or more data elements satisfying a search query received from a search conductor; and decompressing, by the computer, each of the one or more data elements of the one or more data
- a computing system comprises one or more nodes storing one or more collections, each collection comprising a set of one or more records, each record comprising a set of fields storing data; and a compression processor compressing one or more of the fields according to a schema that is associated with a collection.
- FIG. 1 illustrates a data compression apparatus according to an exemplary embodiment.
- FIG. 2 illustrates a structured data table according to an exemplary embodiment.
- FIG. 3 illustrates a token table according to an exemplary embodiment.
- FIG. 4 illustrates a n-gram table according to an exemplary embodiment.
- FIG. 5 illustrates a table describing compressed records according to an exemplary embodiment.
- Node refers to a computer hardware configuration suitable for running one or more modules.
- Cluster refers to a set of one or more nodes.
- Module refers to a computer software component suitable for carrying out one or more defined tasks.
- Selection refers to a discrete set of records.
- Record refers to one or more pieces of information that may be handled as a unit.
- Field refers to one data element within a record.
- Object refers to a logical collection of fields within a data record.
- Array refers to an ordered list of data values within a record.
- Node refers to a field, object, or array within a record.
- Partition refers to an arbitrarily delimited portion of records of a collection.
- Scheme refers to data describing one or more characteristics of a collection, record, or field.
- Compress may refer to reducing the amount of electronic data needed to represent a value.
- “Dictionary” may refer to any computerized list suitable for use as a value reference.
- Token Table refers to a table defining one or more simpler values for one or more other more complex values.
- N-gram refers to N successive integral units of data which can be characters, words, or groups of words where N is greater than or equal to 1—i.e., in the sentence “The quick brown fox jumped over the lazy dog,” “the,” “e,” “he,” and “brown fox” are all valid N-grams.
- N-gram Table refers to a table defining one or more simpler values for one or more other more complex values.
- Search Conductor refers to a module configured to at least run one or more queries on a partition and return the search results to one or more search managers.
- Partitioner refers to a module configured to at least divide one or more collections into one or more partitions.
- Database refers to any system including any combination of clusters and modules suitable for storing one or more collections and suitable to process one or more queries.
- Query refers to a request to retrieve information from one or more suitable partitions or databases.
- Memory refers to any hardware component suitable for storing information and retrieving said information at a sufficiently high speed.
- “Fragment” refers to separating records into smaller records until a desired level of granularity is achieved.
- JSON refers to the JavaScript Object Notation, a data-interchange format.
- BSON refers to Binary JSON, a data-interchange format.
- YAML refers to the coding language “YAML Ain't Markup Language,” a data-interchange format.
- Document refers to a group of structured or semi-structured information.
- Document Database refers to a document-oriented database, designed for storing, retrieving, and managing document-oriented information.
- one or more collections may include structured or semi-structured data that may include any number of records and any suitable number of fields, where the collections may be described using any suitable schema that may define the data structure and the compression method used for one or more fields.
- one or more fields may include information that may have a semantic similarity.
- the fields may be compressed using one or more methods suitable for compressing the type of data stored in the field, where token tables, N-gram compression, serial day number compression, binary number compression, or any other suitable method may be used.
- one or more data in one or more records of a collection may include data that may be better compressed after fragmentation and fragmented data may be stored contiguously in the same partition.
- fragmented record identifiers may be used to identify which record they were fragmented from to ensure the system remains aware that the records originate from the same original record in the collection. Data that would be duplicated across fragments will only have a single representation in the compressed form with an anchor to which other fragments can refer.
- a record may contain an array of data values. Arrays may contain zero or more values. Values may be fields, objects, or other arrays.
- one or more data values may be grouped as an object.
- Objects may contain fields, other objects, or arrays, and may be elements of other objects or arrays.
- Objects within a record may be compressed further by including an anchor value that refers the system to another object or fragment in the partition with identical values.
- an anchor value that refers the system to another object or fragment in the partition with identical values.
- input records may be semi-structured data and may be represented using JSON, BSON, YAML or any other suitable data format.
- one or more data fields may be normalized prior to compression.
- fields including data with a suitably semantic similarity may be compressed using any suitable token table.
- the system determines whether the data may match previously encountered data in the token table. In one or more embodiments, if the data does not match, the system may use an alternate compression method or may update the token table. In other embodiments, the token table may be updated periodically.
- fields including data with a suitably semantic similarity may be compressed using any suitable n-gram table.
- the system determines whether the data may match previously encountered data in the n-gram table. In one or more embodiments, if the data does not match, the system may use an alternate compression method or may update the n-gram table. In other embodiments, the n-gram table may be updated periodically.
- the most frequently occurring values may be stored in the lower numbered indices, which may allow for the most frequently used values to be represented with fewer bytes.
- a longer value may be preferred over a shorter value for inclusion in the token table, which may allow for greater compression by eliminating longer values with the same index size as a smaller value.
- a longer value may be preferred over a shorter value for inclusion in the n-gram table, which may allow for greater compression by eliminating longer values with the same index size as a smaller value.
- records may include zero or more record descriptor bytes, any suitable number of field descriptor bytes, any suitable number of array descriptor bytes, any suitable number of object descriptor bytes, and any suitable number of bytes representing the data associated with the record.
- data in a field associated with a token table may use one or more bits to state whether the information stored in the record is compressed using the compression method defined in the schema or whether another compression method, such as n-gram compression, was used.
- length or offset data included in the one or more record descriptor bytes, field descriptor bytes, array descriptor bytes, and/or object descriptor bytes may be used to navigate through the compressed data without decompressing the records, arrays, objects, or fields.
- FIG. 1 shows Compression Apparatus 100 including Storage Unit 102 , RAM 104 , and CPU 106 .
- one or more of a collection of data records, one or more schema, one or more dictionaries, one or more n-gram tables, and one or more token tables may be stored in a hardware Storage Unit 102 in Compression Apparatus 100 .
- RAM 104 in Compression Apparatus 100 may have loaded into it any data stored in Storage Unit 102 , as well as any suitable modules, including Fragmentation Modules, Compression Modules, and Indexing Modules, amongst others.
- Compression Apparatus 100 may include one or more suitable CPUs 106 ,
- FIG. 2 shows Collection Data Table 200 .
- one or more collections may include structured or semi-structured data as shown in Collection Data Table 200 .
- the structured data may contain any number of fields
- the semi-structured data such as data represented using JSON, BSON, YAML or any other suitable format, may contain that may include any suitable number of fields, arrays, or objects.
- Collections may be described using any suitable schema, where suitable schema may define the data structure and the compression method used for one or more fields in the schema.
- one or more fields may include data values that may have a semantic similarity.
- semantically similar data may include first names, last names, date of birth, and citizenship, amongst others.
- a compression apparatus may compress one or more fields using one or more methods suitable for compressing the type of data stored in the field, where the compression apparatus may use custom token tables.
- a compression apparatus may use n-gram compression as a default compression method for any number of fields with data not associated with a desired method of compression.
- one or more data in one or more fields of a collection may include data that may be better compressed after fragmentation. This type of data is typically where fields have multiple values per record, and a compression apparatus may better achieve matching and scoring by de-normalizing those records into multiple record fragments. Examples of data suitable for fragmentation may include full names, addresses, phone numbers and emails, amongst others.
- a compression apparatus may fragment one or more data prior to compression. A compression apparatus may store fragmented data contiguously in the same partition.
- a compression apparatus may use fragmented record identifiers to identify which record they were fragmented from to ensure the system remains aware that the records originate from the same original record in the collection.
- a record may contain an array of data values.
- Arrays may contain zero or more values and array values may have a null value to represent a missing value while preserving the proper order of values.
- a compression apparatus may group one or more data fields as an object. Objects may contain other objects and may be elements in an array. A compression apparatus may further compress objects within a record by including a value that refers the system to another object in the partition with identical values. When a module may output data to other modules in the system, the module may replace the referring object with the actual object values.
- a compression apparatus may compress one or more data in fields representing numbers using known binary compression methods.
- a compression apparatus may compress one or more data in fields representing dates using known Serial Day Number compressions algorithms.
- a compression apparatus may normalize one or more data prior to compression.
- Data suitable for normalization prior to compression may include street suffixes and prefixes, name suffixes and prefixes, and post/pre directional information (i.e. east, north, west, amongst others), amongst others.
- FIG. 3 shows Token Table 300 .
- a compression apparatus may compress fields including data with a suitably semantic similarity using any suitable token table, where suitable token tables may be similar to Token Table 300 .
- the system determines whether the data may match previously encountered data in the token table. In one or more embodiments, if the data does not match, the system may use an alternate compression method instead of token tables. In one or more other embodiments, if the data does not match, the system may update its token table so as to include the data.
- the token table may be updated periodically and stored data may be re-evaluated to determine if compressibility has improved. If the compressibility of one or more data has improved, the system may decompress and re-compress any suitable data.
- the most frequently occurring values may be stored in the lower numbered indices, which may allow for the most frequently used values to be represented with fewer bytes.
- a longer value may be preferred over a shorter value for inclusion in the token table, which may allow for greater compression by eliminating longer values with the same index size as a smaller value.
- a special index value may be reserved to indicate that no token data exists for the data value.
- FIG. 4 shows N-gram Table 400 .
- a compression apparatus may compress fields including data with a suitably semantic similarity using any suitable n-gram table, where suitable n-gram tables may be similar to N-gram Table 400 .
- the system determines whether the data may match previously encountered data in the n-gram table. In one or more embodiments, if the data does not match, the system may use an alternate compression method instead of n-gram tables. In one or more other embodiments, if the data does not match, the system may update its n-gram table so as to include the data.
- the n-gram table may be updated periodically and stored data may be re-evaluated to determine if compressibility has improved. If the compressibility of one or more data has improved, the system may decompress and re-compress any suitable data.
- the most frequently occurring values may be stored in the lower numbered indices, which may allow for the most frequently used values to be represented with fewer bytes.
- a special index value may be reserved to indicate that no n-gram data exists for the data value.
- FIG. 5 shows Record Representation 500 , which may represent compressed data in one or more embodiments.
- each row value in the record index column may include zero or more record descriptor bytes with information about the record, including the length, offset, or the record's location in memory amongst others.
- each data node (array, field, or object) present in the record may include zero or more descriptor bytes, where suitable information about the node may be included, including a node identifier, the length of the stored data, and number of elements of the array if applicable. Following the zero or more node descriptor bytes, any suitable number of bytes may represent the data associated with the record.
- the data may include one or more bits describing the contents of the data including array separation marker bits.
- data in a field associated with a token table may use one or more bits to state whether the information stored in the record is represented in a suitable Token Table, or whether another suitable compression method, such as N-gram compression, was used.
- a system may use length or offset data included in the one or more record descriptor bytes and/or the one or more node (array, object, or field) descriptor bytes to navigate through the compressed data without decompressing the records or nodes (arrays, objects, or fields).
- any suitable module in a system may index or compress data, including one or more search conductors or one or more partitioners in a MEMDB system.
- a compression apparatus employing one or more compression methods here disclosed allow data to be compressed at rates similar to other prominent compression methods while allowing data to be decompressed and/or accessed at the node (array, object, or field) level.
- a compression apparatus employing one or more compression methods here disclosed allow the system to skip individual records and nodes (arrays, objects, or fields) when accessing information in the records.
- a compression apparatus employing one or more compression methods here disclosed allow the system to exit decompression of a record early when the target fields are found.
- Example #1 illustrates a method for compressing names using a compression apparatus.
- a data set includes a collection including one million full name records with 350 unique first names and 300 unique last names represented. The records were fragmented into a first name field and a last name field.
- the individual tokens were then weighted via the product of their frequency and length from highest to lowest. Tokens with a weight less than a certain threshold were discarded to reduce the token table size.
- a token table was then generated for each field by maximizing the aggregate space savings in assigning indices whereby space savings for an individual token is the product of frequency and the sum of its length minus stored index length.
- the number of entries associated with single byte indices was varied from 1 to 255 to inclusive during the maximization procedure.
- the algorithm guarantees that the generated token table is optimal, and the highest savings will go to the single byte stored index entries while subsequent values compress to two or more bytes. Short or infrequent entries may realize no savings and are not be included in the token table. These values revert to another compression method such as n-gram compression.
- Example #2 illustrates a method for compressing text using a compression apparatus.
- n-grams can represent successive sequences of characters, words, or groups or words.
- the text is usually acquired via analyzing a large column of field data in order to achieve columnar compression results in a field by field horizontal compression.
- n-grams were then weighted via the product of their frequency and length from highest to lowest. N-grams with a weight less than a certain threshold were discarded to reduce the n-gram table size.
- a n-gram table was then generated for the field by maximizing the aggregate space savings in assigning indices whereby space savings for an individual n-gram is the product of frequency and the sum of its length minus stored index length.
- the number of entries associated with single byte indices was varied from 1 to 255 to inclusive during the maximization procedure.
- the algorithm guarantees that the generated n-gram table is optimal, and the highest savings will go to the single byte stored index entries while subsequent values compress to two or more bytes. Infrequent entries may realize no savings and are not be included in the n-gram table. These values revert to some other method of basic storage.
- the field data is compressed from beginning to end using a greedy algorithm to replace the largest amount of data as possible with an indexed value until the entire data is consumed.
- Example #3 is a method for compressing semi-structured data in JSON documents using a compression apparatus.
- JSON input documents are compressed using the following schema, with token table compression for Title, FirstName, LastName, NameSuffix and PhoneType fields, Serial Day Number compression for DateOfBirth field and number n-gram compression for PhoneNumber field:
- the input record below requires 266 bytes to be represented in JSON (after removing unnecessary whitespace). After compressing, using the compression methods described in the schema above, the resulting compressed record requires only 44 bytes.
- the input record below requires 108 bytes to be represented in JSON (after removing unnecessary whitespace). After compressing, using the compression methods described in the schema above, the resulting compressed record requires only 13 bytes.
- Example #4 is an example of fragmenting a record.
- the 53rd record of a collection includes data for a couple, Bob and Carol Wilson, having a first and second address.
- the record is fragmented as shown in the following table.
- the record index is maintained to ensure the system remains aware that the records originate from the same original record in the collection.
- the fragmented records further compress the data by including a value that refers the system to the previous record in the partition, i.e. when the system accesses record the name of record 53.2, the value refers the system back to the value for the name in record 53.1.
- the system in Example #4 outputs data to other modules in the system, even in compressed format, the module replaces the referring values for the actual values.
- Example #5 is an example of compression for archiving semi-structured data.
- JSON documents from a document oriented database such as MongoDB, Cassandra, or CouchDB are compressed using a schema that defines all the desired fields, including the unique identifier of each JSON document.
- An index is then created that maps the unique identifier to the compressed record.
- the resulting compressed records and index consume less than 15% of the storage required for the original document-oriented database and each JSON document or select fields of a document can be immediately accessed without decompressing unwanted data.
- Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, GPUs, hardware description languages, or any combination thereof.
- a code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements.
- a code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents.
- Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.
- the functions When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium.
- the steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module which may reside on a computer-readable or processor-readable storage medium.
- a non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another.
- a non-transitory processor-readable storage media may be any available media that may be accessed by a computer.
- non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor.
- Disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
- the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable medium and/or computer-readable medium, which may be incorporated into a computer program product.
Abstract
Disclosed herein are systems and methods for compressing structured or semi-structured data in a horizontal manner achieving compression ratios similar to vertical compression. Collections include structured or semi-structured data include a number of fields and are described using a schema. Fields include information having semantic similarity and are compressed using methods suitable for compressing the type of data. Data of a collection is compressed after fragmentation or may be normalized prior to compression. Data with semantic similarity is compressed using token tables and/or n-gram tables, where higher weighted, consisting of the product of frequency and length, occurring values may be stored in the lower numbered indices of the data table. Records include record descriptor bytes, field descriptor bytes, zero or more array descriptor bytes, zero or more object descriptor bytes, or bytes representing the data associated with the record. Data is indexed or compressed by a suitable module.
Description
- This application is a continuation of U.S. patent application Ser. No. 14/703,622, filed May 4, 2015, entitled “DATA RECORD COMPRESSION WITH PROGRESSIVE AND/OR SELECTIVE DECOMPRESSION,” which is a continuation of U.S. patent application Ser. No. 14/557,900, filed Dec. 2, 2014, issued on May 5, 2015 as U.S. Pat. No. 9,025,892, entitled “DATA RECORD COMPRESSION WITH PROGRESSIVE AND/OR SELECTIVE DECOMPRESSION,” and which is a non-provisional patent application claiming the benefit of U.S. Provisional Patent Application Ser. No. 61/910,873, entitled “DATA RECORD COMPRESSION WITH PROGRESSIVE AND/OR SELECTIVE DECOMPRESSION,” filed Dec. 2, 2013. Each of the above-referenced applications are incorporated by reference herein, in their entireties for all purposes.
- This application is related to U.S. patent application Ser. No. 14/557,794, filed Dec. 2, 2014, entitled “METHOD FOR DISAMBIGUATING FEATURES IN UNSTRUCTURED TEXT”, and U.S. patent application Ser. No. 14/558,300, filed Dec. 2, 2014, entitled “EVENT DETECTION THROUGH TEXT ANALYSIS USING TRAINED EVENT TEMPLATE MODELS”, and U.S. patent application Ser. No. 14/557,807, filed Dec. 2, 2014 entitled “METHOD FOR FACET SEARCHING AND SEARCH SUGGESTIONS”, and U.S. patent application Ser. No. 14/558,254, filed Dec. 2, 2014, entitled “DESIGN AND IMPLEMENTATION OF CLUSTERED IN-MEMORY DATABASE”, and U.S. patent application Ser. No. 14/557,827, filed Dec. 2, 2014, entitled “REAL-TIME DISTRIBUTED IN MEMORY SEARCH ARCHITECTURE”, and U.S. patent application Ser. No. 14/557,951, filed Dec. 2, 2014, entitled “FAULT TOLERANT ARCHITECTURE FOR DISTRIBUTED COMPUTING SYSTEMS”, and U.S. patent application Ser. No. 14/558,009, filed Dec. 2, 2014, entitled “DEPENDENCY MANAGER FOR DATABASES”, and U.S. patent application Ser. No. 14/558,055, filed Dec. 2, 2014, entitled “PLUGGABLE ARCHITECTURE FOR EMBEDDING ANALYTICS IN CLUSTERED IN-MEMORY DATABASES”, and U.S. patent application Ser. No. 14/558,101, filed Dec. 2, 2014, entitled “NON-EXCLUSIONARY SEARCH WITHIN IN-MEMORY DATABASES”, all of which are incorporated herein by reference in their entirety.
- The present disclosure relates in general to data compression and databases, and more specifically to methods of compression for use in in-memory databases as well as document databases.
- Computers are powerful tools of use in storing and providing access to vast amounts of information, while databases are a common mechanism for storing information on computer systems while providing easy access to users. Typically, a database is an organized collection of information stored as “records” having “fields” of information. (e.g., a restaurant database may have a record for each restaurant in a region, where each record contains fields describing characteristics of the restaurant, such as name, address, type of cuisine, and the like).
- Often, databases may use clusters of computers in order to be able to store and access large amounts of data. This may require a large amount of information storage space. Often, compression may be used to reduce the amount of storage space necessary to host the information, but it may increase the computational load significantly as many common compression methods require the entire record or many records to be decompressed every time they are accessed.
- As such, there is a continuing need for improved methods of storing and retrieving data at high speeds at a large scale.
- Disclosed here are methods for compressing structured or semi-structured data, though it should be appreciated that a variety of suitable compression algorithms may be utilized (i.e., no particular compression algorithm is required). System and method embodiments described herein may implement a combination of suitable data compression processes to each field of database, such that a compressed database record achieves a compression ratio comparable to commercially-accepted ratios, while still allowing decompression of the fields to occur only for the records and fields of interest (i.e., only decompressing data records or fields satisfying a database search query). Implementing compression techniques that facilitate selective decompression of records or fields allows for horizontal record-based storage of the compressed data, but also columnar or vertical access to the fields of the data on decompression. This provides the reduced storage benefits of compression, while avoiding much of the compute power and latency associated with decompression when only specific fields are to be decompressed.
- Systems and methods described herein may also implement N-gram compression techniques. Conventionally, N-grams are restricted to compressing only one of chains of letters (successive characters of a string), or to chains of words (successive strings in text). Conventional N-gram compression is unable to compress chains of letters, individual words, and/or chains of words, within a single implementation of such a compression technique. Described herein is the use of N-gram-related compression for columnar compression during record storage, thereby allowing good overall compression, while still providing low-latency access to a single record or a single field within a record, in response to search queries.
- Systems and methods described herein describe embodiments of compression techniques as applying to in-memory databases and document databases. However, it should be appreciated that such techniques and other aspects of the systems and methods may be applied to more general data compression.
- In one embodiment, a computer-implemented method comprises determining, by a computer, a compression technique to apply to one or more data elements received in a set of data elements, wherein the computer uses a schema to determine the compression technique to apply to each data element based on a data type of the data element; compressing, by a computer, a data element using the compression technique defined by the schema, wherein the compression technique compresses the data element such that the data element is individually decompressed when returned in response to a search query; storing, by the computer, each compressed data element in a field of a record that stores data of the data type of the data element; associating, by the computer, a field notation in a reference table for each field according to a schema, wherein the representative notation identifies the data type of the field; querying, by the computer, the database for a set of one or more data elements satisfying a search query received from a search conductor; and decompressing, by the computer, each of the one or more data elements of the one or more data elements satisfying the search query using the compression technique to apply responsive to identifying the set one or more data elements satisfying the search query, wherein each data element not satisfying the search query remains compressed.
- In another embodiment, a computing system comprises one or more nodes storing one or more collections, each collection comprising a set of one or more records, each record comprising a set of fields storing data; and a compression processor compressing one or more of the fields according to a schema that is associated with a collection.
- Numerous other aspects, features and benefits of the present disclosure may be made apparent from the following detailed description.
- The present disclosure can be better understood by referring to the following figures. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure. In the figures, reference numerals designate corresponding parts throughout the different views.
-
FIG. 1 illustrates a data compression apparatus according to an exemplary embodiment. -
FIG. 2 illustrates a structured data table according to an exemplary embodiment. -
FIG. 3 illustrates a token table according to an exemplary embodiment. -
FIG. 4 illustrates a n-gram table according to an exemplary embodiment. -
FIG. 5 illustrates a table describing compressed records according to an exemplary embodiment. - As used herein, the following terms have the following definitions:
- “Node” refers to a computer hardware configuration suitable for running one or more modules.
- “Cluster” refers to a set of one or more nodes.
- “Module” refers to a computer software component suitable for carrying out one or more defined tasks.
- “Collection” refers to a discrete set of records.
- “Record” refers to one or more pieces of information that may be handled as a unit.
- “Field” refers to one data element within a record.
- “Object” refers to a logical collection of fields within a data record.
- “Array” refers to an ordered list of data values within a record.
- “Node” refers to a field, object, or array within a record.
- “Partition” refers to an arbitrarily delimited portion of records of a collection.
- “Schema” refers to data describing one or more characteristics of a collection, record, or field.
- “Compress” may refer to reducing the amount of electronic data needed to represent a value.
- “Dictionary” may refer to any computerized list suitable for use as a value reference.
- “Token Table” refers to a table defining one or more simpler values for one or more other more complex values.
- “N-gram” refers to N successive integral units of data which can be characters, words, or groups of words where N is greater than or equal to 1—i.e., in the sentence “The quick brown fox jumped over the lazy dog,” “the,” “e,” “he,” and “brown fox” are all valid N-grams.
- “N-gram Table” refers to a table defining one or more simpler values for one or more other more complex values.
- “Search Conductor”, or “S.C.”, refers to a module configured to at least run one or more queries on a partition and return the search results to one or more search managers.
- “Partitioner” refers to a module configured to at least divide one or more collections into one or more partitions.
- “Database” refers to any system including any combination of clusters and modules suitable for storing one or more collections and suitable to process one or more queries.
- “Query” refers to a request to retrieve information from one or more suitable partitions or databases.
- “Memory” refers to any hardware component suitable for storing information and retrieving said information at a sufficiently high speed.
- “Fragment” refers to separating records into smaller records until a desired level of granularity is achieved.
- “JSON” refers to the JavaScript Object Notation, a data-interchange format.
- “BSON” refers to Binary JSON, a data-interchange format.
- “YAML” refers to the coding language “YAML Ain't Markup Language,” a data-interchange format.
- “Document” refers to a group of structured or semi-structured information.
- “Document Database” refers to a document-oriented database, designed for storing, retrieving, and managing document-oriented information.
- The present disclosure describes methods for compressing structured or semi-structured data. In one or more embodiments, one or more collections may include structured or semi-structured data that may include any number of records and any suitable number of fields, where the collections may be described using any suitable schema that may define the data structure and the compression method used for one or more fields.
- In one or more embodiments, one or more fields may include information that may have a semantic similarity. In one or more embodiments, the fields may be compressed using one or more methods suitable for compressing the type of data stored in the field, where token tables, N-gram compression, serial day number compression, binary number compression, or any other suitable method may be used.
- In one or more embodiments, one or more data in one or more records of a collection may include data that may be better compressed after fragmentation and fragmented data may be stored contiguously in the same partition. In one or more embodiments, fragmented record identifiers may be used to identify which record they were fragmented from to ensure the system remains aware that the records originate from the same original record in the collection. Data that would be duplicated across fragments will only have a single representation in the compressed form with an anchor to which other fragments can refer.
- In one or more embodiments, a record may contain an array of data values. Arrays may contain zero or more values. Values may be fields, objects, or other arrays.
- In one or more embodiments, one or more data values may be grouped as an object. Objects may contain fields, other objects, or arrays, and may be elements of other objects or arrays. Objects within a record may be compressed further by including an anchor value that refers the system to another object or fragment in the partition with identical values. When a module may output data to other modules in the system, the module may replace the referring object with the actual object values.
- In one or more embodiments, input records may be semi-structured data and may be represented using JSON, BSON, YAML or any other suitable data format.
- In one or more embodiments, one or more data fields may be normalized prior to compression.
- In one or more embodiments, fields including data with a suitably semantic similarity may be compressed using any suitable token table. When one or more records may be added to a field with an associated token table the system determines whether the data may match previously encountered data in the token table. In one or more embodiments, if the data does not match, the system may use an alternate compression method or may update the token table. In other embodiments, the token table may be updated periodically.
- In one or more embodiments, fields including data with a suitably semantic similarity may be compressed using any suitable n-gram table. When one or more records may be added to a field with an associated n-gram table the system determines whether the data may match previously encountered data in the n-gram table. In one or more embodiments, if the data does not match, the system may use an alternate compression method or may update the n-gram table. In other embodiments, the n-gram table may be updated periodically.
- In one or more embodiments, the most frequently occurring values may be stored in the lower numbered indices, which may allow for the most frequently used values to be represented with fewer bytes.
- In one or more embodiments, a longer value may be preferred over a shorter value for inclusion in the token table, which may allow for greater compression by eliminating longer values with the same index size as a smaller value.
- In one or more embodiments, a longer value may be preferred over a shorter value for inclusion in the n-gram table, which may allow for greater compression by eliminating longer values with the same index size as a smaller value.
- In one or more embodiments, records may include zero or more record descriptor bytes, any suitable number of field descriptor bytes, any suitable number of array descriptor bytes, any suitable number of object descriptor bytes, and any suitable number of bytes representing the data associated with the record.
- In one or more embodiments, data in a field associated with a token table may use one or more bits to state whether the information stored in the record is compressed using the compression method defined in the schema or whether another compression method, such as n-gram compression, was used.
- In one or more embodiments, length or offset data included in the one or more record descriptor bytes, field descriptor bytes, array descriptor bytes, and/or object descriptor bytes may be used to navigate through the compressed data without decompressing the records, arrays, objects, or fields.
-
FIG. 1 showsCompression Apparatus 100 includingStorage Unit 102,RAM 104, andCPU 106. - In one or more embodiments, one or more of a collection of data records, one or more schema, one or more dictionaries, one or more n-gram tables, and one or more token tables may be stored in a
hardware Storage Unit 102 inCompression Apparatus 100.RAM 104 inCompression Apparatus 100 may have loaded into it any data stored inStorage Unit 102, as well as any suitable modules, including Fragmentation Modules, Compression Modules, and Indexing Modules, amongst others. In one or more embodiments,Compression Apparatus 100 may include one or moresuitable CPUs 106, -
FIG. 2 shows Collection Data Table 200. - In one or more embodiments, one or more collections may include structured or semi-structured data as shown in Collection Data Table 200. In one or more embodiments, the structured data may contain any number of fields, and the semi-structured data, such as data represented using JSON, BSON, YAML or any other suitable format, may contain that may include any suitable number of fields, arrays, or objects. Collections may be described using any suitable schema, where suitable schema may define the data structure and the compression method used for one or more fields in the schema.
- In one or more embodiments, one or more fields may include data values that may have a semantic similarity. In one or more embodiments, semantically similar data may include first names, last names, date of birth, and citizenship, amongst others. In one or more embodiments, a compression apparatus may compress one or more fields using one or more methods suitable for compressing the type of data stored in the field, where the compression apparatus may use custom token tables. In one or more embodiments, a compression apparatus may use n-gram compression as a default compression method for any number of fields with data not associated with a desired method of compression.
- In one or more embodiments, one or more data in one or more fields of a collection may include data that may be better compressed after fragmentation. This type of data is typically where fields have multiple values per record, and a compression apparatus may better achieve matching and scoring by de-normalizing those records into multiple record fragments. Examples of data suitable for fragmentation may include full names, addresses, phone numbers and emails, amongst others. In one or more embodiments, a compression apparatus may fragment one or more data prior to compression. A compression apparatus may store fragmented data contiguously in the same partition. In one or more embodiments, a compression apparatus may use fragmented record identifiers to identify which record they were fragmented from to ensure the system remains aware that the records originate from the same original record in the collection.
- In one or more embodiments, a record may contain an array of data values. Arrays may contain zero or more values and array values may have a null value to represent a missing value while preserving the proper order of values.
- In one or more embodiments, a compression apparatus may group one or more data fields as an object. Objects may contain other objects and may be elements in an array. A compression apparatus may further compress objects within a record by including a value that refers the system to another object in the partition with identical values. When a module may output data to other modules in the system, the module may replace the referring object with the actual object values.
- In one or more embodiments, a compression apparatus may compress one or more data in fields representing numbers using known binary compression methods.
- In one or more embodiments, a compression apparatus may compress one or more data in fields representing dates using known Serial Day Number compressions algorithms.
- In one or more embodiments, a compression apparatus may normalize one or more data prior to compression. Data suitable for normalization prior to compression may include street suffixes and prefixes, name suffixes and prefixes, and post/pre directional information (i.e. east, north, west, amongst others), amongst others.
-
FIG. 3 shows Token Table 300. - In one or more embodiments, a compression apparatus may compress fields including data with a suitably semantic similarity using any suitable token table, where suitable token tables may be similar to Token Table 300.
- In one or more embodiments, when one or more records may be added to a field with an associated token table the system determines whether the data may match previously encountered data in the token table. In one or more embodiments, if the data does not match, the system may use an alternate compression method instead of token tables. In one or more other embodiments, if the data does not match, the system may update its token table so as to include the data.
- In one or more embodiments, the token table may be updated periodically and stored data may be re-evaluated to determine if compressibility has improved. If the compressibility of one or more data has improved, the system may decompress and re-compress any suitable data.
- In one or more embodiments, the most frequently occurring values may be stored in the lower numbered indices, which may allow for the most frequently used values to be represented with fewer bytes.
- In one or more embodiments, a longer value may be preferred over a shorter value for inclusion in the token table, which may allow for greater compression by eliminating longer values with the same index size as a smaller value.
- In one or more embodiments a special index value may be reserved to indicate that no token data exists for the data value.
-
FIG. 4 shows N-gram Table 400. - In one or more embodiments, a compression apparatus may compress fields including data with a suitably semantic similarity using any suitable n-gram table, where suitable n-gram tables may be similar to N-gram Table 400.
- In one or more embodiments, when one or more records may be added to a field with an associated n-gram table the system determines whether the data may match previously encountered data in the n-gram table. In one or more embodiments, if the data does not match, the system may use an alternate compression method instead of n-gram tables. In one or more other embodiments, if the data does not match, the system may update its n-gram table so as to include the data.
- In one or more embodiments, the n-gram table may be updated periodically and stored data may be re-evaluated to determine if compressibility has improved. If the compressibility of one or more data has improved, the system may decompress and re-compress any suitable data.
- In one or more embodiments, the most frequently occurring values may be stored in the lower numbered indices, which may allow for the most frequently used values to be represented with fewer bytes.
- In one or more embodiments a special index value may be reserved to indicate that no n-gram data exists for the data value.
-
FIG. 5 shows Record Representation 500, which may represent compressed data in one or more embodiments. - In Record Representation 500, each row value in the record index column may include zero or more record descriptor bytes with information about the record, including the length, offset, or the record's location in memory amongst others. In one or more embodiments, each data node (array, field, or object) present in the record may include zero or more descriptor bytes, where suitable information about the node may be included, including a node identifier, the length of the stored data, and number of elements of the array if applicable. Following the zero or more node descriptor bytes, any suitable number of bytes may represent the data associated with the record. In one or more embodiments, the data may include one or more bits describing the contents of the data including array separation marker bits.
- In one or more embodiments, data in a field associated with a token table may use one or more bits to state whether the information stored in the record is represented in a suitable Token Table, or whether another suitable compression method, such as N-gram compression, was used.
- In one or more embodiments, a system may use length or offset data included in the one or more record descriptor bytes and/or the one or more node (array, object, or field) descriptor bytes to navigate through the compressed data without decompressing the records or nodes (arrays, objects, or fields).
- In one or more embodiments, any suitable module in a system may index or compress data, including one or more search conductors or one or more partitioners in a MEMDB system.
- In one or more embodiments, a compression apparatus employing one or more compression methods here disclosed allow data to be compressed at rates similar to other prominent compression methods while allowing data to be decompressed and/or accessed at the node (array, object, or field) level.
- In one or more embodiments, a compression apparatus employing one or more compression methods here disclosed allow the system to skip individual records and nodes (arrays, objects, or fields) when accessing information in the records.
- In one or more embodiments, a compression apparatus employing one or more compression methods here disclosed allow the system to exit decompression of a record early when the target fields are found.
-
Example # 1 illustrates a method for compressing names using a compression apparatus. - In this example, a data set includes a collection including one million full name records with 350 unique first names and 300 unique last names represented. The records were fragmented into a first name field and a last name field.
- The individual tokens were then weighted via the product of their frequency and length from highest to lowest. Tokens with a weight less than a certain threshold were discarded to reduce the token table size.
- A token table was then generated for each field by maximizing the aggregate space savings in assigning indices whereby space savings for an individual token is the product of frequency and the sum of its length minus stored index length.
- The number of entries associated with single byte indices was varied from 1 to 255 to inclusive during the maximization procedure.
- The algorithm guarantees that the generated token table is optimal, and the highest savings will go to the single byte stored index entries while subsequent values compress to two or more bytes. Short or infrequent entries may realize no savings and are not be included in the token table. These values revert to another compression method such as n-gram compression.
-
Example # 2 illustrates a method for compressing text using a compression apparatus. - In this example, a large body of text was analyzed for frequency of n-grams where n-grams can represent successive sequences of characters, words, or groups or words. The text is usually acquired via analyzing a large column of field data in order to achieve columnar compression results in a field by field horizontal compression.
- The individual n-grams were then weighted via the product of their frequency and length from highest to lowest. N-grams with a weight less than a certain threshold were discarded to reduce the n-gram table size.
- A n-gram table was then generated for the field by maximizing the aggregate space savings in assigning indices whereby space savings for an individual n-gram is the product of frequency and the sum of its length minus stored index length.
- The number of entries associated with single byte indices was varied from 1 to 255 to inclusive during the maximization procedure.
- The algorithm guarantees that the generated n-gram table is optimal, and the highest savings will go to the single byte stored index entries while subsequent values compress to two or more bytes. Infrequent entries may realize no savings and are not be included in the n-gram table. These values revert to some other method of basic storage.
- An example of some of the n-grams generated in the table via this method is as follows:
-
Index N-gram 1 e 11 th 47 and 800 out of - During compression the field data is compressed from beginning to end using a greedy algorithm to replace the largest amount of data as possible with an indexed value until the entire data is consumed.
- Example #3 is a method for compressing semi-structured data in JSON documents using a compression apparatus.
- In this example, JSON input documents are compressed using the following schema, with token table compression for Title, FirstName, LastName, NameSuffix and PhoneType fields, Serial Day Number compression for DateOfBirth field and number n-gram compression for PhoneNumber field:
-
{ “collection”: “Example2”, “record”: [ {“id”: “1”, “type”: “FIELD”, “name”: “Title”, “compressionMethod”: “TitleNameTokenTable”}, {“id”: “2”, “type”: “FIELD”, “name”: “FirstName”, “compressionMethod”: “FirstNameTokenTable”}, {“id”: “3”, “type”: “FIELD”, “name”: “LastName”, “compressionMethod”: “LastNameTokenTable”}, {“id”: “4”, “type”: “FIELD”, “name”: “NameSuffix”, “compressionMethod”: “NameSuffixNameTokenTable”}, {“id”: “5”, “type”: “FIELD”, “name”: “DateOfBirth”, “compressionMethod”: “SerialDayNumber”}, {“id”: “6”, “type”: “ARRAY”, “name”: “Phones”, “array”: [ {“id”: “7”, “type”: “STRUCT”, “struct”: [ {“id”: “8”, “type”: “FIELD”, “name”: “PhoneType”, “compressionMethod”: “PhoneTypeTokenTable”}, {“id”: “9”, “type”: “FIELD”, “name”: “PhoneNumber”, “compressionMethod”: “NumberNgramTable”}, ]} ]} ]} - The input record below requires 266 bytes to be represented in JSON (after removing unnecessary whitespace). After compressing, using the compression methods described in the schema above, the resulting compressed record requires only 44 bytes.
-
{ “Title”: “Dr.”, “FirstName”: “Johnathan”, “LastName”: “Patterson”, “NameSuffix”: “M.D.”, “DateOfBirth”: “19600101”, “Phones”: [ {“PhoneType”: “Office”, “PhoneNumber”: “1234567890”}, {“PhoneType”: “Mobile”, “PhoneNumber”: “2345678901”}, {“PhoneType”: “Home”, “PhoneNumber”: “3456789012”} ]} - The input record below requires 108 bytes to be represented in JSON (after removing unnecessary whitespace). After compressing, using the compression methods described in the schema above, the resulting compressed record requires only 13 bytes.
-
{ “Title”: “Dr.”, “FirstName”: “Johnathan”, “LastName”: “Patterson”, “NameSuffix”: “M.D.”, “DateOfBirth”: “19600101” } - Example #4 is an example of fragmenting a record. In this example, the 53rd record of a collection includes data for a couple, Bob and Carol Wilson, having a first and second address. In this example, the record is fragmented as shown in the following table.
-
Record Index Name Address Original 53 Bob and Carol Address 1, Address 2Wilson Fragmented 53.1 Bob Address 1 53.2 Bob Address 2 53.3 Carol Address 1 53.4 Carol Address 2 - The record index is maintained to ensure the system remains aware that the records originate from the same original record in the collection. In this example, the fragmented records further compress the data by including a value that refers the system to the previous record in the partition, i.e. when the system accesses record the name of record 53.2, the value refers the system back to the value for the name in record 53.1. When the system in Example #4 outputs data to other modules in the system, even in compressed format, the module replaces the referring values for the actual values.
- Example #5 is an example of compression for archiving semi-structured data. In this example, JSON documents from a document oriented database such as MongoDB, Cassandra, or CouchDB are compressed using a schema that defines all the desired fields, including the unique identifier of each JSON document. An index is then created that maps the unique identifier to the compressed record. The resulting compressed records and index consume less than 15% of the storage required for the original document-oriented database and each JSON document or select fields of a document can be immediately accessed without decompressing unwanted data.
- The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
- Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, GPUs, hardware description languages, or any combination thereof. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.
- The actual software code or specialized control hardware used to implement these systems and methods is not limiting of the invention. Thus, the operation and behavior of the systems and methods were described without reference to the specific software code being understood that software and control hardware can be designed to implement the systems and methods based on the description herein.
- When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module which may reside on a computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. A non-transitory processor-readable storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable medium and/or computer-readable medium, which may be incorporated into a computer program product.
- The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.
Claims (20)
1. A method of compression, the method comprising:
storing, by a processor, a collection in a node, wherein the collection contains a plurality of records, wherein each of the records contains a plurality of fields, wherein each of the fields stores a data element;
accessing, by the processor, at least one of the data elements in at least one of the fields; and
compressing, by the processor, the at least one of the data elements in the at least one of the fields according to a schema associated with the collection.
2. The method of claim 1 , wherein the at least one of the fields is associated with an n-gram table, wherein the compressing is based on the n-gram table.
3. The method of claim 2 , further comprising:
modifying, by the processor, the n-gram table upon an identification of first data which does not match second data stored in the n-gram table.
4. The method of claim 1 , wherein the compressing is based on an identification of first data which does not match second data stored in a token table.
5. The method of claim 4 , wherein the token table is associated with the at least one of the fields.
6. The method of claim 1 , wherein the compressing is based on a data type of the at least one element.
7. The method of claim 1 , wherein the compressing is such that the at least one data element is able to be individually decompressed when returned in response to a search query.
8. The method of claim 1 , further comprising:
modifying, by the processor, a reference value stored in a first record based on a second record, wherein the records contain the first record and the second record, wherein the reference value is associated with an object.
9. The method of claim 1 , further comprising:
outputting, by the processor, a value of an object, wherein the object is stored in at least one of the records, wherein the object comprises at least two of the fields.
10. The method of claim 1 , wherein at least two of the fields are grouped according to the schema, wherein the at least two of the fields are associated with a reference value stored in a table, wherein the records comprise a first record and a second record, wherein the first record comprises the at least two of the fields, wherein the second record stores the reference value.
11. A method of decompression, the method comprising:
accessing, by a processor, a storage storing a collection, wherein the collection contains a plurality of records, wherein each of the records contains a plurality of fields, wherein each of the fields stores a data element, wherein at least one of the data elements in at least one of the fields is compressed according to a schema associated with the collection;
initiating, by the processor, a decompression of the at least one of the data elements.
12. The method of claim 11 , wherein the at least one of the fields is associated with an n-gram table, wherein at least one of the data elements in at least one of the fields is compressed based on the n-gram table.
13. The method of claim 12 , wherein the at least one of the data elements in the at least one of the fields is compressed based on a modification of the n-gram table upon an identification of first data which does not match second data stored in the n-gram table.
14. The method of claim 11 , wherein the at least one of the data elements in the at least one of the fields is compressed based on an identification of first data which does not match second data stored in a token table.
15. The method of claim 14 , wherein the token table is associated with the at least one of the fields.
16. The method of claim 11 , wherein the at least one of the data elements in the at least one of the fields is compressed based on a data type of the at least one element.
17. The method of claim 11 , wherein the decompression comprises an individual decompression of the at least one data element, wherein the individual decompression is responsive to a search query.
18. The method of claim 11 , wherein the decompression is based on the at least one of the data elements in the at least one of the fields being compressed based on a modification of a reference value stored in a first record based on a second record, wherein the records contain the first record and the second record, wherein the reference value is associated with an object.
19. The method of claim 11 , herein the decompression is based on the at least one of the data elements in the at least one of the fields being compressed based on an output of a value of an object, wherein the object is stored in at least one of the records, wherein the object comprises at least two of the fields.
20. The method of claim 11 , wherein at least two of the fields are grouped according to the schema, wherein the at least two of the fields are associated with a reference value stored in a table, wherein the records comprise a first record and a second record, wherein the first record comprises the at least two of the fields, wherein the second record stores the reference value.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/066,824 US20160196277A1 (en) | 2013-12-02 | 2016-03-10 | Data record compression with progressive and/or selective decompression |
Applications Claiming Priority (13)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201361910873P | 2013-12-02 | 2013-12-02 | |
US14/558,009 US9348573B2 (en) | 2013-12-02 | 2014-12-02 | Installation and fault handling in a distributed system utilizing supervisor and dependency manager nodes |
US14/557,951 US9201744B2 (en) | 2013-12-02 | 2014-12-02 | Fault tolerant architecture for distributed computing systems |
US14/557,900 US9025892B1 (en) | 2013-12-02 | 2014-12-02 | Data record compression with progressive and/or selective decomposition |
US14/558,254 US9430547B2 (en) | 2013-12-02 | 2014-12-02 | Implementation of clustered in-memory database |
US14/558,101 US9355152B2 (en) | 2013-12-02 | 2014-12-02 | Non-exclusionary search within in-memory databases |
US14/558,055 US9659108B2 (en) | 2013-12-02 | 2014-12-02 | Pluggable architecture for embedding analytics in clustered in-memory databases |
US14/557,807 US9424294B2 (en) | 2013-12-02 | 2014-12-02 | Method for facet searching and search suggestions |
US14/558,300 US9177254B2 (en) | 2013-12-02 | 2014-12-02 | Event detection through text analysis using trained event template models |
US14/557,794 US9239875B2 (en) | 2013-12-02 | 2014-12-02 | Method for disambiguated features in unstructured text |
US14/557,827 US9223875B2 (en) | 2013-12-02 | 2014-12-02 | Real-time distributed in memory search architecture |
US14/703,622 US9710517B2 (en) | 2013-12-02 | 2015-05-04 | Data record compression with progressive and/or selective decomposition |
US15/066,824 US20160196277A1 (en) | 2013-12-02 | 2016-03-10 | Data record compression with progressive and/or selective decompression |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/703,622 Continuation US9710517B2 (en) | 2013-12-02 | 2015-05-04 | Data record compression with progressive and/or selective decomposition |
Publications (1)
Publication Number | Publication Date |
---|---|
US20160196277A1 true US20160196277A1 (en) | 2016-07-07 |
Family
ID=53001762
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/557,900 Active - Reinstated US9025892B1 (en) | 2013-12-02 | 2014-12-02 | Data record compression with progressive and/or selective decomposition |
US14/703,622 Active 2035-06-01 US9710517B2 (en) | 2013-12-02 | 2015-05-04 | Data record compression with progressive and/or selective decomposition |
US15/066,824 Abandoned US20160196277A1 (en) | 2013-12-02 | 2016-03-10 | Data record compression with progressive and/or selective decompression |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/557,900 Active - Reinstated US9025892B1 (en) | 2013-12-02 | 2014-12-02 | Data record compression with progressive and/or selective decomposition |
US14/703,622 Active 2035-06-01 US9710517B2 (en) | 2013-12-02 | 2015-05-04 | Data record compression with progressive and/or selective decomposition |
Country Status (1)
Country | Link |
---|---|
US (3) | US9025892B1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108228843A (en) * | 2018-01-09 | 2018-06-29 | 闫健 | A kind of handout compression transmission and restoring method based on internet |
US10565182B2 (en) * | 2015-11-23 | 2020-02-18 | Microsoft Technology Licensing, Llc | Hardware LZMA compressor |
Families Citing this family (30)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10223550B2 (en) * | 2013-11-06 | 2019-03-05 | Vmware, Inc. | Generating canonical representations of JSON documents |
US9984427B2 (en) | 2013-12-02 | 2018-05-29 | Qbase, LLC | Data ingestion module for event detection and increased situational awareness |
US9230041B2 (en) | 2013-12-02 | 2016-01-05 | Qbase, LLC | Search suggestions of related entities based on co-occurrence and/or fuzzy-score matching |
US9025892B1 (en) | 2013-12-02 | 2015-05-05 | Qbase, LLC | Data record compression with progressive and/or selective decomposition |
US9348573B2 (en) * | 2013-12-02 | 2016-05-24 | Qbase, LLC | Installation and fault handling in a distributed system utilizing supervisor and dependency manager nodes |
US9424294B2 (en) | 2013-12-02 | 2016-08-23 | Qbase, LLC | Method for facet searching and search suggestions |
US9547701B2 (en) | 2013-12-02 | 2017-01-17 | Qbase, LLC | Method of discovering and exploring feature knowledge |
US9424524B2 (en) | 2013-12-02 | 2016-08-23 | Qbase, LLC | Extracting facts from unstructured text |
US9922032B2 (en) | 2013-12-02 | 2018-03-20 | Qbase, LLC | Featured co-occurrence knowledge base from a corpus of documents |
CN106164890A (en) | 2013-12-02 | 2016-11-23 | 丘贝斯有限责任公司 | For the method eliminating the ambiguity of the feature in non-structured text |
US9223875B2 (en) | 2013-12-02 | 2015-12-29 | Qbase, LLC | Real-time distributed in memory search architecture |
US9544361B2 (en) | 2013-12-02 | 2017-01-10 | Qbase, LLC | Event detection through text analysis using dynamic self evolving/learning module |
US9208204B2 (en) | 2013-12-02 | 2015-12-08 | Qbase, LLC | Search suggestions using fuzzy-score matching and entity co-occurrence |
US9336280B2 (en) | 2013-12-02 | 2016-05-10 | Qbase, LLC | Method for entity-driven alerts based on disambiguated features |
US9355152B2 (en) | 2013-12-02 | 2016-05-31 | Qbase, LLC | Non-exclusionary search within in-memory databases |
US9201744B2 (en) | 2013-12-02 | 2015-12-01 | Qbase, LLC | Fault tolerant architecture for distributed computing systems |
US9659108B2 (en) | 2013-12-02 | 2017-05-23 | Qbase, LLC | Pluggable architecture for embedding analytics in clustered in-memory databases |
US9619571B2 (en) | 2013-12-02 | 2017-04-11 | Qbase, LLC | Method for searching related entities through entity co-occurrence |
US9223833B2 (en) | 2013-12-02 | 2015-12-29 | Qbase, LLC | Method for in-loop human validation of disambiguated features |
US9177262B2 (en) | 2013-12-02 | 2015-11-03 | Qbase, LLC | Method of automated discovery of new topics |
US9317565B2 (en) | 2013-12-02 | 2016-04-19 | Qbase, LLC | Alerting system based on newly disambiguated features |
US9542477B2 (en) | 2013-12-02 | 2017-01-10 | Qbase, LLC | Method of automated discovery of topics relatedness |
US9430547B2 (en) | 2013-12-02 | 2016-08-30 | Qbase, LLC | Implementation of clustered in-memory database |
JP2015115652A (en) * | 2013-12-09 | 2015-06-22 | キヤノン株式会社 | Information processing apparatus, information processing method and program |
US9355114B1 (en) | 2014-06-25 | 2016-05-31 | Groupon, Inc. | Graph-based compression of data records |
US10275426B1 (en) * | 2015-09-22 | 2019-04-30 | Amazon Technologies, Inc. | Dynamic kerning pair reduction for digital font rendering |
US20170212698A1 (en) * | 2016-01-22 | 2017-07-27 | Samsung Electronics Co., Ltd. | Computing system with cache storing mechanism and method of operation thereof |
US10684993B2 (en) | 2016-06-15 | 2020-06-16 | International Business Machines Corporation | Selective compression of unstructured data |
US10366026B1 (en) * | 2016-12-23 | 2019-07-30 | Amazon Technologies, Inc. | Random access to decompressed blocks |
US11579807B2 (en) | 2021-04-08 | 2023-02-14 | Syncadd Systems, Inc. | Efficiently accessing, storing and transmitting data elements |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020165847A1 (en) * | 2001-05-02 | 2002-11-07 | Mccartney Jason | Logical semantic compression |
US20070250519A1 (en) * | 2006-04-25 | 2007-10-25 | Fineberg Samuel A | Distributed differential store with non-distributed objects and compression-enhancing data-object routing |
US20090089626A1 (en) * | 2007-09-28 | 2009-04-02 | Arm Limited | Techniques for generating a trace stream for a data processing apparatus |
US8345998B2 (en) * | 2010-08-10 | 2013-01-01 | Xerox Corporation | Compression scheme selection based on image data type and user selections |
US8423522B2 (en) * | 2011-01-04 | 2013-04-16 | International Business Machines Corporation | Query-aware compression of join results |
US20140022100A1 (en) * | 2000-10-03 | 2014-01-23 | Realtime Data LLC DBA IXO | Methods for encoding and decoding data |
US8782018B2 (en) * | 2011-06-03 | 2014-07-15 | Samsung Electronics Co., Ltd. | Storage device and data processing device utilizing determined dictionary compression |
Family Cites Families (139)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5894556A (en) | 1996-03-21 | 1999-04-13 | Mpath Interactive, Inc. | Network match maker matching requesters based on communication attribute between the requesters |
US6178529B1 (en) | 1997-11-03 | 2001-01-23 | Microsoft Corporation | Method and system for resource monitoring of disparate resources in a server cluster |
JP4183311B2 (en) | 1997-12-22 | 2008-11-19 | 株式会社リコー | Document annotation method, annotation device, and recording medium |
US6353926B1 (en) | 1998-07-15 | 2002-03-05 | Microsoft Corporation | Software update notification |
US6266781B1 (en) | 1998-07-20 | 2001-07-24 | Academia Sinica | Method and apparatus for providing failure detection and recovery with predetermined replication style for distributed applications in a network |
US6226635B1 (en) | 1998-08-14 | 2001-05-01 | Microsoft Corporation | Layered query management |
US6338092B1 (en) | 1998-09-24 | 2002-01-08 | International Business Machines Corporation | Method, system and computer program for replicating data in a distributed computed environment |
GB9821787D0 (en) | 1998-10-06 | 1998-12-02 | Data Limited | Apparatus for classifying or processing data |
US6959300B1 (en) | 1998-12-10 | 2005-10-25 | At&T Corp. | Data compression method and apparatus |
US7099898B1 (en) | 1999-08-12 | 2006-08-29 | International Business Machines Corporation | Data access system |
US6691108B2 (en) | 1999-12-14 | 2004-02-10 | Nec Corporation | Focused search engine and method |
JP3524846B2 (en) | 2000-06-29 | 2004-05-10 | 株式会社Ssr | Document feature extraction method and apparatus for text mining |
US6738759B1 (en) | 2000-07-07 | 2004-05-18 | Infoglide Corporation, Inc. | System and method for performing similarity searching using pointer optimization |
US7813915B2 (en) | 2000-09-25 | 2010-10-12 | Fujitsu Limited | Apparatus for reading a plurality of documents and a method thereof |
US6832373B2 (en) | 2000-11-17 | 2004-12-14 | Bitfone Corporation | System and method for updating and distributing information |
US6691109B2 (en) | 2001-03-22 | 2004-02-10 | Turbo Worx, Inc. | Method and apparatus for high-performance sequence comparison |
GB2374687A (en) | 2001-04-19 | 2002-10-23 | Ibm | Managing configuration changes in a data processing system |
US6961723B2 (en) | 2001-05-04 | 2005-11-01 | Sun Microsystems, Inc. | System and method for determining relevancy of query responses in a distributed network search mechanism |
US20030028869A1 (en) | 2001-08-02 | 2003-02-06 | Drake Daniel R. | Method and computer program product for integrating non-redistributable software applications in a customer driven installable package |
US6954456B2 (en) | 2001-12-14 | 2005-10-11 | At & T Corp. | Method for content-aware redirection and content renaming |
US6829606B2 (en) | 2002-02-14 | 2004-12-07 | Infoglide Software Corporation | Similarity search engine for use with relational databases |
US7421478B1 (en) | 2002-03-07 | 2008-09-02 | Cisco Technology, Inc. | Method and apparatus for exchanging heartbeat messages and configuration information between nodes operating in a master-slave configuration |
US20110099163A1 (en) | 2002-04-05 | 2011-04-28 | Envirospectives Corporation | System and method for indexing, organizing, storing and retrieving environmental information |
US6817558B1 (en) | 2002-04-23 | 2004-11-16 | Uop Llc | Parallel sizing, dosing and transfer assembly and method of use |
US8015143B2 (en) | 2002-05-22 | 2011-09-06 | Estes Timothy W | Knowledge discovery agent system and method |
US20040010502A1 (en) | 2002-07-12 | 2004-01-15 | Bomfim Joanes Depaula | In-memory database for high performance, parallel transaction processing |
US7570262B2 (en) | 2002-08-08 | 2009-08-04 | Reuters Limited | Method and system for displaying time-series data and correlated events derived from text mining |
US7249312B2 (en) | 2002-09-11 | 2007-07-24 | Intelligent Results | Attribute scoring for unstructured content |
US8090717B1 (en) | 2002-09-20 | 2012-01-03 | Google Inc. | Methods and apparatus for ranking documents |
US20100100437A1 (en) | 2002-09-24 | 2010-04-22 | Google, Inc. | Suggesting and/or providing ad serving constraint information |
US7058846B1 (en) | 2002-10-17 | 2006-06-06 | Veritas Operating Corporation | Cluster failover for storage management services |
US20040205064A1 (en) | 2003-04-11 | 2004-10-14 | Nianjun Zhou | Adaptive search employing entropy based quantitative information measurement |
US7139752B2 (en) | 2003-05-30 | 2006-11-21 | International Business Machines Corporation | System, method and computer program product for performing unstructured information management and automatic text analysis, and providing multiple document views derived from different document tokenizations |
US7543174B1 (en) | 2003-09-24 | 2009-06-02 | Symantec Operating Corporation | Providing high availability for an application by rapidly provisioning a node and failing over to the node |
US8244725B2 (en) | 2004-03-10 | 2012-08-14 | Iron Mountain Incorporated | Method and apparatus for improved relevance of search results |
US9009153B2 (en) | 2004-03-31 | 2015-04-14 | Google Inc. | Systems and methods for identifying a named entity |
US7818615B2 (en) | 2004-09-16 | 2010-10-19 | Invensys Systems, Inc. | Runtime failure management of redundantly deployed hosts of a supervisory process control data acquisition facility |
US20080077570A1 (en) | 2004-10-25 | 2008-03-27 | Infovell, Inc. | Full Text Query and Search Systems and Method of Use |
US7403945B2 (en) | 2004-11-01 | 2008-07-22 | Sybase, Inc. | Distributed database system providing data and space management methodology |
US7739270B2 (en) | 2004-12-07 | 2010-06-15 | Microsoft Corporation | Entity-specific tuned searching |
US20060179026A1 (en) | 2005-02-04 | 2006-08-10 | Bechtel Michael E | Knowledge discovery tool extraction and integration |
US20070174167A1 (en) | 2005-05-20 | 2007-07-26 | Stefano Natella | Derivative relationship news event reporting |
US20070005654A1 (en) | 2005-05-20 | 2007-01-04 | Avichai Schachar | Systems and methods for analyzing relationships between entities |
US20060294071A1 (en) | 2005-06-28 | 2006-12-28 | Microsoft Corporation | Facet extraction and user feedback for ranking improvement and personalization |
US7630977B2 (en) | 2005-06-29 | 2009-12-08 | Xerox Corporation | Categorization including dependencies between different category systems |
US7849048B2 (en) | 2005-07-05 | 2010-12-07 | Clarabridge, Inc. | System and method of making unstructured data available to structured data analysis tools |
US8386463B2 (en) | 2005-07-14 | 2013-02-26 | International Business Machines Corporation | Method and apparatus for dynamically associating different query execution strategies with selective portions of a database table |
US8285739B2 (en) | 2005-07-28 | 2012-10-09 | International Business Machines Corporation | System and method for identifying qualifying data records from underlying databases |
US7681075B2 (en) | 2006-05-02 | 2010-03-16 | Open Invention Network Llc | Method and system for providing high availability to distributed computer applications |
US20070100806A1 (en) | 2005-11-01 | 2007-05-03 | Jorey Ramer | Client libraries for mobile content |
US20070250501A1 (en) | 2005-09-27 | 2007-10-25 | Grubb Michael L | Search result delivery engine |
US20070073708A1 (en) | 2005-09-28 | 2007-03-29 | Smith Adam D | Generation of topical subjects from alert search terms |
US7447940B2 (en) | 2005-11-15 | 2008-11-04 | Bea Systems, Inc. | System and method for providing singleton services in a cluster |
US8341622B1 (en) | 2005-12-15 | 2012-12-25 | Crimson Corporation | Systems and methods for efficiently using network bandwidth to deploy dependencies of a software package |
WO2007071548A1 (en) | 2005-12-22 | 2007-06-28 | International Business Machines Corporation | Method and system for automatically generating multilingual electronic content from unstructured data |
US20070174289A1 (en) | 2006-01-17 | 2007-07-26 | Tom Utiger | Management of non-traditional content repositories |
US7899871B1 (en) | 2006-01-23 | 2011-03-01 | Clearwell Systems, Inc. | Methods and systems for e-mail topic classification |
US7519613B2 (en) | 2006-02-28 | 2009-04-14 | International Business Machines Corporation | Method and system for generating threads of documents |
US8726267B2 (en) | 2006-03-24 | 2014-05-13 | Red Hat, Inc. | Sharing software certification and process metadata |
US20070282959A1 (en) | 2006-06-02 | 2007-12-06 | Stern Donald S | Message push with pull of information to a communications computing device |
US8615800B2 (en) | 2006-07-10 | 2013-12-24 | Websense, Inc. | System and method for analyzing web content |
US7624118B2 (en) | 2006-07-26 | 2009-11-24 | Microsoft Corporation | Data processing over very large databases |
US8122026B1 (en) | 2006-10-20 | 2012-02-21 | Google Inc. | Finding and disambiguating references to entities on web pages |
US7783640B2 (en) | 2006-11-03 | 2010-08-24 | Oracle International Corp. | Document summarization |
US7853611B2 (en) | 2007-02-26 | 2010-12-14 | International Business Machines Corporation | System and method for deriving a hierarchical event based database having action triggers based on inferred probabilities |
US7734641B2 (en) | 2007-05-25 | 2010-06-08 | Peerset, Inc. | Recommendation systems and methods using interest correlation |
US8352455B2 (en) | 2007-06-29 | 2013-01-08 | Allvoices, Inc. | Processing a content item with regard to an event and a location |
US7984043B1 (en) | 2007-07-24 | 2011-07-19 | Amazon Technologies, Inc. | System and method for distributed query processing using configuration-independent query plans |
US8332258B1 (en) | 2007-08-03 | 2012-12-11 | At&T Mobility Ii Llc | Business to business dynamic pricing system |
US20090043792A1 (en) | 2007-08-07 | 2009-02-12 | Eric Lawrence Barsness | Partial Compression of a Database Table Based on Historical Information |
US10698886B2 (en) | 2007-08-14 | 2020-06-30 | John Nicholas And Kristin Gross Trust U/A/D | Temporal based online search and advertising |
KR100898339B1 (en) | 2007-10-05 | 2009-05-20 | 한국전자통신연구원 | Autonomous fault processing system in home network environments and operation method thereof |
US8594996B2 (en) | 2007-10-17 | 2013-11-26 | Evri Inc. | NLP-based entity recognition and disambiguation |
US8396838B2 (en) | 2007-10-17 | 2013-03-12 | Commvault Systems, Inc. | Legal compliance, electronic discovery and electronic document handling of online and offline copies of data |
US8375073B1 (en) | 2007-11-12 | 2013-02-12 | Google Inc. | Identification and ranking of news stories of interest |
US8294763B2 (en) | 2007-12-14 | 2012-10-23 | Sri International | Method for building and extracting entity networks from video |
US20090222395A1 (en) | 2007-12-21 | 2009-09-03 | Marc Light | Systems, methods, and software for entity extraction and resolution coupled with event and relationship extraction |
US20090216734A1 (en) | 2008-02-21 | 2009-08-27 | Microsoft Corporation | Search based on document associations |
US8326847B2 (en) | 2008-03-22 | 2012-12-04 | International Business Machines Corporation | Graph search system and method for querying loosely integrated data |
US20100077001A1 (en) | 2008-03-27 | 2010-03-25 | Claude Vogel | Search system and method for serendipitous discoveries with faceted full-text classification |
US8712926B2 (en) | 2008-05-23 | 2014-04-29 | International Business Machines Corporation | Using rule induction to identify emerging trends in unstructured text streams |
US8358308B2 (en) | 2008-06-27 | 2013-01-22 | Microsoft Corporation | Using visual techniques to manipulate data |
CA2686796C (en) | 2008-12-03 | 2017-05-16 | Trend Micro Incorporated | Method and system for real time classification of events in computer integrity system |
US8150813B2 (en) | 2008-12-18 | 2012-04-03 | International Business Machines Corporation | Using relationships in candidate discovery |
US8874576B2 (en) | 2009-02-27 | 2014-10-28 | Microsoft Corporation | Reporting including filling data gaps and handling uncategorized data |
US20100235311A1 (en) | 2009-03-13 | 2010-09-16 | Microsoft Corporation | Question and answer search |
US8972396B1 (en) | 2009-03-16 | 2015-03-03 | Guangsheng Zhang | System and methods for determining relevance between text contents |
US8213725B2 (en) | 2009-03-20 | 2012-07-03 | Eastman Kodak Company | Semantic event detection using cross-domain knowledge |
US8161048B2 (en) | 2009-04-24 | 2012-04-17 | At&T Intellectual Property I, L.P. | Database analysis using clusters |
US8055933B2 (en) | 2009-07-21 | 2011-11-08 | International Business Machines Corporation | Dynamic updating of failover policies for increased application availability |
US9727842B2 (en) | 2009-08-21 | 2017-08-08 | International Business Machines Corporation | Determining entity relevance by relationships to other relevant entities |
US9165034B2 (en) | 2009-10-15 | 2015-10-20 | Hewlett-Packard Development Company, L.P. | Heterogeneous data source management |
WO2011053755A1 (en) | 2009-10-30 | 2011-05-05 | Evri, Inc. | Improving keyword-based search engine results using enhanced query strategies |
US20110125764A1 (en) | 2009-11-26 | 2011-05-26 | International Business Machines Corporation | Method and system for improved query expansion in faceted search |
CN102356393B (en) | 2010-01-29 | 2014-04-09 | 松下电器产业株式会社 | Data processing device |
US9710556B2 (en) | 2010-03-01 | 2017-07-18 | Vcvc Iii Llc | Content recommendation based on collections of entities |
WO2011127655A1 (en) | 2010-04-14 | 2011-10-20 | Hewlett-Packard Development Company, L.P. | Method for keyword extraction |
US8266170B2 (en) | 2010-04-26 | 2012-09-11 | International Business Machines Corporation | Peer to peer (P2P) missing fields and field valuation feedback |
US8595234B2 (en) | 2010-05-17 | 2013-11-26 | Wal-Mart Stores, Inc. | Processing data feeds |
US9189357B2 (en) | 2010-05-25 | 2015-11-17 | Red Hat, Inc. | Generating machine state verification using number of installed package objects |
US8429256B2 (en) | 2010-05-28 | 2013-04-23 | Red Hat, Inc. | Systems and methods for generating cached representations of host package inventories in remote package repositories |
US9443008B2 (en) | 2010-07-14 | 2016-09-13 | Yahoo! Inc. | Clustering of search results |
US8538959B2 (en) | 2010-07-16 | 2013-09-17 | International Business Machines Corporation | Personalized data search utilizing social activities |
US8321443B2 (en) | 2010-09-07 | 2012-11-27 | International Business Machines Corporation | Proxying open database connectivity (ODBC) calls |
US20120102121A1 (en) | 2010-10-25 | 2012-04-26 | Yahoo! Inc. | System and method for providing topic cluster based updates |
US8645298B2 (en) | 2010-10-26 | 2014-02-04 | Microsoft Corporation | Topic models |
WO2012075335A2 (en) | 2010-12-01 | 2012-06-07 | Google Inc. | Recommendations based on topic clusters |
US9245022B2 (en) | 2010-12-30 | 2016-01-26 | Google Inc. | Context-based person search |
US20120246154A1 (en) | 2011-03-23 | 2012-09-27 | International Business Machines Corporation | Aggregating search results based on associating data instances with knowledge base entities |
US20120310934A1 (en) | 2011-06-03 | 2012-12-06 | Thomas Peh | Historic View on Column Tables Using a History Table |
US9104979B2 (en) | 2011-06-16 | 2015-08-11 | Microsoft Technology Licensing, Llc | Entity recognition using probabilities for out-of-collection data |
WO2013003770A2 (en) | 2011-06-30 | 2013-01-03 | Openwave Mobility Inc. | Database compression system and method |
US9032387B1 (en) | 2011-10-04 | 2015-05-12 | Amazon Technologies, Inc. | Software distribution framework |
US9026480B2 (en) | 2011-12-21 | 2015-05-05 | Telenav, Inc. | Navigation system with point of interest classification mechanism and method of operation thereof |
US9037579B2 (en) | 2011-12-27 | 2015-05-19 | Business Objects Software Ltd. | Generating dynamic hierarchical facets from business intelligence artifacts |
US10908792B2 (en) | 2012-04-04 | 2021-02-02 | Recorded Future, Inc. | Interactive event-based information system |
US20130290232A1 (en) | 2012-04-30 | 2013-10-31 | Mikalai Tsytsarau | Identifying news events that cause a shift in sentiment |
US8948789B2 (en) | 2012-05-08 | 2015-02-03 | Qualcomm Incorporated | Inferring a context from crowd-sourced activity data |
US9275135B2 (en) | 2012-05-29 | 2016-03-01 | International Business Machines Corporation | Annotating entities using cross-document signals |
US20130325660A1 (en) | 2012-05-30 | 2013-12-05 | Auto 100 Media, Inc. | Systems and methods for ranking entities based on aggregated web-based content |
US8719219B2 (en) | 2012-09-13 | 2014-05-06 | Sap Ag | Managing feed in in-memory database system |
US9053420B2 (en) | 2012-09-25 | 2015-06-09 | Reunify Llc | Methods and systems for scalable group detection from multiple data streams |
US8996555B2 (en) | 2012-11-26 | 2015-03-31 | Sap Se | Question answering framework for structured query languages |
US9703833B2 (en) | 2012-11-30 | 2017-07-11 | Sap Se | Unification of search and analytics |
US20140229476A1 (en) | 2013-02-14 | 2014-08-14 | SailMinders, Inc. | System for Information Discovery & Organization |
US9542652B2 (en) | 2013-02-28 | 2017-01-10 | Microsoft Technology Licensing, Llc | Posterior probability pursuit for entity disambiguation |
US20140255003A1 (en) | 2013-03-05 | 2014-09-11 | Google Inc. | Surfacing information about items mentioned or presented in a film in association with viewing the film |
US9104710B2 (en) | 2013-03-15 | 2015-08-11 | Src, Inc. | Method for cross-domain feature correlation |
US9298778B2 (en) | 2013-05-14 | 2016-03-29 | Google Inc. | Presenting related content in a stream of content |
US8977600B2 (en) | 2013-05-24 | 2015-03-10 | Software AG USA Inc. | System and method for continuous analytics run against a combination of static and real-time data |
US9087005B2 (en) | 2013-05-31 | 2015-07-21 | International Business Machines Corporation | Increasing resiliency of a distributed computing system through lifeboat monitoring |
US9734221B2 (en) | 2013-09-12 | 2017-08-15 | Sap Se | In memory database warehouse |
US10261950B2 (en) | 2013-11-26 | 2019-04-16 | Sap Se | Table as query language parameter |
US9659108B2 (en) | 2013-12-02 | 2017-05-23 | Qbase, LLC | Pluggable architecture for embedding analytics in clustered in-memory databases |
US9025892B1 (en) | 2013-12-02 | 2015-05-05 | Qbase, LLC | Data record compression with progressive and/or selective decomposition |
CN106164890A (en) | 2013-12-02 | 2016-11-23 | 丘贝斯有限责任公司 | For the method eliminating the ambiguity of the feature in non-structured text |
US9355152B2 (en) | 2013-12-02 | 2016-05-31 | Qbase, LLC | Non-exclusionary search within in-memory databases |
US9201744B2 (en) | 2013-12-02 | 2015-12-01 | Qbase, LLC | Fault tolerant architecture for distributed computing systems |
US9430547B2 (en) | 2013-12-02 | 2016-08-30 | Qbase, LLC | Implementation of clustered in-memory database |
-
2014
- 2014-12-02 US US14/557,900 patent/US9025892B1/en active Active - Reinstated
-
2015
- 2015-05-04 US US14/703,622 patent/US9710517B2/en active Active
-
2016
- 2016-03-10 US US15/066,824 patent/US20160196277A1/en not_active Abandoned
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140022100A1 (en) * | 2000-10-03 | 2014-01-23 | Realtime Data LLC DBA IXO | Methods for encoding and decoding data |
US20020165847A1 (en) * | 2001-05-02 | 2002-11-07 | Mccartney Jason | Logical semantic compression |
US20070250519A1 (en) * | 2006-04-25 | 2007-10-25 | Fineberg Samuel A | Distributed differential store with non-distributed objects and compression-enhancing data-object routing |
US20090089626A1 (en) * | 2007-09-28 | 2009-04-02 | Arm Limited | Techniques for generating a trace stream for a data processing apparatus |
US8345998B2 (en) * | 2010-08-10 | 2013-01-01 | Xerox Corporation | Compression scheme selection based on image data type and user selections |
US8423522B2 (en) * | 2011-01-04 | 2013-04-16 | International Business Machines Corporation | Query-aware compression of join results |
US8782018B2 (en) * | 2011-06-03 | 2014-07-15 | Samsung Electronics Co., Ltd. | Storage device and data processing device utilizing determined dictionary compression |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10565182B2 (en) * | 2015-11-23 | 2020-02-18 | Microsoft Technology Licensing, Llc | Hardware LZMA compressor |
CN108228843A (en) * | 2018-01-09 | 2018-06-29 | 闫健 | A kind of handout compression transmission and restoring method based on internet |
Also Published As
Publication number | Publication date |
---|---|
US9025892B1 (en) | 2015-05-05 |
US9710517B2 (en) | 2017-07-18 |
US20150234899A1 (en) | 2015-08-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9710517B2 (en) | Data record compression with progressive and/or selective decomposition | |
US9424294B2 (en) | Method for facet searching and search suggestions | |
US8838551B2 (en) | Multi-level database compression | |
US10289714B2 (en) | Compression of serialized B-tree data | |
EP3620930B1 (en) | File format for accessing data quickly and efficiently | |
US9535940B2 (en) | Intra-block partitioning for database management | |
US8933829B2 (en) | Data compression using dictionary encoding | |
US11023452B2 (en) | Data dictionary with a reduced need for rebuilding | |
CN102804168A (en) | Data Compression For Reducing Storage Requirements In A Database System | |
WO2015099961A1 (en) | Systems and methods for hosting an in-memory database | |
US9104726B2 (en) | Columnar databases | |
JP6726690B2 (en) | Performing multidimensional search, content-associative retrieval, and keyword-based retrieval and retrieval on losslessly reduced data using basic data sieves | |
EP3173947A1 (en) | Paged inverted index | |
US11880368B2 (en) | Compressing data sets for storage in a database system | |
US9715525B2 (en) | Method and system for searching and storing data | |
US8812523B2 (en) | Predicate result cache | |
US11520763B2 (en) | Automated optimization for in-memory data structures of column store databases | |
WO2018080560A1 (en) | Encoding edges in graph databases | |
US20130173564A1 (en) | System and method for data compression using multiple encoding tables | |
Dong et al. | Content-aware partial compression for textual big data analysis in hadoop | |
EP3014482B1 (en) | Method and system for searching and storing data | |
Dong et al. | Record-aware two-level compression for big textual data analysis acceleration | |
US9501491B1 (en) | File format article of manufacture | |
CN114372077A (en) | Performance index data retrieval method and device, electronic equipment and storage medium | |
JP2016153976A (en) | Program and cluster system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: QBASE, LLC, VIRGINIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIGHTNER, SCOTT;WECKESSER, FRANZ;ZIMMERMAN, BRYAN;REEL/FRAME:045749/0147 Effective date: 20141201 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |