US8463045B2

US8463045B2 - Hierarchical sparse representation for image retrieval

Info

Publication number: US8463045B2
Application number: US12/943,805
Authority: US
Inventors: Linjun Yang; Qi Tian; Bingbing Ni
Original assignee: Microsoft Corp
Current assignee: Microsoft Technology Licensing LLC
Priority date: 2010-11-10
Filing date: 2010-11-10
Publication date: 2013-06-11
Also published as: US20120114248A1

Abstract

A hierarchical sparse codebook allows efficient search and comparison of images in image retrieval. The hierarchical sparse codebook includes multiple levels and allows a gradual determination/classification of an image feature of an image into one or more groups or nodes by traversing the image feature through one or more paths to the one or more groups or nodes of the codebook. The image feature is compared with a subset of nodes at each level of the codebook, thereby reducing processing time.

Description

BACKGROUND

Content-based image retrieval (CBIR) is gradually gaining momentum among Internet users nowadays. Some websites or search engines offer content-based image search services to Internet users. Specifically, a user submits a query image which is similar to his/her desired image to a website or search engine that provides CBIR services. Based on the query image, the website or search engine subsequently returns one or more stored images to the user. In order to allow efficient retrieval of stored images, the website or search engine represents or encodes the stored images in terms of image features. The website or search engine compares the image features of the stored images with image features of the query image, and retrieves one or more stored images that have image features similar to the image features of the query image.

Given the increasing popularity of CBIR services, academic or business communities have conducted significant research to determine an image representation that can provide efficient comparison and retrieval of images. A number of algorithms and strategies such as Bags of Words (BOW) have been proposed. However, these proposed algorithms or strategies are either restricted to a small set of images or are too computationally intensive to be performed in real time.

SUMMARY

This summary introduces simplified concepts of a hierarchical sparse codebook that may be used for content-based image retrieval, which is further described below in the Detailed Description. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.

This application describes example techniques for generating a hierarchical sparse codebook. In one embodiment, training image features are received. A hierarchical sparse codebook is then generated based at least upon the training image features. The generated hierarchical sparse codebook includes multiple levels, with each level being associated with a sparseness factor.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is set forth with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items.

FIG. 1 illustrates an exemplary environment including an example hierarchical sparse coding system 110.

FIG. 2 illustrates the example hierarchical sparse coding system 110 of FIG. 1 in more detail.

FIG. 3 illustrates a first example hierarchical sparse codebook.

FIG. 4 illustrates an exemplary method of generating a hierarchical sparse codebook.

FIG. 5 illustrates a second example hierarchical sparse codebook.

FIG. 6 illustrates an exemplary method of representing an image using a hierarchical sparse codebook.

DETAILED DESCRIPTION

Overview

As noted above, existing image search algorithms or strategies are limited to a small set of images, and are not scalable to include a large number of images. Furthermore, these algorithms or strategies require significant processing time and power, and therefore cannot be performed in real time.

This disclosure describes a hierarchical sparse coding using a hierarchical sparse codebook. The described codebook includes multiple levels. The described codebook allows a gradual determination/classification of an image feature into one or more groups or nodes by traversing the image feature through one or more paths to the one or more groups or nodes. That is, the described codebook compares an image feature of an image with nodes or nodal features of the nodes, beginning from a root level down to a leaf level of the codebook. Furthermore, the image feature is only compared with a subset of nodes at each level of the codebook, and therefore processing time is significant reduced relative to existing image search strategies. The number of determined/classified groups for the image feature is small/sparse in comparison with the total number of available groups or nodes in the codebook. Using the described codebook allows an efficient determination or classification of an image feature, and therefore provides an efficient and time-saving way of representing an image in terms of image features. Furthermore, image retrieval can be enhanced by comparing extracted features of an image with the codebook to obtain a representation of the image that can be used as an index or a reference for retrieving one or more stored images in a database.

Multiple and varied implementations and embodiments are described below. The following section describes an exemplary environment that is suitable for practicing various implementations. After this discussion, representative implementations of systems, devices, and processes for generating a hierarchical sparse codebook or representing an image using the hierarchical sparse codebook are described.

Exemplary Architecture

FIG. 1 illustrates an exemplary environment 100 usable to implement hierarchical sparse representation for image retrieval. The environment 100 includes one or more users 102-1, 102-2, . . . 102-N (which are collectively referred to as 102), a search engine 104, a website 106, an image database 108, a hierarchical sparse coding system 110, and a network 112. The user 102 communicates with the search engine 104, the website 106 or the hierarchical sparse coding system 110 through the network 112 using one or more devices 114-1, 114-2, . . . 114-M, which are collectively referred to as 114.

The devices 114 may be implemented as a variety of conventional computing devices including, for example, a server, a desktop personal computer, a notebook or portable computer, a workstation, a mainframe computer, a mobile computing device, a handheld device, a mobile phone, an Internet appliance, a network router, etc. or a combination thereof.

The network 112 may be a wireless or a wired network, or a combination thereof. The network 112 may be a collection of individual networks interconnected with each other and functioning as a single large network (e.g., the Internet or an intranet). Examples of such individual networks include, but are not limited to, Local Area Networks (LANs), Wide Area Networks (WANs), and Metropolitan Area Networks (MANs). Further, the individual networks may be wireless or wired networks, or a combination thereof.

In one embodiment, the device 114 includes a processor 116 coupled to a memory 118. The memory 118 includes a browser 120 and other program data 122. The memory 118 may be coupled to or associated with, and/or accessible to other devices, such as network servers, router, and/or other devices 114.

In one embodiment, the user 102 uses the browser 120 of the device 114 to submit an image query to the search engine 104 or the website 106. Upon receiving the image query from the user 102, the search engine 104 or the website 106 compares image query with images stored in the image database 108 and retrieves one or more stored images from the image database 108 using a hierarchical sparse codebook that is generated by the hierarchical sparse coding system 110. The search engine 104 or the website 106 then presents the one or more stored images to the user 102.

In another embodiment, the hierarchical sparse coding system 110 generates a hierarchical sparse codebook using images stored in the image database 108 either upon request from the search engine 104 or the website 106, or on a regular basis.

In still another embodiment, the hierarchical sparse coding system 110 encodes or represents an image received from the user 102, the search engine 104 or the website 106 based on the hierarchical sparse codebook. The hierarchical sparse coding system 110 may return a representation of the received image to the user 102, the search engine 104 or the website 106. Additionally or alternatively, the hierarchical sparse coding system 110 may store the representation of the received image or send the image representation to the image database 108 for storage. This image representation may further be stored as an index or a reference for the received image in the image database 108.

FIG. 2 illustrates various components of the exemplary hierarchical sparse coding system 110 in more detail. In one embodiment, the system 110 can include, but is not limited to, a processor 202, a network interface 204, a system memory 206, and an input/output interface 208.

The memory 206 includes a computer-readable media in the form of volatile memory, such as Random Access Memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash RAM. The memory 206 includes program modules 210 and program data 212. The program data 212 may include a hierarchical sparse codebook 214 and other program data 216. Additionally, the memory 206 may further include a feature database 218 storing training image features that are used for generating the hierarchical sparse codebook 214. In one embodiment, the hierarchical sparse codebook 214 may include a hierarchical tree. For example, FIG. 3 shows an example of a hierarchical sparse codebook 214 in a form of a hierarchical tree. The hierarchical codebook may comprise L number of levels, including a root level 302-1, one or more intermediate levels 302-2, . . . 302-L−1, and a leaf level 302-L. Each node of the root level and the one or more intermediate levels may include K number of child nodes. Each node of the hierarchical codebook is associated with a nodal feature. As used herein, a nodal feature is a trained image feature associated with a node of the hierarchical codebook. The nodal feature may be in a form of a vector, for example. Additionally, each node may further be assigned a subset of the training image features. In one embodiment, each level of the hierarchical sparse codebook is associated with a sparseness factor to determine a degree of sparseness for each level. A degree of sparseness for a level is defined as an average number of nodes or nodal features used to represent each training image feature at that level divided by the total number of nodal features at that same level.

The program module 210 may further include an image receiving module 220. The image receiving module 220 may receive an image from the user 102, the search engine 104 or the website 106. The image may be a query image that the user 102 uses to find his/her desired image(s). Upon receiving the image, the image receiving module 220 may transfer the image to a feature extraction module 222, which extracts features that are representative of the image. The feature extraction module 222 may adopt one or more feature extraction techniques such as singular vector decomposition (SVD), Bag of Visual Words (BoW), etc. Examples of the features include, but are not limited to, scale-invariant feature transform (SIFT) features and intensity histograms.

Depending on which mode the system 110 is performing, the feature extraction module 222 may send the extracted features to a feature determination module 224, the feature database 218, or both.

Representing an Image Using a Hierarchical Sparse Codebook

In one embodiment, in response to receiving the extracted features, the feature determination module 224 determines one or more leaf nodes of the hierarchical sparse codebook 214 to represent each extracted feature. Specifically, the feature determination module 224 compares each extracted feature with nodal features associated with a subset of nodes of the hierarchical sparse codebook 214 level by level.

Table 1 shows a first example algorithm for representing an image using the hierarchical sparse codebook 214. The hierarchical sparse codebook 214 in FIG. 3 is one example. When an extracted feature arrives at the root level 302-1 in FIG. 3 or level 0 in Table 1 (below), the feature determination module 224 compares the extracted feature with each nodal feature associated with each node at next level 302-2, i.e., level 1 in Table 1.

TABLE 1

Algorithm 1: Encode an Image using a hierarchical sparse codebook

[Input]: Feature vector set X = {x₁, x₂, x₃, ... x_M}, e.g., M

feature vectors extracted from the image l; Constructed

hierarchical sparse codebook T

[Output]: Histogram representation h

[initialization]: Active set A^l= {v₁, v₂, v₃, ... v_k} for tree level l = 1

[Main]

For i = 1 to M

For l = 1 to L−1

1. Measure a distance or a degree of overlap between

feature vector x_iand each nodal feature vector in the active set A^l;

2. Generate active set A^l+1 by selecting a node or a

nodal feature at tree level l+1 whose parent has a distance from or a

degree of overlap with the feature vector x_irespectively less than a

predetermined distance threshold or greater than a predetermined

overlap threshold in step 1;

l = l + 1

End For

i = i + l

End For

The histogram representation h is calculated by counting number of times each node or nodal feature at level l = L of the codebook T is selected for all X = {x₁, x₂, x₃, ... x_M}.

The feature determination module 224 may employ a distance measurement module 226 to determine a distance or a degree of overlap between the extracted feature and each nodal feature. The distance measurement module 226 may measure the distance or the degree of overlap according to a predetermined distance metric. For example, if features (i.e., the extracted feature and the nodal feature) are expressed in terms of feature vectors, the predetermined distance metric may include computing a normalized Lp-distance between the extracted feature and the nodal feature, where p can be any integer greater than zero. In one embodiment, the predetermined distance metric may include computing a normalized L2-distance (i.e., Euclidean distance) or a normalized L1-distance (i.e., Manhattan distance) between the extracted feature and the nodal feature. Alternatively, the predetermined distance metric may include computing an inner product of the extracted feature and the nodal feature to determine a degree of overlap therebetween.

In response to determining the distance or the degree of overlap between the extracted feature and each nodal feature at level 302-2, the feature determination module 224 may select a node at level 302-2 whose parent has a distance from the extracted feature that is less than a predetermined distance threshold (e.g., 0.2). Alternatively, the feature determination module 224 may select a node at level 302-2 whose parent has a degree of overlap with the extracted feature that is greater than a predetermined overlap threshold (e.g., zero). The predetermined distance threshold or the predetermined overlap threshold can be adaptively adjusted for each level in order to control a degree of sparseness for each level. A degree of sparseness for a level is defined as an average number of nodes or nodal features used to represent each training image feature at that particular level divided by the total number of nodes or nodal features at that same level. The feature determination module 224 repeats distance measurement for those selected nodes at level 302-2 and node selection for child nodes of the selected nodes at level 302-3. In the above algorithm 1, the feature determination module 224 leaves those unselected nodes at level 302-2 and respective child nodes or branches untouched. More specifically, the feature determination module 224 does not perform any distance determination or node selection for the child nodes of the unselected nodes of level 302-2.

Once the leaf level 302-L of the codebook 214 is reached, one or more leaf nodes are selected according to the above algorithm and are used to represent the extracted feature by the feature determination module 224.

After comparing each extracted feature with the hierarchical sparse codebook 214, the feature determination module 224 may generate a histogram representation of the image. The histogram representation of the image may be generated by counting a number of times each node or nodal feature at a leaf level (i.e., level 302-L in FIG. 3 or level L−1 in Table 1) of the codebook 214 is selected for the extracted features of the images. The histogram representation may be used to represent the image, and may be stored in the image database 108 as an index or a comparison reference for the image.

In some embodiments, the feature determination module 224 may additionally or alternatively employ a cost module 228 to determine which nodes are selected and which nodes are not selected for the extracted feature at each level of the codebook 214. Specifically, the cost module 228 may include a cost function. Table 2 (below) shows a second example algorithm for representing an image using the hierarchical sparse codebook 214.

The hierarchical sparse codebook in FIG. 3 is used for illustration. When an extracted feature x_iarrives at the root level 302-1 in FIG. 3 or level 0 in Table 2, an active set A¹is initially set to include each nodal feature associated with each node at a next level 302-2, i.e., level 1. A cost function |x_i−u_iA^l|_L1+λ|u_i|_L1is then minimized with respect to a response u_i. Each entry, u_i ^j, in the response u_irepresents a response of the extracted feature x_ito corresponding nodal feature v_j. After minimizing the cost function, a new active set A²may be created by selecting a node or a nodal feature in level 302-3 in FIG. 3 or level 2 in Table 2 whose parent at level 302-2 or level 1 gives a response u_i ^jgreater than a predetermined response threshold. The processes of cost function minimization and nodal feature selection are repeated for level 302-3 in FIG. 3 or level 2 in Table 2, until leaf level (i.e., level 320-L in FIG. 3 or level L−1 in Table 2) of the codebook 214 is reached. One or more nodes or nodal features at the leaf level of the codebook 214 having a response with the extracted feature x_igreater than a predetermined response threshold may be selected to represent the extracted feature x_i.

Upon representing each extracted feature of the image using one or more nodes or nodal features at the leaf level of the codebook 214, the feature determination module 224 may generate a histogram representation of the image by summing and normalizing all responses of all X={x₁, x₂, x₃, . . . x_m} at the leaf level of the codebook 214.

TABLE 2

Algorithm 2: Encode an Image using a hierarchical sparse codebook

[Input]: Feature vector set X = {x₁, x₂, x₃, ... x_M}, e.g., M feature

vectors extracted from the image l; Constructed hierarchical sparse

codebook T

[Output]: Histogram representation h

[Main]

For i = 1 to M

For l = 1 to L−1

1. Encode feature vector x_iusing the active set A^lby

minimizing a cost function |x_i− u_iA^l|_L1+ λ|u_i|_L1, where λ is a parameter

to control a degree of sparseness for representing the feature vector x_i

in terms of nodal feature vectors in A^l, and | |_L1represents L1-norm.

2. Generate active set A^l+1 by selecting a node or a nodal

feature at tree level l+1 whose parent gives a response u_i ^jgreater than a

predetermined response threshold in step 1;

l = l + 1

End For

i = i + 1

End For

The histogram representation h is calculated by summing and normalizing all responses of all X = {x₁, x₂, x₃, ... x_M} at level l = L of the codebook tree T.

In some embodiments, a parameter λ, which controls the degree of sparseness, may be different for different levels of the codebook 214. For example, the parameter λ may be smaller for levels closer to the root level to allow more nodes or nodal features to be selected at those levels, and may gradually increase towards the leaf level of the codebook 214 to avoid over-number of selected nodes or nodal features at the leaf level. However, once the parameter λ is determined for each level, the parameter λ will not be modified until the codebook 214 is reconstructed or representations of the images are redone.

Although two example algorithms for representing an image are described above, the present disclosure is not limited thereto. Any algorithm that takes advantage of the described hierarchical sparse codebook 214 and represents each extracted feature of an image in terms of a sparse representation of one or more nodes or nodal features of the codebook 214 are covered in the present disclosure.

In one embodiment, an image may be represented using a combination of the above two algorithms. For example, algorithm 1 may first be used to find an active set up to a predetermined level of the codebook 214 for each image feature of the image. Algorithm 2 may then be used for the rest of the levels of the codebook 214 to obtain one or more nodes or nodal features at the leaf level of the codebook 214 for each image feature. Depending on values of the thresholds employed in algorithm 1, algorithm 1 can allow more nodes or nodal features to be selected for an image feature at each level, and therefore permits a broader exploration of nodal features to represent the image feature. This avoids pre-mature elimination of nodes or nodal features that are actually good candidates for representing the image feature. As the image feature traverses towards the leaf level however, algorithm 2 may be employed to limit number of selected nodes or nodal features at subsequent levels in order to prevent the number of selected nodes or nodal features (i.e., active set in Table 1) from going too large in size.

Upon obtaining a representation (e.g., histogram representation) of the image using one of the above algorithms, the feature determination module 224 may save the representation in the image database 108 and use this representation as an index for retrieving the image. Additionally or alternatively, this representation can be saved as a reference for comparison with representations of other images such as a query image during image retrieval.

In one embodiment, if the image is a query image submitted by the user 102, or forwarded by the search engine 104 or the website 106, the representation of the query image may be used to retrieve one or more stored images in the image database 108. For example, the representation of the query image may be compared with representations of images stored in the image database 108.

In another embodiment, a classifier may be used to classify the query image into one of a plurality of classes (e.g., automobile class) based on the representation of the query image. The classifier may include a neural network, a Bayesian belief network, support vector machines (SVMs), fuzzy logic, Hidden Markov Model (HMM), or any combination thereof, etc. The classifier may be trained on a subset of the representations of the images stored in the image database 108. Upon classifying the query image into a class, stored images within that class may be retrieved and presented to the user 102 according to respective frequencies of retrieval within a certain interval (e.g., the past one day, past one week, past one month, etc).

Additionally or alternatively, the representation of the query image may be compared with the representations of the stored images according to an image similarity metric. The image similarity metric is a measure of similarity between two images, and may return a similarity score to represent a relative resemblance of a stored image with respect to the query image. In one embodiment, a similarity measurement module 230 may be used to calculate a similarity score of a stored image with respect to the query image based upon the representation of the query image. For example, the similarity measurement module 230 calculates the similarity score based on a ratio of the number of common features in the representations of the query image and the stored image with respect to their average number of features.

In another embodiment, the similarity measurement module 230 may compute a correlation between the representation of the query image with representation of a stored image. For example, if an image is represented in the form of a histogram as described above, a correlation between a histogram representation of the query image and a histogram representation of a stored image may be computed to obtain a similarity score therebetween. In one embodiment, each of these histogram representations may first be normalized such that a respective area integral of the histogram representations are normalized to one, for example.

Based on the similarity scores of these stored images with respect to the query image, one or more stored images may be presented to the user 102, and arranged according to their similarity scores, for example, in a descending order of their similarity scores.

The program module 210 may further include a codebook generation module 232. The codebook generation module 232 generates the hierarchical sparse codebook 214 based on the training image features that are stored in the feature database 218. Additionally or alternatively, the codebook generation module 232 generates the hierarchical sparse codebook 214 based on images stored in the image database 108. In one embodiment, the codebook generation module 232 generates or reconstructs the hierarchical sparse codebook 214 on a regular basis, e.g., each day, each week, each month, or each year. Alternatively, the hierarchical sparse codebook 214 may be generated upon request, for example, from the search engine 104 or the website 106.

In still another embodiment, the hierarchical sparse codebook 214 is reconstructed based on performance of the codebook 214 in retrieving stored images in response to query images submitted from the user 102. For example, the program data 212 may further include image query data 234. The image query data 234 may include query images that have been submitted by one or more users 102 and stored images that were returned in response to the query images. Additionally or alternatively, the image query data 234 may include one or more stored images that have been selected by the users 102 in response to the query images. In one embodiment, the image query data 234 may further include similarity scores of the one or more selected images with respect to the query images. In an event that the image query data 234 includes the similarity scores of the selected images, the codebook 214 may be reconstructed in response to an average similarity score of the selected images in the image query data 234 being less than a predetermined similarity threshold. The predetermined similarity threshold may be set by an administrator or operator of the system 110 according to the accuracy and/or computing requirements, for example. For example, if a perfect match between a query image and a stored image has a similarity score of one, the codebook 214 may be reconstructed in response to the average similarity score being less than 0.7, for example.

Generating a Hierarchical Sparse Codebook

When a hierarchical sparse codebook 214 is generated or reconstructed, the codebook generation module 232 may receive a plurality of training image features from the feature database 218. Additionally or alternatively, the codebook generation module 232 may receive a plurality of images from the image database 108 and use the feature extraction module 222 to extract a plurality of image features for training purposes. Upon receiving the plurality of training image features, the codebook generation module 232 generates a hierarchical sparse codebook 214 according to a codebook generation algorithm. An example algorithm is illustrated in Table 3 (below).

For example, k number of nodes at level 1 are branched out from a root node at level 0. Each node at level 1 is associated with a nodal feature which is a training image feature randomly selected from the plurality of training image features. The plurality of training image features are then compared with each nodal feature at level 1 in order to assign a subset of training image features to the corresponding node at level 1. The subset of training image features assigned to a node includes a training image feature that has a response (e.g., a degree of overlap) to a nodal feature associated with that node greater than a predetermined response threshold, e.g., zero. Upon assigning a subset of training image features to a node at level 1, a set of k nodal features are trained with respect to the assigned subset of training image features for the node. Specifically, based on the assigned subset of training image features, a cost function is minimized with respect to the set of k nodal features:
Σ_i |x _li ^j −u _li ^j V _l ^j|_L1+λ_lΣ_i |u _li ^j|_L1 (1)
where

- x_li ^jrepresents a training image feature in a subset X_l ^j
- V_l ^jrepresents the set of k nodal features that are trained for node j at level l, i.e., o_l ^j,
- u_li ^jrepresents a response of x_li ^jto V_l ^j, and
- λ_lrepresents a parameter to control a degree of sparseness for level l.

Upon obtaining the set of k nodal features that minimizes the above cost function for the node, this set of k nodal features are assigned to child nodes of the node at next level, i.e., level 2. These processes of cost function minimization and nodal feature assignment are repeated for each node at each level until each node at the leaf level of the codebook is assigned a nodal feature and a subset of training image features or leaf level of the codebook is reached. At this point, the hierarchical sparse codebook is generated.

TABLE 3

Algorithm 3: Generate a hierarchical sparse codebook

[Input]: Feature vector set X = {x₁, x₂, x₃, ... x_N}, e.g., N feature

vectors from a set of training images

[Output]: K-branch tree T, level l = 0, 1, 2, ... L, each node being

associated with a nodal feature vector v

[initialization]: Branch a root node (at level l = 0) into K nodes

(at level l = 1), each of the K nodes at level l = 1 is randomly

selected from the feature vector set X [Main]

For l = 1 to L−1

1. For each node j at level l, i.e., o_l ^j, collect a subset of X

which has a response with a nodal feature vector associated with

node o_l ^jgreater than a predetermined response threshold, and is

denoted as X_l ^j;

2. For each node j at level l, o_l ^j, based on X_l ^j, train a set of

K nodal features V_l ^j, by minimizing a cost function Σ_i|x_li ^j− u_li ^jV_l ^j|_L1+

λ_lΣi|u_li ^j|_L1with respect to a visual codebook associated with node o_l ^j,

i.e., V_l ^j, then child nodes of node o_l ^jat level l+1, are associated with

nodal features of V_l ^j;

l = l + 1

End For

The parameter λ_l(which is also called a sparseness factor for level l) can be adaptively adjusted to change a degree of sparseness for the level l. In one embodiment, the parameter λ_lor the degree of sparseness for a level is adjusted to be less than a predetermined threshold level. In another embodiment, the parameter λ_lor the degree of sparseness for a level is adjusted to be within a predetermined range. In still another embodiment, the parameter λ_lor the degree of sparseness for each level is collectively adjusted to obtain an overall degree of sparseness for the codebook and the plurality of training image features that is less than a predetermined overall threshold or within a predetermined overall range. The predetermined threshold level or the predetermined range may be the same or different for different levels.

In one embodiment, the above algorithm may further be modified. Specifically, after randomly assigning k number of training image features to be nodal features associated with the nodes at level 1, the algorithm may further train these nodal features to minimize the above cost function for level 1. Upon obtaining a set of optimized nodal features that minimize the cost function of level 1, the algorithm may assign these optimized nodal features to the nodes of level 1. The algorithm further assigns a subset of training image features that have responses greater than a predetermined response threshold to each node of level 1.

Additionally or alternatively, the algorithm may further specify that a training image feature that is assigned to a node is also a training image feature that has been assigned to the parent of the node.

Alternative Embodiments

Although the hierarchical sparse codebook 214 is described to include a hierarchical tree in the foregoing embodiments, the codebook 214 is not limited thereto. The hierarchical sparse codebook 214 can include any hierarchical structure. In one embodiment, the hierarchical sparse codebook 214 may initially include a hierarchical tree. After or during the training phase of the hierarchical sparse codebook 214, however, a node (i.e., a node at an intermediate level and/or a leaf level of the codebook 214) may be purged based on an average degree of overlap between associated training image features and corresponding nodal feature of the node. For example, a node may be purged if corresponding average degree of overlap between associated training image features and corresponding nodal feature is less than a predetermined threshold. Furthermore, this predetermined threshold may vary among different levels. In one embodiment, the predetermined threshold for average degree of overlap is lower at a higher level (i.e., a level closer to the root level of the codebook 214), and increases towards the leaf level of the codebook 214. This is because the number of training image features assigned to a node at the higher level is usually greater and a nodal feature associated with the node is more generalized with respect to the assigned training image features. Having a lower threshold therefore avoids pre-mature purging of the node at the higher level. On the other hand, a node at a lower level is usually assigned with a fewer number of training image features, and a corresponding nodal feature may be more specific to the assigned training image features. Therefore, the predetermined threshold associated with the node at the lower level can be higher to reflect a change from generality to specificity of nodal features from a high level to a low level of the codebook 214.

In another embodiment, the hierarchical sparse codebook may be a hierarchical structure having a plurality of levels, with each level having a predetermined number of nodes. Rather than having an equal number of intermediate child nodes for each node at one level, the number of intermediate child nodes of a node at that level may be determined upon the number of training image features assigned to that particular node. For example, the number of intermediate child nodes of a first node at one level is greater than the number of intermediate child nodes of a second node at the same level if the number of training image features assigned to the first node is greater than the number of training image features assigned to the second node. In essence, a node having a greater number of training image features is allocated more resources (i.e., child nodes) to represent these training image features while a node having a fewer number of training image features is allocated fewer resources, thereby optimizing the use of resources which are usually limited.

Exemplary Methods

Exemplary methods for generating a hierarchical sparse codebook or representing an image using the hierarchical sparse codebook are described with reference to FIGS. 4-6. These exemplary methods can be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, and the like that perform particular functions or implement particular abstract data types. The methods can also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communication network. In a distributed computing environment, computer executable instructions may be located both in local and remote computer storage media, including memory storage devices.

The exemplary methods are illustrated as a collection of blocks in a logical flow graph representing a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof. The order in which the methods are described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the methods, or alternate methods. Additionally, individual blocks may be deleted from the methods without departing from the spirit and scope of the subject matter described herein. In the context of software, the blocks represent computer instructions that, when executed by one or more processors, perform the recited operations.

FIG. 4 illustrates an exemplary method 400 of generating a hierarchical sparse codebook.

At block 402, a plurality of training image features are received. This plurality of training image features may be obtained from one or more databases and/or one or more search engines. The plurality of training image features may be extracted from a plurality of images that are stored in the one or more databases and/or the one or more search engines.

At block 404, a hierarchical sparse codebook is generated based at least upon the plurality of training image features. The hierarchical sparse codebook may be generated to include a plurality of levels. In one embodiment, each of the plurality of levels may be associated with a sparseness factor as shown in FIG. 3, for example. Each level of the hierarchical sparse codebook is generated by adjusting corresponding sparseness factors to be less than respective predetermined thresholds or within respective predetermined ranges. Additionally or alternatively, the hierarchical sparse codebook may be generated by adjusting the sparseness factor of each level to obtain an overall degree of sparseness for the codebook and the plurality of training image features. In one embodiment, the sparseness factor of each level is adjusted to obtain an overall degree of sparseness that is less than a predetermined overall threshold or within a predetermined overall range. This predetermined overall threshold or predetermined overall range may be set by an administrator or an operator of the system 112 based on specified computing requirements or needs.

Additionally or alternatively, generating the hierarchical sparse codebook at block 404 may include representing each training image feature by a sparse number of leaf nodes or nodal features that are associated with the leaf nodes of the hierarchical sparse codebook. FIG. 5 shows an example of this hierarchical sparse codebook. Upon generating the hierarchical sparse codebook, each training image feature j is represented by a sparse number of nodes or nodal features at the leaf level of the codebook.

FIG. 6 illustrates an exemplary method 600 of representing or encoding an image using a hierarchical sparse codebook.

At block 602, an image is received. This image may be received from a user for image query. Alternatively, this image may be received from a search engine or a website for encoding the image.

At block 604, a plurality of image features are extracted from the image.

At block 606, each image feature of the image is compared with a hierarchical sparse codebook to obtain one or more leaf-level features (i.e., nodal features at leaf level) of the codebook. The one or more leaf-level features represent a sparse code representation of the respective image feature.

At block 608, a histogram for the image is generated based upon the one or more leaf-level features of each image feature of the image. In one embodiment, the histogram represents respective number of times that each leaf-level feature of the codebook is encountered by the plurality of image features of the image.

At block 610, the image is represented by the histogram. The histogram may further be stored in a database as an index for the image. Additionally or alternatively, the histogram may be acted a reference for comparison between another image such as a query image during image retrieval. For example, the histogram of the query image may be compared with histograms of a subset of stored images in the database. In one embodiment, the comparison may be performed by computing correlations between the histogram of the query image and the histograms of the subset of stored images. One or more stored images having a correlation greater than a predetermined correlation threshold may be retrieved and presented to the user.

Any of the acts of any of the methods described herein may be implemented at least partially by a processor or other electronic device based on instructions stored on one or more computer-readable media. Computer-readable media can be any available media that can be accessed during generation of the hierarchical sparse codebook or encoding an image using the hierarchical sparse codebook. By way of example, and not limitation, computer-readable media may comprise volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer-readable media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information. Combinations of any of the above should also be included within the scope of computer-readable media.

CONCLUSION

Although the invention has been described in language specific to structural features and/or methodological acts, it is to be understood that the invention is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the invention.

Claims

What is claimed is:

1. One or more memory storage devices storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

receiving a plurality of training image features; and

generating a hierarchical sparse codebook based at least upon the plurality of training image features, the generating comprising creating a plurality of levels for the hierarchical sparse codebook, each level being associated with a respective sparseness factor.

2. The one or more memory storage devices as recited in claim 1, wherein the generating further comprises adjusting the respective sparseness factor of each level to obtain a degree of sparseness which is less than a predetermined threshold or within a predetermined range.

3. The one or more memory storage devices as recited in claim 1, wherein the generating further comprises adjusting the respective sparseness factor of each level to obtain an overall degree of sparseness for the hierarchical sparse codebook, which is less than a predetermined threshold or within a predetermined range, the overall degree of sparseness.

4. The one or more memory storage devices as recited in claim 1, wherein the generating further comprises assigning an image feature set to a node of a level of the plurality of levels, the image feature set comprising a subset of the plurality of training image features.

5. The one or more memory storage devices as recited in claim 4, wherein the assigning comprises assigning a training image feature of the plurality of training image features to the node if a degree of overlap between the training image feature and a nodal feature of the node is greater than or equal to a predetermined threshold.

6. The one or more memory storage devices as recited in claim 1, wherein the plurality of levels comprises at least a root level, a first level and a second level.

7. The one or more memory storage devices as recited in claim 6, wherein the generating further comprises:

creating a first number of nodes at the first level, the first number of nodes comprising a first number of first-level nodal features;

adjusting the first number of first-level nodal features to minimize a cost of a first cost function based at least upon the plurality of training image features, the first cost function comprising a first sparseness function to control a degree of sparseness associated with representations of the plurality of training image features at the first level; and

for each of the first number of nodes at the first level, assigning a first-level nodal feature set based at least upon the plurality of training image features and an assignment scheme.

8. The one or more memory storage devices as recited in claim 7, wherein the generating further comprises:

for each of the first number of nodes at the first level,

generating a second number of nodes at the second level, the second number of nodes comprising a second number of second-level nodal features,

adjusting the second number of second-level nodal features to minimize a cost of a second cost function based at least upon the respective first-level nodal feature set, the second cost function comprising a second sparseness function to control a degree of sparseness associated with representations of the plurality of training image features at the second level; and

for each of the second number of nodes at the second level, assigning a second-level nodal feature set based at least upon the respective first-level nodal feature set and the assignment scheme.

9. The one or more memory storage devices as recited in claim 8, wherein the assignment scheme comprises a predetermined threshold, and wherein assigning the first-level nodal feature or assigning the second-level nodal feature comprises assigning a training image feature to a node if a degree of overlap between the training image feature and a nodal feature associated with the node is greater than or equal to the predetermined threshold.

10. The one or more memory storage devices as recited in claim 1, further comprising:

receiving an image query comprising an example image from a user;

extracting one or more image features from the example image;

comparing each of the one or more image features with the hierarchical sparse codebook to obtain one or more leaf-level features of the codebook;

generating a histogram for the example image based on the one or more leaf-level features of each of the one or more image features from the example image;

retrieving one or more database images based at least upon the histogram; and

presenting the one or more database images to the user.

11. The one or more memory storage devices as recited in claim 10, wherein the retrieving comprises:

computing correlations between the histogram of the example image with histograms of a subset of database images stored in a database; and

retrieving the one or more database images that have corresponding correlations greater than a predetermined correlation threshold.

12. A computer-implemented method for generating a hierarchical sparse codebook, the method comprising:

receiving a plurality of training image features; and

generating a hierarchical sparse codebook based at least upon the plurality of training image features, the generating comprising encoding each training image feature using a sparse number of nodal features that are associated with leaf nodes of the hierarchical sparse codebook.

13. The computer-implemented method as recited in claim 12, wherein the generating further comprises:

generating a plurality of levels for the hierarchical sparse codebook, each level comprising a predetermined number of nodes;

associating each node of each level with a nodal feature;

adjusting each nodal feature of each node to minimize a cost of a cost function of the respective level based at least upon an image feature set of a parent node of the respective node; and

assigning, to each node, a subset of the image feature set of the respective parent node.

14. The computer-implemented method as recited in claim 13, wherein the cost function of each level comprises a sparseness function defining a degree of sparseness of representation of each training image feature at the respective node level.

15. The computer-implemented method as recited in claim 13, wherein the assigning comprises:

for each node, obtaining a degree of overlap between the respective nodal feature and an image feature of the image feature set of the respective parent node;

assigning, to the respective node, the image feature of the image feature set of respective parent node if the degree of overlap is greater than a predetermined threshold.

16. The computer-implemented method as recited in claim 12, further comprising:

receiving an image query comprising an example image from a user;

extracting one or more image features from the example image;

retrieving one or more database images based at least upon the histogram; and

presenting the one or more database images to the user.

17. The computer-implemented method as recited in claim 16, wherein the retrieving comprises:

18. A computer-implemented method comprising:

receiving an image;

extracting a plurality of image features from the image;

comparing each image feature with a hierarchical sparse codebook to obtain one or more leaf-level features of the codebook, the one or more leaf-level features representing a sparse code representation of the respective image feature;

generating a histogram for the image based at least upon the one or more leaf-level features of each image feature of the image; and

representing the image by the histogram.

19. The computer-implemented method as recited in claim 18, further comprising:

generating an index for the image based at least upon the histogram; and

storing the index and the image in a database.

20. The computer-implemented method as recited in claim 18, wherein the image comprises an example image received from a user for image query, and wherein the method further comprises:

retrieving one or more database images based at least upon the histogram; and

presenting the one or more database images to the user.