WO2006112843A1

WO2006112843A1 - Distributed acoustic fingerprint based recognition

Info

Publication number: WO2006112843A1
Application number: PCT/US2005/013267
Authority: WO
Inventors: Sean Ward; Isaac J. Richards
Original assignee: Sean Ward
Priority date: 2005-04-19
Filing date: 2005-04-19
Publication date: 2006-10-26

Abstract

A system and method for recognizing both commercially available and user created media files, in an accurate and scalable manner using fingerprint analysis of media files. The system is made of four major parts (figure 1): the file analysis component, or recognition client (40); the edge node component (30); the index node component (10); and the central fingerprint analysis and issuer component, or central server (20).

Description

A System and Method for Distributed Acoustic Fingerprint Based Recognition

Claim for Priority/Cross-Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application

Serial No. 60/562,530 (filed April 15, 2004), which is incorporated herein by reference in its entirety.

Technical Field

[0002] The present invention is related to a method for the distributed recognition of media files, such as digital audio and video files. More specifically, it relates to the recognition of such files using a combination of acoustic and bit based fingerprints and metadata.

Background of the Invention

[0003] Generally, media file identification has been based on metadata, a bit based hash, or an acoustic fingerprint, and has utilized centralized server architecture. This can be effective in small-scale systems, or in large scale ones by utilizing a less robust acoustic fingerprint, and a less exhaustive metadata matching technique. One such example system for the former is the Napster system that was deployed in 2001 for large- scale acoustic fingerprint and bit based hash identification. An example of a hybrid metadata and acoustic fingerprint based system is Musicbrainz, which utilizes a single central repository of metadata and fingerprints, moderated by users, to serve client requests. Both systems have scalability issues when the problem of identifying the billions of distinct media files available on large peer-to-peer systems is attempted with them. Additionally, existing scalable systems such as CD TOC based recognition (Gracenote) do not function with media files as they are not on a CD, and which are commonly mislabeled, and do not contain any form of global identifier within them. [0004] Some work in terms of distributed recognition systems has been performed (George Tzanetakis, Jun Gao & Peter Steenkiste - A scalable peer-to-peer system for music content and information retrieval), but it suffers from being unable to intelligently automatically index new content, validate matches from an identification certifier, and does not provide a mechanism for reporting matches, if needed for royalty allocation or similar actuarial activities.

Summary of the Invention

[0005] This system for distributed media file recognition comprises four major parts: the media file analysis component, the edge node component, the index node component, and the central fingerprint analysis and issuer (server) component. Fingerprints are built, using the media file analysis component, off a sound stream, which may be sourced from a compressed audio file, a CD, a radio broadcast, a microphone, or any of the available digital audio sources. Fingerprints are formed by the subdivision of an audio stream into discrete frames, wherein acoustic features, such as zero crossing rates, spectral residuals, Haar wavelet residuals, Mel cepstrals, and trailing spectral power deltas are extracted, summarized, and organized into frame feature vectors. The sampling of these feature frames can be continuous, or fixed windows within the audio stream can be utilized.

[0006] The edge node component will optimally be located with the media analysis component in each end user client, and is responsible for distributed signed fingerprint database storage, as well as fingerprint resolution. A certain percentage of edge nodes can be promoted to index nodes, which are registered with the central server, and maintain the neighborhood set of edge nodes which contain a single image of the fingerprint database, for fingerprint resolution.

[0007] Lastly, the central fingerprint analysis and issuer component is responsible for maintaining the central authoritative fingerprint database, as well as providing a last resort search location in the event of network disruption, data aggregation for automatic database growth, and optionally, query logging for system analysis and content population measurement.

[0008] It is therefore an object of this invention to allow the recognition of both commercially available and user created media files, in an accurate and scalable manner. It is also an object of this invention to allow the monitoring of recognitions, to allow actuarial type activities to be performed based on recognitions. Additionally, it is an object of this invention to allow the automated indexing of new content as the system encounters it. Finally, it is an object of this invention to be able to degrade functionality gracefully if the central server is overloaded or unavailable, and to scale recognition capacity to query volume.

Brief Description of the Drawings

[0009] FIG. 1 is a block diagram, showing the components of the distributed media file recognition system.

[OOIOJ FIG. 2 is a logic flow diagram, showing the process of fingerprinting a media file, extracting metadata, forming a media file recognition packet, and performing a recognition query.

[0011] FlG. 3 is a logic flow diagram, showing the process of adding and populating a new index or edge node to the recognition system.

[0012] FIG. 4 is a logic flow diagram, showing in detail the process of resolving a media file recognition packet once it hits an index node.

[0013] FIG. 5 is a block diagram, showing the components of an index node.

[0014] FIG. 6 is a block diagram, showing the components of an edge node.

[0015] FIG. 7 is a block diagram, showing the components of the central server.

[0016] FIG. 8 is a block diagram, showing the components of a signed fingerprint record.

[0017] FIG. 9 is a logic flow diagram, showing the process of propagating a signed fingerprint record update from the central server.

[0018] FIG. 10 is a logic flow diagram, showing the process of adding new entries to the confirmed identification database in the central server.

Detailed Description

[0019] The preferred embodiment of the present invention places the recognition client 40, edge node 30, and index node 10 components of the system within a peer-to- peer client. The central fingerprint analyzer and issuer component 20 is ideally centrally hosted, although aspects of it (like failover copies of the current database) can be geographically dispersed. Upon being tasked to identify a piece of content, the recognition client will proceed to decode (100), fingerprint with one or more bit based and acoustic based fingerprints (110), and summarize any available file metadata (120) into a media recognition request packet (130), as described in FIG 2. Optionally, this packet can also contain information on the request originator, allowing replies to be sent directly from the edge nodes 30 to the source recognition client 40, and may also optionally be posted to the central server 20 for logging purposes. [0020] To resolve the media recognition request, first an available index node is selected from the recognition client index node list. This list may be obtained from the central server, synchronized from another peer, or hard coded as a set of generally available "master" index nodes. Similar methods for synchronizing the master index node list, such as DNS requests, and broadcast based discovery requests are also contemplated by this invention. Upon contacting an available index node, the media recognition packet is then propagated to the index node for a resolution request (140). [0021] The index node 10 then proceeds to perform the fingerprint hash function

(410) to determine the appropriate db partition set to route the media recognition packet to, as described in FIG 4. A locality preserving hash function, as well as a range search are both suggested hash functions for db partition selection, and more complex methods like centrally generated kd-tree indexes, with the leaf nodes corresponding to db partition slices are also contemplated. Upon selecting a db partition subset list (420), the index node proceeds to route the media recognition packet to one or more currently available edge nodes (430).

[0022] Upon receiving a media recognition request, the edge node 30 proceeds to perform a brute force fingerprint comparison between the incoming media recognition request and any signed fingerprint records 800 (as described in FIG 8) stored on said edge node 30. Additionally, if sufficient signed fingerprint records are stored on an edge node 30 to warrant indexing the local database, such an index can be used to further select a subset of the edge node records to perform the search against. If one or more signed fingerprint records are an apparent match to the incoming fingerprint request , the complete signed fingerprint record set that matches is then returned to the index node 10, or, optionally, routed directly back to the originating recognition client 40. [0023] At the index node 10, any matching signed records are aggregated (440), and returned to the recognition client. If, after validating the signature 840 and expiration 830 in each matching signed record, more than one potential match exists, the metadata records 820 can be utilized to select the best matching record using a fuzzy text matching algorithm. If no match is sufficiently distinct from metadata matching as well, the available matches and overall confidence scores can be displayed to the user at the recognition client 40, to allow a final determination of an appropriate match. [0024] In the event that no candidate signed records are returned from the index node 10, the user selects no match from the record set at the recognition client 40, or all signed records fail to validate against the central issuer certificate, then the media recognition query can be routed directly to the central server 20, for a last attempt at recognition, and to add the media recognition record to the pending table 740 at the central server 20 in the event of no match. This allows batch analysis of pending unmatched queries, to enable the organic insertion of new entries based on repeated matching, user moderation of the pending table, matching against known works databases, and matching against known fingerprint databases into the current active database. One such method of automated insertion is described in FIG 10. [0025] Specifically, for each record in the pending table 740 at the central server

20, a check is performed to see if one or more fingerprints in the record resolve against the reference fingerprint database 750. If a match is found (1020), the master fingerprint database 700 is updated with a new fingerprint to metadata association. If no reference fingerprint is found (1030), then the current unverified fingerprint database 730 is queried. If no matching fingerprint clusters are found (1050), then a new cluster is added to the unverified fingerprint database 730, and the record is left in the pending table 740. [0026] If a matching cluster is found in the unverified fingerprint database 730, then a test can be performed (1040) to check if the associated metadata records for that cluster correlate. Additionally, manual review, or matching against the master confirmed metadata database 710 can be used to confirm a cluster. If the cluster is unconfirmed, the record is left in the pending table, otherwise, the master fingerprint database 700 is updated with a new record from the confirmed cluster (1070), and the record and confirmed cluster are removed from the pending table 740, and unconfirmed fingerprint database 730, respectively.

[0027] Upon inserting, or updating new signed fingerprint records in the central server database, the update mechanism described in FIG 9 is followed. Specifically, the version and expiration fields of the signed fingerprint record (910) are updated to expire any existing record that has been distributed to the edge nodes. Next, the updated record is signed with the central server issuer key 760, certifying the source of the record. The master index node list 720 is then accessed, and the new signed fingerprint record, and the database partition number where the record should be stored, is sent to each available index node (940). Upon receiving a signed fingerprint record, the index node 10 proceeds to contact each edge node 30 stored in the edge node table 520 associated with the database partition number suggested by the central server. Finally, each contacted edge node 30 then updates the local database 610 with the new signed fingerprint record, and validates the signature on the record.

[0028] In the preferred embodiment, when the associated peer-to-peer client is started, a check is made to see whether the host computer has sufficient spare resources (memory, cpu time, bandwidth, and disk space) to serve as an edge or index node. Assuming the host computer passes this check, the process described in FIG 3 is then performed. First an edge node is initialized (300), which proceeds to contact one or more index nodes 10 to check whether a promotion is needed to an index node (320). Several methods for determining the optimal index node to edge node ratio are contemplated, including index node load level, fixed ratios of edge to index nodes, central server control, and geographic and network based location. If the edge node is to become an index node, it registers itself with the central server 20, and one or more peer index nodes are returned. The edge node database 520 and state table 530 of one or more of these peer index nodes is then synchronized directly with the new index node 340. [0029] In the event that the edge node 30 is to stay an edge node, the contacted index node 10 will assign a database partition number to the new edge node 30. This assignment can be done in a variety of fashions, including selecting the database partition with the smallest number of active edge nodes, selecting the partition with the highest query throughput, or selecting based on geographic location (mirroring the most "distant" peer). Upon assigning a database partition to the new edge node (350), a list of currently known peers from that partition group is also provided. The edge node then directly contacts its peer group (380), to synchronize the current database image for that database partition. In the event that no edge node peers are available, the central server is directly contacted to receive the database partition image directly (370), and the edge node registers itself as being ready for queries with one or more index nodes 10 from the index node db 620.

[0030] While this invention has been described in conjunction with specific embodiments thereof, it is evident that many alternative modifications and variations will be apparent to those skilled in the art. Accordingly, the preferred embodiments of the invention as set forth herein are intended to be illustrative, not limiting.

Claims

What Is Claimed Is:

1. A method for resolving acoustic fingerprints of a digital audio signal, comprising:

Routing a resolution query to an index node;

Identifying one or more potential database subsets which could resolve the query;

Contacting one or more edge nodes containing said database subsets; Aggregating any potential match candidates; and Routing the result set back to the originating query source.

2. The method according to Claim 1, wherein the additional step of routing a resolution query to a central resolution server is conducted if no potential match candidates are located.

3. The method according to Claim 1, wherein the additional step of copying a resolution query to a central resolution server is conducted for logging purposes.

4. A method for propagating acoustic fingerprint database updates, comprising:

Incrementing a version and expiration field in a fingerprint record;

Signing the fingerprint record with a central issuer key;

Sending the signed fingerprint record to one or more index nodes identified in a central index node table;

At each index node, sending signed fingerprint record to one or more edge nodes stored in a local edge node table; and

At each edge node, validating signed fingerprint record, and inserting or replacing existing record in local fingerprint record database.

5. A method for updating a master fingerprint database comprising:

Storing unknown fingerprint records in an unresolved table;

Forming fingerprint clusters from entries in said unresolved table;

Validating said fingerprint clusters;

Removing said validated fingerprint cluster and records from said unresolved table; and

Adding validated fingerprint cluster as new record in master fingerprint database.

6. The method according to claim 5, wherein the step of validating said fingerprint cluster is based on manual review of metadata records in said fingerprint cluster.

7. The method according to claim 5, wherein the step of validating said fingerprint cluster is based on correlation between a majority of the metadata records in said fingerprint cluster.

8. The method according to claim 5, wherein the step of validating said fingerprint cluster is based on correlation between the metadata records in said fingerprint cluster and a confirmed metadata table.

9. A system as substantially herein described.