WO2016016539A1

WO2016016539A1 - Method, system and platform for the creation and operation of a multimedia database

Info

Publication number: WO2016016539A1
Application number: PCT/FR2015/051971
Authority: WO
Inventors: Frédéric NOWAK; Carlos MORENO GOMEZ
Original assignee: Shootshareshow
Priority date: 2014-08-01
Filing date: 2015-07-17
Publication date: 2016-02-04
Also published as: FR3024570A1; FR3024570B1

Abstract

The invention relates to a method, system and platform for the creation and operation of databases in order to allow searches for relevant multimedia data and images.

Description

METHOD, SYSTEM AND PLATFORM FOR CONSTITUTION AND

OPERATING A MULTIMEDIA DATABASE

Field of the invention

The present invention relates to the field of image processing and the dynamic constitution of databases for searching multimedia information associated with contexts such as a type of sport. More particularly, the invention relates to the field of the constitution and operation of databases relating to sports events, to enable the search of relevant multimedia images and data for documenting a sporting activity or more generally the context of sports events. 'a sports activity.

State of the art

Various solutions are known in the state of the art for automating the association of metadata with images.

US patent application US2014003648 discloses a method of determining a particular level of interest of a person for a digital image. The digital image, or metadata associated with the digital image, is or is analyzed to designate one or more image elements in the digital image. Levels of familiarity of a particular person vis-à-vis designated picture elements are determined. The level of interest shown by the particular person for the digital image is then determined in response to the determined level of familiarity. In some embodiments, the picture elements contain people and contexts of the scene, digital images containing more familiar people and less familiar scene contexts corresponding to higher levels of interest.

US Pat. No. 8625904 discloses a method of identifying groups of related digital images in a digital image collection. This method comprises the steps of:

- analyzing each of the digital images to produce associated feature descriptors that correspond to the content of the images or the image capture conditions;

store the feature descriptors associated with the digital images in a metadata database;

automatically analyzing the metadata database to identify a plurality of sets of frequent items, each of said sets forming a feature descriptor group jointly appearing in at least a predefined fraction of the digital images;

determine a probability of occurrence for each set of identified frequent items; determining a quality score for each identified set of frequent items in response to the determined probability of occurrence;

classifying sets of frequent articles at least on the basis of the determined quality results; and identifying one or more groups of related digital images corresponding to one or more sets of frequent items that have the highest ranking.

US2011161174 discloses a system for locating, segmenting, annotating, and extracting media files that provides a metadata database for multimedia files, a database manager and a client database for accessing the data contained in the database. The client database, in conjunction with the metadata database and the database manager, provides a variety of different functionalities, namely, deep link functionality, segmentation functionality, metadata annotation feature , an extraction feature, and an access feature. The user, through the client database, annotates the media file or a segment of a media file with metadata that is stored in the database. When the user wishes to locate a media file, the metadata is searched or browsed to locate the database entry associated with the media file in question.

The system further allows the user to load, open and start playback of a multimedia file from an intermediate point between its starting point and an endpoint and without the user having to read the part between the starting point and the intermediate point, fragmenting larger files into shorter segments or individual frames (or still images) or regions in frames (or still images) without creating copies the selected segments of the relevant multimedia files, thereby avoiding rights-of-use issues.

Disadvantages of prior art

The solutions of the prior art do not make it possible to calculate contextual metadata, which are abstract and do not derive from extraction of graphic elements and static processing. For example, the recognition, in an image, of a baseball cap does not make it possible to determine if the context is that of a sporting competition, or a burglary in a jewelery shop.

None of the solutions of the prior art makes it possible to remove such ambiguity, and these solutions are therefore not technically able to provide a contextual basis for carrying out searches as targeted by the invention.

The main purpose of the patent lies in the possibility for multiple users of the service, to access multimedia components (texts, photos, videos, sound files, etc.) coming from various sources, from a platform of creation and dissemination.

This platform brings together a set of low-level technical services to manage media and a reference database. This technical foundation serves as a support for business services and will also enable the commercialization of raw data and media streams.

The inventors have found that no prior similar technique exists to allow access, share and broadcast multimedia sequences created jointly from their multimedia components with an automatic extraction of a subset from criteria that are specific to a context that was not recorded during the creation of the contents and which are defined independently of the stages of creation and diffusion of the contents transmitted to the platform.

The technical problem is to produce multimedia digital files that can be exploited by connected users to select sub-sets of files based on contextual data that did not exist in the media files transmitted to the platform, and that can be defined independently of the nature of its multimedia files transmitted to the platform. Solution provided by the invention

In order to overcome these drawbacks and to allow an automatic processing of images to associate them with descriptors representative of a context not included in the information associated with the images, the present invention relates, according to its most general meaning, to a process for constituting images. a multimedia database associated with contextual metadata, comprising:

A step of analyzing a new image to extract metadata by automatic graphics pattern recognition processing

A step of indexing said image according to said extracted metadata

characterized in that said method further comprises:

A step of enriching said metadata with contextual data calculated according to

o on the one hand raw data attached to said image, including the date and / or geolocation of the shot,

o and on the other hand at least one reference database containing generic data including the date and / or calendar and / or geolocation of an event, as well as metadata associated with a context.

Preferably, the method according to the invention further comprises an additional step of enriching said metadata by contextual data by a processing of extraction of a graphical information belonging to a predetermined category and comparison of said graphical information extracted with a database. reference data containing generic data including reference graphics [logo, player face, color of the coat, ..] as well as metadata associated with a context.

The invention relates to a system for implementing the aforementioned method. The system includes a media and metadata platform including a first programming interface for interoperability with external media file transfer resources, and a second programming interface for interoperability with external search and recovery resources. the set of contents of the platform, said second programming interface providing dialogue with a search engine with access in web mode, the system further comprising a search engine comprising a load balancer ensuring the distribution of the applications through the network between different computers and geographical locations.

In another aspect, the invention relates to a platform for implementing the aforementioned method. The platform according to the invention comprises:

- a first module for collecting and storing media (graphic files, videos, etc.)

a second module for collecting and storing reference data

- a third media broadcasting module.

Advantageously, the first module for collecting and storing media comprises a set of software applications that perform the following treatments on digital video or image files:

- extraction of the metadata recorded in the files during their constitution, for example the date of shooting, the technical data relating to the shooting and the equipment used for the shooting, GPS location provided by a GPS module integrated in the equipment, and if necessary data entered by the operator of the shooting, or data from equipment and intermediate processing (image processing software and software). editing of metadata for example for the press)

comparing the metadata with a database of events and places to associate each image with an event and / or a place

- detection of graphic elements and comparison with reference graphic models associated with descriptors (name of a person, type of sport, ...), for the characterization of these elements and the indexation of the image processed in function of the result of this comparison (for example face recognition for indexing the analyzed image according to the names of recognized persons, recognition of typology of scenes, the number of people present in the image, etc.)

- inference of meta-data detected to calculate other meta-data from an information base, to enrich the meta-data directly extracted from the image (for example name of the sponsor of the event, .. .)

archiving the file and associated metadata in a memory space, according to the access times targeted. Typically, graphics data is stored in a storage space separate from that for storing metadata.

assigning access rights descriptors associated with images and associated metadata.

Advantageously, the second module for collecting and storing reference data comprises a set of software applications for the capture of additional global data (event, place, person, nature of the sport, etc.) to constitute a database of data. reference. Advantageously, the third media broadcasting module comprises a set of software applications carrying out the following treatments:

automatic sharing of files and metadata associated with users connected to the platform

automatic file sharing to third-party platforms (eg FTP server) via a standard file transfer protocol,

display graphic files of the platform on websites through publishing application.

Description of a nonlimiting example of embodiment

The present invention will be better understood on reading the description which follows, refer to a nonlimiting exemplary embodiment illustrated by the appended drawings and diagrams where:

FIG. 1 represents a schematic view of the architecture of the services implemented by the platform according to the invention

FIG. 2 represents a schematic view of the platform

FIG. 3 represents a schematic view of the treatments carried out by the method according to the invention.

Technical Architecture of Services

The various resources are distributed between a media and metadata platform (100) including a first programming interface (101) for interoperability with the external media file transfer resources, and a second programming interface (102) for the interoperability with external resources for searching and retrieving content subsets of the platform. This API (102) allows the dialogue with a search engine (103) with access in web mode.

The search engine (103) includes a load balancer (104) for distributing applications across the network between different computers and geographic locations.

The visit of a resource of the platform (100) causes the issuing of HTTP requests. Each request contains the name of the server for which it is intended, which will then have to process it. The server work is lightened by sending the following queries to other servers in turn. The server name is associated with multiple computers through a regular modification of the DNS service mapping tables.

The components of the platform (100) include: a media search engine (HTML + CSS + JAVASCRIPT)

- a metadata search engine (HTML + CSS + JAVASCRIPT)

- a media gallery (HTML + CSS + JAVASCRIPT)

- a sports database (HTML + CSS + JAVASCRIPT) - an account management area (HTML + CSS + JAVASCRIPT)

- an album management area (HTML + CSS + JAVASCRIPT)

- a search management area (HTML + CSS + JAVASCRIPT)

- a platform administration area (HTML + CSS +

JAVASCRIPT)

- a media sharing form (HTML + CSS + JAVASCRIPT)

- a photo capture application (ANDROID JAVA) - a photo capture application (WINDOWS PHONE, C) - a photo capture application (IPHONE OBJECTIVE, C)

The content provider has computer equipment (105), for example a tablet, a computer or a telephone on which an application (106) adapted to interact with the API (101) is executed.

The user also has a computer equipment (107) on which is executed an application (108) adapted to interact with the API (101) and a browser (109) to interact with the search engine (103) .

Definition of the architecture of the platform The platform (100) shown in schematic view in FIG. 2, for the implementation of an example of the invention comprises several modules:

a first module (1) for collecting and storing media (graphic files, videos, etc.)

a second module (2) for collecting and storing reference data

a third module (3) for broadcasting media. The first media collection and storage module (1) comprises a set of software applications that perform the following treatments on digital video or image files:

- extraction of the metadata recorded in the files during their constitution, for example the date of shooting, the technical data relating to the shooting and the equipment used for the shooting, the GPS location provided by a GPS module integrated in the equipment, and if necessary data entered by the operator of the shooting, or data from equipment and intermediate processing (computer software). image processing and editing of metadata for example for the press)

This module (1) for collecting and storing media comprises a series of tables and databases:

a base (201) for recording the thumbnails associated with the multimedia resources

- a base (202) for recording marked media

a base (203) for recording grayscale images

a base (204) for recording the profile images a base (205) for recording the marked images

a base (206) for recording thumbnails associated with the marked multimedia resources

a base (207) for recording the raw images

a base (208) for recording face images. This multimedia data is transmitted from external applications via an API (120) associated with a media server (122) via a load balancer (121). The second module (2) for collecting and storing reference data comprises a set of software applications for the capture of additional global data (event, place, person, nature of the sport, etc.) to constitute a database reference. These additional data can be entered by an operator, or automatically by extraction from external or internal information sources. It comprises a partitioned documentary database (240), a partitioned image database (241) and a graph database (242), as well as an index database (130).

Access to this data is via a web server (104) associated with a load balancer (103).

The third media delivery module (3) includes a download server (125), an API (101) for interoperability with multimedia data transfer applications, and a set of software applications performing the following processes:

automatic sharing of files and metadata associated with users connected to the platform automatic file sharing to third-party platforms (eg FTP server) via a standard file transfer protocol,

The platform further includes a mail server (124). Definition of the data model

To allow the calculation of metadata by inference algorithms, the data is represented by a semantic graph. Each data is linked to other data by a named relation.

Exploitation of metadata

The processing of metadata capture and additional metadata calculation by inference on the one hand and processing with external data is intended to associate each graphic or video file with data to automate the search and filtering from the data. common criteria that answer the questions: Who did What Where, When and Why:

The "where" is the most trivial to automate. Just match the GPS data of the media (all have now) to sites or areas of sports sites. The automation here allows a level of detail that can not be achieved manually because it would require a very sharp knowledge and a processing time too long to be specified manually. On the 24H of Le Mans and racing in general, the turn of the circuit where the picture was taken is very popular with fans and the press. For example, we are talking about the excess in the curve of Hunaudières, the hanging at the Playstation chicane ... These are the terms that should allow to find the media in a documentary base.

The "when" is also quite simple to detect automatically. The GPS coordinates and the timestamp of the media make it possible to detect the sports event corresponding to the media. The challenge is to have a sufficiently comprehensive database for the events to be detected to be present.

The "what" requires the implementation of computer vision techniques around the classification of patterns that do not require large amounts of reference data. Modern techniques use learning algorithms, such as vector-based learning techniques, which require positive and negative pairs of media. In the case of sports, the challenge will be to identify the sport or type of scene on a sporting event (portrait, action, day, night) or the type of sport when it is impossible to find it otherwise. It is also possible to deduce what from the detection of the sports event associated with a media. In addition, organization detection is just as important as that of competitors. Federations and sponsors seek visibility on the media they use in their communication. The detection of logos is therefore a very important feature for the platform.

The "who" requires the implementation of computer vision techniques requiring very large databases of reference. The recognition of people or subjects on still and moving images is always a challenge of science. A mixed approach based on conventional algorithms and large volumes of data can lead to very good results. Several strategies can be implemented: face recognition from learning a huge base of faces, the detection of jersey numbers and its confrontation with competitors' lists, recognition of helmets of pilots.

The "why" is more complex. Part of the why can be deduced from the detected sporting event. We know that a media captured at the 24H Le Mans will be associated with the World Endurance Championship. We can deduce a part of the context. The nature of the action is more complex to identify. However, there are solutions in computer vision technologies. This treatment makes it possible to detect the nature of the scene (inside, outside, portrait, groups). These characterizations of the scene are also standardized in the IPTC and EXIF standards.

Figure 3 shows the different steps implemented by the invention.

Step (51) concerns the transfer of media from photographers to publishers (people in charge of adding metadata). To automate this step, the platform (100) is able to collect media through various and secure protocols. She is also able to identify the photographer, and even the current contract, to be able to associate the right copyright management policy. The transmission in real time by the photographer, has been tested via cell phones or dedicated platforms (WIFI, Satellite). The identification of the photographer can be done either by login password or by detecting the serial number of the camera. Contract identification can be done via specific metadata (JOBID in IPTC standard). Step 52 is the most critical, it is the phase where almost all metadata is associated with the media. This phase is done manually by dedicated people, often in the press room or at a distance if the conditions of transmission permit. The goal is to define the five criteria that answer the questions: Who did Where, When and Why.

Steps 53 to 58 consist mainly of routing the media to the right targets. The media can no longer carry the metadata, because they then depend on the use that is made of it. The platform must therefore maintain a second category of metadata which are the labels, albums, visibility rights that each user will have put on a media, to be able to share and distribute it. For mainstream use, the platform must also be able to support massive volumes of users.

By centralizing all uses of the sports media value chain, the platform significantly reduces the number of media copy operations. In the practical case described below, each step induces a single or multiple copy and resize operations or reframing according to the uses of the media.

The main challenge is to ensure that each sport can be represented in the database with its specificities, without changing the overall technical architecture and processes of data processing, media and metadata.

This challenge will be met by combining two approaches:

· An automatic: by connecting the platform to various official sports data sources.

• A manual: by creating a network of partners and customers who, while benefiting from the services of the platform, will also be official data providers. The automation of data collection will rely on conventional flow subscription techniques but also referential alignment techniques, inspired by the semantic web and ontologies. The manual approach is to offer our users data entry services and web components to integrate them on their applications. Entry of driver and race car description sheets will have to be made on platform forms.

Claims

claims

1 - Method of constituting a multimedia database associated with contextual metadata, comprising:

A step of indexing said image according to said extracted metadata

characterized in that said method further comprises:

A step of enriching said metadata by contextual data calculated according to

2 - Method of constituting a multimedia database associated with contextual metadata according to claim 1, characterized in that it further comprises an additional step of enriching said contextual data metadata by an extraction processing of a graphical information belonging to a predetermined category and comparing said extracted graphical information with a reference database containing generic data including reference graphics [logo, player's face, color of the jacket, ..] as well as metadata associated with a context. 3 - System for implementing the method according to claim 1 or 2 characterized in that it comprises a media and metadata platform (100) comprising a first programming interface (101) for interoperability with the external resources of transmitting media files, and a second programming interface (102) for interoperability with the platform's external content search and retrieval resources, said second programming interface (102) providing for dialogue with a user. Search engine (103) with web mode access, the system further comprising a search engine (103) including a load balancer (104) for distributing applications across the network between different computers and geographic locations.

4 - Platform for implementing the method according to claim 1 or 2 characterized in that it comprises:

a second module (2) for collecting and storing reference data

a third module (3) for broadcasting media.

5 - Platform for the implementation of the method according to claim 4, characterized in that said first media collection and storage module (1) comprises a set of software applications performing digital video or image files the following treatments:

- extraction of the metadata recorded in the files during their constitution, for example the date of shooting, the technical data relating to the shooting and the equipment used for the shooting, the GPS location provided by a GPS module integrated in the equipment, and if necessary data entered by the operator of shooting, or data from equipment and intermediate processing (image processing software and editing metadata for example for the press )

assigning access rights descriptors associated with images and associated metadata. 6 - Platform for implementing the method according to claim 4 characterized in that said second module (2) for collecting and storing reference data comprises a set of software applications for the capture of additional global data (event, location , nobody, nature of sport, ...) to build a reference database.

7 - Platform for the implementation of the method according to claim 4 characterized in that said third module (3) for broadcasting media comprises a set of software applications performing the following treatments: