WO2008157352A2

WO2008157352A2 - Interactive messages and advertising

Info

Publication number: WO2008157352A2
Application number: PCT/US2008/066917
Authority: WO
Inventors: Rajarathnam Chandramouli; Koduvayur P. Subbalakshmi
Original assignee: The Trustees Of Stevens Institute Of Technology
Priority date: 2007-06-14
Filing date: 2008-06-13
Publication date: 2008-12-24
Also published as: WO2008157352A3

Abstract

The present invention, in one embodiment, is a method on a server information processing system. The method begins with detecting, using software for automatically identifying object primitives, a plurality of objects (12) from a file with a primary multimedia presentation (14) without making changes to the primary multimedia presentation (14). Next a list of list of objects (16) detected in the primary multimedia presentation (14) is created for linking with a plurality secondary multimedia presentations. Identifiers (18) are layered in the primary multimedia presentation (14), without making changes to the file with the primary multimedia presentation (14), using a layering and scripting software application wherein the identifiers (18) have been matched with at least one of the objects (12) in the list of objects (16). The primary multimedia presentation (14) is presented to at least one viewer along with the identifiers (18) which have been layered. A user selection of at least one identifier (18) matched with at least one of the objects (12) in the list (16) causes a secondary multimedia presentation to be provided the user.

Description

INTERACTIVE MESSAGES AND ADVERTISING Cross -Reference to Related Application

The present application claims the priority of U.S. Patent Application Serial No. 60/943,978 filed June 14, 2007, the entire disclosure of which is incorporated herein by reference. Technical Field of the Invention

The present invention relates generally to message/advertisement, and, more particularly, to using object detection and embedding message/advertisements in digital and analog media. Background Art

The message/advertisement industry is a nearly $75 billion industry. Advertising agencies estimate nearly half of this amount to be wasted since many users do not prefer to view message/advertisements due to a variety of reasons. There are many drawbacks and shortcomings to known advertising delivery. One drawback is that advertisements often interrupt regular programming. There is no mechanism to time-shift the advertisements and other messages after the regular programming. Another drawback is from an individual user's perspective in that a specific message/advertisement may be irrelevant or not interesting. Another drawback is that typical television-based advertisement does not take advantage of alternate delivery methods such as printing, displaying a second audio/video presentation, e-mail, fax, and/or wired and wireless handsets. Moreover, digital devices such as digital video recorders (e.g., TiVO) make skipping advertisements easier. Still another disadvantage with known advertising delivery is that the video itself must be changed or modified to embed advertisements. This is undesirable because many owners and producers of video do not want to have their contents altered. Many of the current Internet advertising (ad) technologies are intrusive in nature. That is, they are pop-up ads, banner ads, etc. Software filters, such as a pop-up blocker, can block these ads easily. Therefore, the return on investment from these types of ad technologies may be low.

Accordingly, what is needed is a method and system to overcome the aforementioned drawbacks encountered in the prior art and to provide an unobtrusive, low cost method that has multiple delivery channels for advertisements and messages. Disclosure of the Invention

The present invention has many advantages over conventional systems. One advantage is that the present invention automatically detects significant objects in video content and assigns some of them as "hotspots" for message advertisement linking using a layering facility, such as Adobe's Flash. Another advantage is that the advertisement is delivered upon the request of the user and may be time- shifted after the primary presentation to be less obtrusive to the user. Alternatively, the advertisement may be delivered over different channels to the user from the primary presentation, such as by printing, displaying a second audio/video presentation, e-mail, fax, and wired and/or wireless handsets. The present invention provides target messaging and advertising. A software tool such as one based on C++, Adobe Flash or Microsoft ASP CDO maybe used to detect objects in a video stream, embed messages and advertisements into the stream related to these objects, and transmit these messages to a client, for example, via e-mail or text messaging. The video program is uninterrupted throughout this process.

The present invention is cost effective because existing hardware is used to enable this technology. Through the use of two-way messaging, such as back-channels, the present invention is compliant with all international standards, including analog and digital media such as T. V., radio, IPTV, cable TV, TiVo, Internet images, and documents. The invention applies non-intrusive technologies, with low-cost overhead, compared to conventional systems. The present invention provides a software tool that differs from the state-of-the-art in that it automatically detects video objects in hotspots in a video stream, whereby the detected hotspots are presented to an advertiser. The advertiser then chooses from a list of the hotspots and pays a fee to embed messages or advertisements contextually related to the chosen hotspots. More particularity, web links related to the hotspot are embedded (i.e., context based advertising) such that when a user clicks the hotspot, the server transmits a message or advertisement corresponding to that object or hotspot. The embedded advertisement is unobtrusive and resilient to ad-blocking filters. The embedded advertisement may be transmitted to a client's secondary device (e.g., mobile device, computer etc.) in response to the client clicking the video frame with the embedded ad, and the main video program is uninterrupted. Other embodiments of the software tool described herein are applicable to IPTV, mobile phones, mobile devices, cable TV, etc.

Detecting hotspots in a video stream requires finding all the major objects in the stream. The present invention may also utilize, for instance, Intel's Integrated Performance Primitives that implements object recognition modules. The identified objects are then put on a candidate list of hotspots. The listed hotspots are filtered for duplications, small sized objects, short-lived objects etc. A final list of hotspots is presented to an advertiser for possible advertisement embedding. These hotspots can be tracked throughout the video using IPP. This enables embedding the corresponding advertisement in several video frames that may not overlap in time.

An advertisement may be hidden in to a hotspot using the transparent layering technology provided by Adobe Flash. Flash technology is supported by more than 95% of the websites, mobile phones, set-top boxes and other devices. The embedded advertisement is hidden and not seen by the user. A visible logo is also embedded in the frames that contain the advertisement. Therefore, when the user views the video frames that contain the advertisements, the logo appears and prompts the user to click on the video to obtain the hidden message or advertisement via e-mail, etc. The logo disappears when the video frames not containing the advertisement appear. ^;

Using Flash, an action script that points to a server is also embedded. When the user clicks on the video frame upon seeing the logo, a webpage in the server is opened. A message box is opened by a

JavaScript indicating to the user that the click was successful. Then a connection to an e-mail server is made. The e-mail server then sends the hidden advertisement to the user's e-mail address. Note that the user has to first register with the server providing details such as the e-mail address, phone number etc. Collecting the user information, connecting the e-mail server, and sending the advertisement to the user via e-mail may be implemented in ASP and JavaScript.

In an alternative embodiment, alternative scripting languages such as VB Script may be used. In an alternative embodiment, other high or low level languages can be used for video object detection.

In an alternative embodiment, audio processing software modules can be used to identify audio based hotspots for message embedding. In an alternative embodiment, when the user moves a computer mouse over the video, a logo may appear on the objects in the video scene that contains advertisements. In an alternative embodiment, when the user moves a computer mouse over the video, a list of advertisements may be shown to the user.

In an alternative embodiment of the present invention, the video may be hosted on the server.

In an alternative embodiment, a short message service (SMS) gateway may be used to transmit the embedded advertisement to the client's mobile phone.

In an alternative embodiment, the client may connect to the server by playing the video on a mobile phone or other mobile devices.

In an alternative embodiment, the client software may be implemented on a set-top box to support cable TV, IPTV and other applications.

As an alternative embodiment message embedding may be done in real to near-real time.

The features and advantages of the present invention will be apparent from the following detailed description of invention and the accompanying drawings. Brief Description of the Drawings The subject matter, which is regarded as the invention, is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features, and advantages of the invention will be apparent from the following detailed description taken in conjunction with the accompanying drawings in which: FIG. 1 is a schematic diagram for a process for detecting video objects of interest or hotspots in a given video program, wherein computer vision and object recognition algorithms are implemented in C++ using the Intel Integrated Performance Primitives (IPP). FIG. 2 is a schematic diagram for a process for presenting to an advertiser the list of detected video hotspots, whereby an advertiser chooses from the list and pays for embedding links in the server that correspond to advertisements or messages related to the chosen hotspots. FIG. 3 is a schematic diagram of a process for embedding messages and advertisements using Adobe Flash's layering technology, wherein the messages or advertisements are embedded into the video hotspots as a transparent layer, and wherein an html link that connects the hotspot to a web page in the server is also embedded.

FIG. 4 is a flow diagram showing steps involved in a client- server interaction.

Best Mode for Carrying Out the Invention

It should be understood that the following embodiments of the present invention are only exemplary. In general, statements made in the specification of the present application do not necessarily limit any of the various claimed inventions. Moreover, some statements may apply to some inventive features but not to others. In general, unless otherwise indicated, singular elements may be in the plural and vice versa with no loss of generality. Referring to FIG. 1 , a server side information processing system 10 methodology is illustrated. The method begins with detecting, using software for automatically identifying object primitives, a plurality of objects 12 from a file with a primary multimedia presentation (i.e., video program) 14 without making changes to the primary multimedia presentation 14. Next a list of objects 16 detected in the primary multimedia presentation 14 is created for linking with a plurality of secondary multimedia presentations (not shown). Identifiers 18 are layered in the primary multimedia presentation 14, without making changes to the file with the primary multimedia presentation 14, using layering and scripting software application (e.g., application/tools described hereinabove), wherein the identifiers 18 have been matched with at least one of the objects 12 in the list 16. The primary multimedia presentation 14 is presented to at least one viewer/user (not shown) of the file with the primary multimedia presentation 14 along with the identifiers 18 which have been layered. The user selection of at least one identifier 18 matched with at least one of the objects 12 in the list 16 causes secondary multimedia presentation(s) to be presented to the user.

In one exemplary embodiment, the secondary multimedia presentation, which has been linked with the object 12 that is matched with the user-selected identifier 18, is presented over a secondary (different) physical end user device 20 (see FIG. 4) than the user's primary device 22 (see FIG. 4) upon which the primary multimedia presentation 14 is presented. In another exemplary embodiment, the secondary multimedia presentation, which has been linked with the object 12 that is matched with the user-selected identifier 18, is presented after the completion of the primary multimedia presentation 14. In another exemplary embodiment, the secondary multimedia presentation is delivered via facsimile, radio, telephone, pager, mail, and/or e-mail.

In another exemplary embodiment, the detecting of the objects 12, using software for automatically identifying object primitives, includes using Intel's Integrated Performance Primitives and wherein the layering identifiers 18 in the primary multimedia presentation 14 include layering identifiers 18 with Adobe Flash technology.

In another exemplary embodiment, the layering identifiers 18 in the primary multimedia presentation 14 include layering logos (not shown).

In another exemplary embodiment the creation of a list of objects 16 detected in the primary multimedia presentation 12, for linking with a plurality secondary multimedia presentations 14, includes linking secondary multimedia presentations selected from third-party advertisers. Referring now to FIGS. 2-3, in one embodiment, the invention maybe implemented using a software tool based on C++, Adobe Flash, and ASP programming for unobtrusive message and advertisement embedding in digital media. The tool detects hotspots 12 in video for advertisement embedding. An advertiser 24 selects and prepares a list of chosen hotspots 26 in which advertisements 28 are hidden in the hotspots 12. The appearance of a logo (not shown) prompts the user click on the video (e.g., the log) to receive the hidden advertisement 28. The server software transmits the hidden advertisement 28 to the user's secondary device 22 via e-mail, text messaging, etc. The main program 14 is uninterrupted.

Referring to FIG. 4, the client-side of the client-service interaction 30 is illustrated for one embodiment of the present invention. A user terminal 32 contacts a web server 34 with a URL link to the primary multimedia presentation (e.g., video stream). The web server 34 responds with a swf file that contains the software to enable interactivity. The swf file allows the user terminal 32 to download or stream the multimedia presentation from a streaming server 36. The swf file acts as a container for the streamed multimedia presentation. When a video frame(s) or a video object(s) containing interactive messages is displayed, the swf file detects it and enables the interactive feature in real-time.

The software maintains synchronization between the displayed video and the related messages. Automatic adjustments are made by the software to compensate for network delays and losses. If the user clicks on an interactive video object, the click is recorded and sent to another server 38 (e.g., e-mail server, SMS gateway, etc.). The server 38 sends the message corresponding to the video object to the user's primary device 22 or secondary device 20. The click is stored in a database server 40 for tracking the user's clicks and other applications. In another exemplary embodiment the messages related to the video object 12 may be overlaid transparently using SMIL.

In another exemplary embodiment, the invention may be implemented as a software tool based on xml, SQL, or scripting to enable advertisers to log into a centralized server and choose from a list of videos and corresponding objects in order to embed a message. It also allows the advertiser to choose the date, time duration, targeted audience or other parameters for delivering the messages. More particularly, spatial and time coordinates of the detected objects are stored in a database along with the video objects, so as to enable the video objects to be hotspots. The database information is displayed to an advertiser upon login to a central server. The advertiser can choose the video objects to embed a message, the date, time, geolocation (e.g., IP address based), etc. When multiple advertisers choose the same object, for instance, all of the messages may be sent, or a message from the highest bidder may be sent, etc. If an object 12 is not chosen, then that object disabled as a hotspot.

In another embodiment, the invention may be implemented as a software tool, based on synchronized multimedia integration language (SMIL), Java Swing, Adobe Flash and other technologies. This tool can be used by individual users (e.g., myspace.com) to embed messages in still pictures and videos as overlays. The user will be able to control the location, duration, objects, transparency, etc. for the overlay.

In one embodiment, the invention is implemented as a software tool based on J2ME or other technologies. This tool can be used within mobile phones to embed messages in videos or still images in the phone's memory. Likewise, embedding and delivering messages to a mobile device (e.g., phone) may utilize multimedia message service (MMS). C++ , Adobe Flash, Microsoft ASP CDO and JavaScript based software tools may be used to detect video objects, embed contextually meaningful messages and advertisements corresponding to the objects, and transmit the embedded messages to a user or client's secondary device upon being clicked. The software tool enables operation at the client as well as the server side.

The server side software provides the following functionality: automatically detects significant objects in the video content; identifies which objects are "hotspots" for message or advertisement embedding; embeds advertisements or messages in the hotspots using

Flash's layering facility; and transmits the embedded advertisement to a user's secondary device using e-mail, text messaging, etc., and the main video program is uninterrupted at the client side, even after clicking on the video. The client side software tool can be executed on computers, mobile devices, set-top boxes, and other devices.

This invention presents a software tool based on, for instance, C++, Adobe Flash, ASP and JavaScript programming for unobtrusive message and advertisement embedding in digital media. This software tool provides the following novel functionalities: automatic detection of objects or hotspots in video; detected hotspots are presented to an advertiser, which chooses from the list of chosen hotspots and pays a fee to embed messages or advertisements contextually related to the chosen hotspots; appearance of a logo prompts the user click on the video to receive the hidden advertisement; unobtrusive transmission of the hidden advertisement to the user's secondary device via e-mail, text messaging etc.; embedded advertisement is resilient to ad-blocking filters; and the main program is uninterrupted.

Operating Modes of the Present Invention The present invention can operate in conjunction with digital video recorders such as TiVO. Tivo is actively exploring product advertising technologies.

The present invention can also operate with T. V. and radio stations, cable operators, Internet service providers , as well as: the movie industry for product placement inside digital movies; the gaming industry to embed message/advertisement inside games; the IPTV product manufacturers such as slingbox; and the smartphone manufacturers and content providers.

The present invention can be realized in hardware, software, or a combination of hardware and software. A system according to a preferred embodiment of the present invention can be realized in a centralized fashion in one computer system or in a distributed fashion where different elements are spread across several interconnected computer systems. Any kind of computer system, or other apparatus adapted for carrying out the methods described herein may be used. A typical combination of hardware and software could be a general purpose computer system with a computer program that, when being loaded and executed, controls the computer system such that it carries out the processes described hereinabove.

In general, the routines executed to implement the embodiments of the present invention, whether implemented as part of an operating system or a specific application, component, program, module, object or sequence of instructions, may be referred to herein as a "program." The computer program typically is comprised of a multitude of instructions that will be translated by the native computer into a machine- readable format and hence executable instructions. Also, programs are comprised of variables and data structures that either reside locally to the program or are found in memory or on storage devices. In addition, various programs described herein may be identified based upon the application for which they are implemented in a specific embodiment of the invention. However, it should be appreciated that any particular program nomenclature hereinabove is used merely for convenience, and thus the invention should not be limited to use solely in any specific application identified and/or implied by such nomenclature.

Although numerous embodiments of the present invention have been disclosed, it will be understood by those having skill in the art that changes can be made to embodiments described hreinabove without departing from the spirit and scope of the invention. The scope of the invention is not to be restricted, therefore, to the specific embodiments described hereinabove, and it is intended that the appended claims cover any and all such applications, modifications, and embodiments within the scope of the present invention.

Claims

1. A method on a server information processing system, comprising the steps of: detecting, using software for automatically identifying object primitives, a plurality of objects from a file with a primary multimedia presentation without making changes to the primary multimedia presentation; creating a list of objects detected in the primary multimedia presentation for linking with a plurality secondary multimedia presentations; layering identifiers in the primary multimedia presentation, without making changes to the file with the primary multimedia presentation, using a layering and scripting software application wherein the identifiers have been matched with at least one of the objects in the list; presenting to at least one viewer the file with the primary multimedia presentation along with the identifiers which have been layered; receiving a selection from the viewer of at least one identifier matched with at least one of the objects in the list; and presenting a secondary multimedia presentation to the user which has been linked with the object that is matched with the identifier which has been selected by the user, wherein the second multimedia presentation is presented over a different physical end user device than the primary multimedia presentation.

2. The method of claim 1 , wherein the secondary multimedia presentation is delivered over facsimile, radio, telephone, pager, mail, and/or e-mail.

3. The method of claim 1 , wherein the detecting, using software for automatically identifying object primitives, includes using Intel Integrated Peformance Primitives and wherein the layering identifiers in the primary multimedia presentation, includes layering identifiers with Adobe flash technology, wherein the coordinates of the identified object primitives are passed as parameters using XML or other technologies for message embedding.

4. The method of claim 1 , wherein the layering identifiers in the primary multimedia presentation includes layering logos.

5. The method of claim 1 , wherein the creating a list of objects detected in the primary multimedia presentation for linking with a plurality secondary multimedia presentations includes linking secondary multimedia presentations selected from third-party advertisers.

6. The method of claim 1 , wherein the spatial and time coordinates of the detected objects are stored in a database along with the video objects so as to enable the video objects to be hotspots

7. A method on a server information processing system, comprising the steps of: detecting, using software for automatically identifying object primitives, a plurality of objects from a file with a primary multimedia presentation without making changes to the primary multimedia presentation; creating a list of objects detected in the primary multimedia presentation for linking with a plurality secondary multimedia presentations; layering identifiers in the primary multimedia presentation, without making changes to the file with the primary multimedia presentation, using a layering and scripting software application wherein the identifiers have been matched with at least one of the objects in the list; presenting to at least one viewer the file with the primary multimedia presentation along with the identifiers which have been layered; receiving a selection from the viewer of at least one identifier matched with at least one of the objects in the list; and presenting a secondary multimedia presentation to the user which has been linked with the object that is matched with the identifier which has been selected by the user, wherein the second multimedia presentation is presented after the primary multimedia presentation is presented.

8. A method on a server information processing system, comprising the steps of: authoring interactive hotspots video using SMIL; and using Java Swing based GUI to support a multitude of video formats to choose videos, frames, and/or objects to embed a message, wherein overlays and invisible message embedding can be achieved by controlling a transparency parameter.

9. A method on a server information processing system, comprising the steps of. authoring interactive hotspots in J2ME; and embedding and delivering messages to a mobile device using a multimedia message service (MMS).

10. A method on a server information processing system, comprising the steps of: authoring interactive hotspots using SMIL; and embedding and delivering messages to a mobile device using multimedia message service (MMS).

11. A method on a server information processing system, comprising the step of. utilizing SMIL set-top box profile for interactive messages in a T.V. program; wherein clicking on video objects in the T.V. program delivers embedded messages to primary or secondary devices.