US20120134409A1 - EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES - Google Patents
EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES Download PDFInfo
- Publication number
- US20120134409A1 US20120134409A1 US13/363,187 US201213363187A US2012134409A1 US 20120134409 A1 US20120134409 A1 US 20120134409A1 US 201213363187 A US201213363187 A US 201213363187A US 2012134409 A1 US2012134409 A1 US 2012134409A1
- Authority
- US
- United States
- Prior art keywords
- experience
- codec
- encoding
- sentio
- recited
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/156—Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/164—Feedback from the receiver or from the transmission channel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234327—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into layers, e.g. base layer and one or more enhancement layers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/631—Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/637—Control signals issued by the client directed to the server or network components
- H04N21/6377—Control signals issued by the client directed to the server or network components directed to server
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/637—Control signals issued by the client directed to the server or network components
- H04N21/6377—Control signals issued by the client directed to the server or network components directed to server
- H04N21/6379—Control signals issued by the client directed to the server or network components directed to server directed to encoder, e.g. for requesting a lower encoding rate
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
Definitions
- the present teaching relates to experience or “sentio” codecs enabling encoding and transmission for data streams involving a variety of dimensions and data types including video, group participation, gesture recognition, heterogeneous device use, emotions, etc.
- the present invention contemplates a variety of experience or “sentio” codecs, and methods and systems for enabling an experience platform, as well as a Quality of Experience (QoE) engine which allows the sentio codec to select a suitable encoding engine or device.
- the sentio codec is capable of encoding and transmitting data streams that correspond to participant experiences with a variety of different dimensions and features.
- the following description provides one paradigm for understanding the multi-dimensional experience available to the participants, and as implemented utilizing a sentio codec. There are many suitable ways of describing, characterizing and implementing the sentio codec and experience platform contemplated herein.
- FIG. 1 illustrates a system architecture for composing and directing user experiences
- FIG. 2 is a block diagram of an experience agent
- FIG. 3 is a block diagram of a sentio codec
- FIG. 4 provides a screen shot useful for illustrating how a hybrid encoding scheme can be used to accomplish low-latency transmission.
- the present invention contemplates a variety of experience or “sentio” codecs, and methods and systems for enabling an experience platform, as well as a Quality of Experience (QoS) engine which allows the sentio codec to select a suitable encoding engine or device.
- the sentio codec is capable of encoding and transmitting data streams that correspond to participant experiences with a variety of different dimensions and features.
- the term “sentio” is Latin roughly corresponding to perception or to perceive with one's senses, hence the original nomenclature “sensio codec.”
- video codec The primary goal of a video codec is to achieve maximum compression rate for digital video while maintaining great picture quality video; audio codecs are similar. But video and audio codecs alone are insufficient to generate and capture a full experience, such as a real-time experience enabled by hybrid encoding, and encoding of other experience aspects such as gestures, emotions, etc.
- FIG. 4 will now be described to provide an example experience showing 4 layers where video encoding alone is inadequate.
- the “layer” concept will be described below in more detail with reference to FIGS. 1-3 .
- a first layer is generated by Autodesk 3ds Max instantiated on a suitable layer source, such as on an experience server or a content server.
- a second layer is an interactive frame around the 3ds Max layer, and in this example is generated on a client device by an experience agent.
- a third layer is the black box in the bottom-left corner with the text “FPS” and “bandwidth”, and is generated on the client device but pulls data by accessing a service engine available on the service platform.
- a fourth layer is a red-green-yellow grid which demonstrates an aspect of a low-latency transfer protocol (e.g., different regions being selectively encoded) and is generated and computed on the service platform, and then merged with the 3ds Max layer on the experience server.
- a low-latency transfer protocol e.g., different regions being selectively encoded
- FIG. 4 illustrates how a hybrid encoding approach can be used to accomplish low-latency transmission.
- the first layer provides an Autodesk 3ds Max image including a rotating teapot, the first layer moving images, static or nearly static images, and graphic and/or text portions.
- a hybrid approach encoding some regions with a video encoder, other regions with a picture encoder, and other portions as command, results in better transmission results, and can be optimized based on factors such as the state of the network and the capabilities of end devices.
- These different encoding regions are illustrated by the different coloring of the red-green-yellow grid of layer 4 .
- a video codec alone is inadequate to accomplish the hybrid encoding scheme covering video, pictures and commands. While it is theoretically possible to encode the entire first layer using only a video codec, latency and other issues can prohibit real-time and/or quality experiences. A low-latency protocol can solve this problem by efficiently encoding the data.
- a multiplicity of video codecs can be used to improve encoding and transmission.
- h.264 can be used if a hardware decoder is available, thus saving battery life and improving performance, or a better video codec (e.g., low latency) can be used if the device fails to support h.264.
- the present teaching contemplates an experience or sentio codec capable of encoding and transmitting data streams that correspond to experiences with a variety of different dimensions and features.
- These dimensions include known audio and video, but further may include any conceivable element of a participant experience, such as gestures, gestures+voice commands, “game mechanics” (which you can use to boost QoE when current conditions (such as network) do not allow you to do so—i.e. apply sound distortion effect specific to a given experience when loss of data happened), emotions (perhaps as detected via voice or facial expressions, various sensor data, microphone input, etc.
- virtual experiences can be encoded via the sentio codec.
- virtual goods are evolved into virtual experiences. Virtual experiences expand upon limitations imposed by virtual goods by adding additional dimensions to the virtual goods.
- User A transmits flowers as a virtual good to User B.
- the transmission of the virtual flowers is enhanced by adding emotion by way of sound, for example.
- the virtual flowers are also changed to a virtual experience when User B can do something with the flowers, for example User B can affect the flowers through any sort of motion or gesture.
- User A can also transmit the virtual goods to User B by making a “throwing” gesture using a mobile device, so as to “toss” the virtual goods to User B.
- the sentio codec improves the QoE to a consumer or experience participant on the device of their choice. This is accomplished through a variety of mechanisms, selected and implemented, possibly dynamically, based on the specific application and available resources.
- the sentio codec encodes multi-dimensional data streams in real-time, adapting to network capability.
- a QoE engine operating within the sentio codec a makes decisions on how to use different available codecs.
- the network stack can be implemented as hybrid, as described above, and in further detail with reference to Vonog et al.'s U.S. patent application Ser. No. 12/569,876.
- the sentio codec can include 1) a variety of codecs for each segment of experience described above, 2) a hybrid network stack with network intelligence, 3) data about available devices, and 4) a QoE engine that makes decisions on how to encode. It will be appreciated that QoE is achieved through various strategies that work differently for each given experience (say a zombie karaoke game vs. live stadium rock concert experience), and adapt in real-time to the network and other available resources, know the devices involved and take advantages of various psychological tricks to conceal imperfections which inevitably arise, particularly when the provided experience is scaled for many participants and devices.
- FIG. 1 illustrates a block diagram of a system 10 .
- the system 10 can be viewed as an “experience platform” or system architecture for composing and directing a participant experience.
- the experience platform described herein provides, by way of example only, one platform suitable for incorporating and taking advantage of the sentio codec described herein.
- the experience platform 10 is provided by a service provider to enable an experience provider to compose and direct a participant experience.
- the participant experience can involve one or more experience participants.
- the experience provider can create an experience with a variety of dimensions, as will be explained further now.
- the sentio codec enables the encoding and transmission of data streams representing this variety of dimensions.
- the following description provides one paradigm for understanding the multi-dimensional experience available to the participants. There are many suitable ways of describing, characterizing and implementing the experience platform contemplated herein.
- services are defined at an API layer of the experience platform.
- the services are categorized into “dimensions.”
- the dimension(s) can be recombined into “layers.”
- the layers form to make features in the experience.
- the sentio codec enables encoding and transmission of the data streams representing the various dimensions and features.
- the following are some of the dimensions that can be supported on a suitable experience platform, and the related data streams encoded by a suitable sentio codec. It will be appreciated that not all dimensions are necessarily available or needed on specific experience platforms or specific devices, and that the sentio codec can be implemented with general, all encompassing capabilities, or with only those capabilities needed for the specific implementation, or with a suitable subset.
- Video is the near or substantially real-time streaming of the video portion of a video or film with near real-time display and interaction.
- Audio is the near or substantially real-time streaming of the audio portion of a video, film, karaoke track, song, with near real-time sound and interaction.
- Live is the live display and/or access to a live video, film, or audio stream in near real-time that can be controlled by another experience dimension.
- a live display is not limited to single data stream.
- Encore is the replaying of a live video, film or audio content. This replaying can be the raw version as it was originally experienced, or some type of augmented version that has been edited, remixed, etc.
- Graphics is a display that contains graphic elements such as text, illustration, photos, freehand geometry and the attributes (size, color, location) associated with these elements. Graphics can be created and controlled using the experience input/output command dimension(s) (see below).
- Input/Output Command(s) are the ability to control the video, audio, picture, display, sound or interactions with human or device-based controls. Some examples of input/output commands include physical gestures or movements, voice/sound recognition, and keyboard or smart-phone device input(s).
- Interaction is how devices and participants interchange and respond with each other and with the content (user experience, video, graphics, audio, images, etc.) displayed in an experience.
- Interaction can include the defined behavior of an artifact or system and the responses provided to the user and/or player.
- Game Mechanics are rule-based system(s) that facilitate and encourage players to explore the properties of an experience space and other participants through the use of feedback mechanisms.
- Some services on the experience Platform that could support the game mechanics dimensions include leader boards, polling, like/dislike, featured players, star-ratings, bidding, rewarding, role-playing, problem-solving, etc.
- Ensemble is the interaction of several separate but often related parts of video, song, picture, story line, players, etc. that when woven together create a more engaging and immersive experience than if experienced in isolation.
- Auto Tune is the near real-time correction of pitch in vocal and/or instrumental performances. Auto Tune is used to disguise off-key inaccuracies and mistakes, and allows singer/players to hear back perfectly tuned vocal tracks without the need of singing in tune.
- Auto Filter is the near real-time augmentation of vocal and/or instrumental performances.
- Types of augmentation could include speeding up or slowing down the playback, increasing/decreasing the volume or pitch, or applying a celebrity-style filter to an audio track (like a Lady Gaga or Heavy-Metal filter).
- Remix is the near real-time creation of an alternative version of a song, track, video, image, etc. made from an original version or multiple original versions of songs, tracks, videos, images, etc.
- Viewing 360°/Panning is the near real-time viewing of the 360° horizontal movement of a streaming video feed on a fixed axis. Also the ability to for the player(s) to control and/or display alternative video or camera feeds from any point designated on this fixed axis.
- the experience platform 10 includes a plurality of devices 20 and a data center 40 .
- the devices 12 may include devices such as an iPhone 22 , an android 24 , a set top box 26 , a desktop computer 28 , and a netbook 30 . At least some of the devices 12 may be located in proximity with each other and coupled via a wireless network.
- a participant utilizes multiple devices 12 to enjoy a heterogeneous experience, such as using the iPhone 22 to control operation of the other devices. Multiple participants may also share devices at one location, or the devices may be distributed across various locations for different participants.
- Each device 12 has an experience agent 32 .
- the experience agent 32 includes a sentio codec and an API.
- the sentio codec can include 1) a variety of codecs for each segment of experience described above, 2) a hybrid network stack with network intelligence, 3) data about available devices, and 4) a QoE decision engine that makes decisions on how to encode. It will be appreciated that QoE is achieved through various strategies that work differently for each given experience (say a zombie karaoke game vs. live stadium rock concert experience), and adapt in real-time to the network and other available resources, know the devices involved and take advantages of various psychological tricks to conceal imperfections which inevitably arise, particularly when the provided experience is scaled for many participants and devices.
- the sentio codec and the API enable the experience agent 32 to communicate with and request services of the components of the data center 40 .
- the experience agent 32 facilitates direct interaction between other local devices. Because of the multi-dimensional aspect of the experience, the sentio codec and API should fully enable the desired experience. However, the functionality of the experience agent 32 , including the sentio codec, is typically tailored to the needs and capabilities of the specific device 12 on which the experience agent 32 is instantiated. In some embodiments, services implementing experience dimensions are implemented in a distributed manner across the devices 12 and the data center 40 . In other embodiments, the devices 12 have a very thin experience agent 32 with little functionality beyond a minimum API and sentio codec, and the bulk of the services and thus composition and direction of the experience are implemented within the data center 40 .
- Data center 40 includes an experience server 42 , a plurality of content servers 44 , and a service platform 46 .
- data center 40 can be hosted in a distributed manner in the “cloud,” and typically the elements of the data center 40 are coupled via a low latency network.
- the experience server 42 , servers 44 , and service platform 46 can be implemented on a single computer system, or more likely distributed across a variety of computer systems, and at various locations.
- the experience server 42 includes at least one experience agent 32 , an experience composition engine 48 , and an operating system 50 .
- the experience agent 32 again includes a sentio codec with the various capabilities as described herein.
- the experience composition engine 48 is defined and controlled by the experience provider to compose and direct the experience for one or more participants utilizing devices 12 .
- Direction and composition is accomplished, in part, by merging various content layers and other elements into dimensions generated from a variety of sources such as the service provider 42 , the devices 12 , the content servers 44 , and/or the service platform 46 .
- the content servers 44 may include a video server 52 , an ad server 54 , and a generic content server 56 .
- Any content suitable for encoding by the sentio codec of an experience agent can be included as an experience layer. These include well know forms such as video, audio, graphics, and text. As described in more detail earlier and below, other forms of content such as gestures, emotions, temperature, proximity, etc., are contemplated for encoding and inclusion in the experience via a sentio codec, and are suitable for creating dimensions and features of the experience.
- the service platform 46 includes at least one experience agent 32 , a plurality of service engines 60 , third party service engines 62 , and a monetization engine 64 .
- each service engine 60 or 62 has a unique, corresponding experience agent with a corresponding sentio codec.
- the sentio codecs may have separate code and utilize different and/or combinations of the same local hardware.
- the implementation may be distinct to each application.
- a single experience agent 32 can support multiple service engines 60 or 62 .
- the service engines and the monetization engines 64 can be instantiated on one server, or can be distributed across multiple servers.
- the service engines 60 correspond to engines generated by the service provider and can provide services such as audio remixing, gesture recognition, and other services referred to in the context of dimensions above, etc.
- Third party service engines 62 are services included in the service platform 46 by other parties.
- the service platform 46 may have the third-party service engines instantiated directly therein, or within the service platform 46 these may correspond to proxies which in turn make calls to servers under control of the third-parties.
- Monetization of the service platform 46 can be accomplished in a variety of manners.
- the monetization engine 64 may determine how and when to charge the experience provider for use of the services, as well as tracking for payment to third-parties for use of services from the third-party service engines 62 .
- FIG. 2 illustrates a block diagram of an experience agent 100 .
- the experience agent 100 includes an application programming interface (API) 102 and a sentio codec 104 .
- the API 102 is an interface which defines available services, and enables the different agents to communicate with one another and request services.
- the sentio codec 104 is a combination of hardware and/or software which enables encoding of many types of data streams for operations such as transmission and storage, and decoding for operations such as playback and editing.
- These data streams can include standard data such as video and audio. Additionally, the data can include graphics, sensor data, gesture data, and emotion data.
- FIG. 3 illustrates a block diagram of one embodiment of a sentio codec 200 .
- the sentio codec 200 includes a plurality of codecs such as video codecs 202 , audio codecs 204 , graphic language codecs 206 , sensor data codecs 208 , and emotion codecs 210 .
- the sentio codec 200 further includes a quality of experience (QoE) decision engine 212 and a network engine 214 .
- QoE quality of experience
- the codecs, the QoE decision engine 212 , and the network engine 214 work together to encode one or more data streams and transmit the encoded data according to a low-latency transfer protocol supporting the various encoded data types.
- One suitable low-latency protocol and more details related to the network engine 214 can be found in Vonog et al.'s U.S. patent application Ser. No. 12/569,876.
- the sentio codec 200 can be designed to take all aspects of the experience platform into consideration when executing the transfer protocol.
- the parameters and aspects include available network bandwidth, transmission device characteristics and receiving device characteristics.
- the sentio codec 200 can be implemented to be responsive to commands from an experience composition engine or other outside entity to determine how to prioritize data for transmission.
- audio is the most important component of an experience data stream.
- a specific application may desire to emphasize video or gesture commands.
- the sentio codec provides the capability of encoding data streams corresponding to many different senses or dimensions of an experience.
- a device 12 may include a video camera capturing video images and audio from a participant.
- the user image and audio data may be encoded and transmitted directly or, perhaps after some intermediate processing, via the experience composition engine 48 , to the service platform 46 where one or a combination of the service engines can analyze the data stream to make a determination about an emotion of the participant.
- This emotion can then be encoded by the sentio codec and transmitted to the experience composition engine 48 , which in turn can incorporate this into a dimension of the experience.
- a participant gesture can be captured as a data stream, e.g. by a motion sensor or a camera on device 12 , and then transmitted to the service platform 46 , where the gesture can be interpreted, and transmitted to the experience composition engine 48 or directly back to one or more devices 12 for incorporation into a dimension of the experience.
- the sentio codec delivers the best QoE to a consumer on the device of their choice through current network. This is accomplished through a variety of mechanisms, selected and implemented based on the specific application and available resources.
- the sentio codec encodes multi-dimensional data streams in real-time, adapting to network capability.
- a QoE engine operating within the sentio codec a makes decisions on how to use different available codecs.
- the network stack can be implemented as hybrid, as described above, and in further detail with reference to Vonog et al.'s U.S. patent application Ser. No. 12/569,876.
Abstract
Description
- The present application is a continuation of co-pending U.S. patent application Ser. No. 13/136,870 entitled “Experience or “Sentio” Codecs, and Methods and Systems for Improving QoE and Encoding Based on QoE Experiences”, filed Aug. 12, 2011, which claims the benefit of and priority to U.S. Provisional Patent Application No. 61/373,236 entitled “Experience or “Sentio” Codecs, and Methods and Systems for Improving QoE and Encoding Based on QoE Experiences,” filed on Aug. 12, 2010, both of which are hereby, incorporated by reference.
- 1. Field of Invention
- The present teaching relates to experience or “sentio” codecs enabling encoding and transmission for data streams involving a variety of dimensions and data types including video, group participation, gesture recognition, heterogeneous device use, emotions, etc.
- 2. Summary of the Invention
- The present invention contemplates a variety of experience or “sentio” codecs, and methods and systems for enabling an experience platform, as well as a Quality of Experience (QoE) engine which allows the sentio codec to select a suitable encoding engine or device. As will be described in more detail below, the sentio codec is capable of encoding and transmitting data streams that correspond to participant experiences with a variety of different dimensions and features. As will be appreciated, the following description provides one paradigm for understanding the multi-dimensional experience available to the participants, and as implemented utilizing a sentio codec. There are many suitable ways of describing, characterizing and implementing the sentio codec and experience platform contemplated herein.
- These and other objects, features and characteristics of the present invention will become more apparent to those skilled in the art from a study of the following detailed description in conjunction with the appended claims and drawings, all of which form a part of this specification. In the drawings:
-
FIG. 1 illustrates a system architecture for composing and directing user experiences; -
FIG. 2 is a block diagram of an experience agent; -
FIG. 3 is a block diagram of a sentio codec; and -
FIG. 4 provides a screen shot useful for illustrating how a hybrid encoding scheme can be used to accomplish low-latency transmission. - The present invention contemplates a variety of experience or “sentio” codecs, and methods and systems for enabling an experience platform, as well as a Quality of Experience (QoS) engine which allows the sentio codec to select a suitable encoding engine or device. As will be described in more detail below, the sentio codec is capable of encoding and transmitting data streams that correspond to participant experiences with a variety of different dimensions and features. (The term “sentio” is Latin roughly corresponding to perception or to perceive with one's senses, hence the original nomenclature “sensio codec.”)
- The primary goal of a video codec is to achieve maximum compression rate for digital video while maintaining great picture quality video; audio codecs are similar. But video and audio codecs alone are insufficient to generate and capture a full experience, such as a real-time experience enabled by hybrid encoding, and encoding of other experience aspects such as gestures, emotions, etc.
-
FIG. 4 will now be described to provide an example experience showing 4 layers where video encoding alone is inadequate. (The “layer” concept will be described below in more detail with reference toFIGS. 1-3 .) A first layer is generated by Autodesk 3ds Max instantiated on a suitable layer source, such as on an experience server or a content server. A second layer is an interactive frame around the 3ds Max layer, and in this example is generated on a client device by an experience agent. A third layer is the black box in the bottom-left corner with the text “FPS” and “bandwidth”, and is generated on the client device but pulls data by accessing a service engine available on the service platform. A fourth layer is a red-green-yellow grid which demonstrates an aspect of a low-latency transfer protocol (e.g., different regions being selectively encoded) and is generated and computed on the service platform, and then merged with the 3ds Max layer on the experience server. -
FIG. 4 illustrates how a hybrid encoding approach can be used to accomplish low-latency transmission. The first layer provides an Autodesk 3ds Max image including a rotating teapot, the first layer moving images, static or nearly static images, and graphic and/or text portions. Rather then encoding all the information with a video encoder alone, a hybrid approach encoding some regions with a video encoder, other regions with a picture encoder, and other portions as command, results in better transmission results, and can be optimized based on factors such as the state of the network and the capabilities of end devices. These different encoding regions are illustrated by the different coloring of the red-green-yellow grid of layer 4. One example of this low-latency protocol is described in more detail in Vonog et al.'s U.S. patent application Ser. No. 12/569,876, filed Sep. 29, 2009, and incorporated herein by reference for all purposes including the low-latency protocol and related features such as the network engine and network stack arrangement. - As is seen from the example of
FIGS. 4 , a video codec alone is inadequate to accomplish the hybrid encoding scheme covering video, pictures and commands. While it is theoretically possible to encode the entire first layer using only a video codec, latency and other issues can prohibit real-time and/or quality experiences. A low-latency protocol can solve this problem by efficiently encoding the data. - In another example, a multiplicity of video codecs can be used to improve encoding and transmission. For example, h.264 can be used if a hardware decoder is available, thus saving battery life and improving performance, or a better video codec (e.g., low latency) can be used if the device fails to support h.264.
- As yet another example, consider the case of multiple mediums where an ability to take into account the nature of human perception would be beneficial. For example, assume we have video and audio information. If network quality degrades, it could be better to prioritize audio and allow the video to degrade. To do so would require using psychoacoustics to improve the QoE.
- Accordingly, the present teaching contemplates an experience or sentio codec capable of encoding and transmitting data streams that correspond to experiences with a variety of different dimensions and features. These dimensions include known audio and video, but further may include any conceivable element of a participant experience, such as gestures, gestures+voice commands, “game mechanics” (which you can use to boost QoE when current conditions (such as network) do not allow you to do so—i.e. apply sound distortion effect specific to a given experience when loss of data happened), emotions (perhaps as detected via voice or facial expressions, various sensor data, microphone input, etc.
- It is also contemplated that virtual experiences can be encoded via the sentio codec. According to one embodiment, virtual goods are evolved into virtual experiences. Virtual experiences expand upon limitations imposed by virtual goods by adding additional dimensions to the virtual goods. By way of example, User A transmits flowers as a virtual good to User B. The transmission of the virtual flowers is enhanced by adding emotion by way of sound, for example. The virtual flowers are also changed to a virtual experience when User B can do something with the flowers, for example User B can affect the flowers through any sort of motion or gesture. User A can also transmit the virtual goods to User B by making a “throwing” gesture using a mobile device, so as to “toss” the virtual goods to User B.
- The sentio codec improves the QoE to a consumer or experience participant on the device of their choice. This is accomplished through a variety of mechanisms, selected and implemented, possibly dynamically, based on the specific application and available resources. In certain embodiments, the sentio codec encodes multi-dimensional data streams in real-time, adapting to network capability. A QoE engine operating within the sentio codec a makes decisions on how to use different available codecs. The network stack can be implemented as hybrid, as described above, and in further detail with reference to Vonog et al.'s U.S. patent application Ser. No. 12/569,876.
- The sentio codec can include 1) a variety of codecs for each segment of experience described above, 2) a hybrid network stack with network intelligence, 3) data about available devices, and 4) a QoE engine that makes decisions on how to encode. It will be appreciated that QoE is achieved through various strategies that work differently for each given experience (say a zombie karaoke game vs. live stadium rock concert experience), and adapt in real-time to the network and other available resources, know the devices involved and take advantages of various psychological tricks to conceal imperfections which inevitably arise, particularly when the provided experience is scaled for many participants and devices.
-
FIG. 1 illustrates a block diagram of asystem 10. Thesystem 10 can be viewed as an “experience platform” or system architecture for composing and directing a participant experience. As will be appreciated, the experience platform described herein provides, by way of example only, one platform suitable for incorporating and taking advantage of the sentio codec described herein. In one embodiment, theexperience platform 10 is provided by a service provider to enable an experience provider to compose and direct a participant experience. The participant experience can involve one or more experience participants. The experience provider can create an experience with a variety of dimensions, as will be explained further now. The sentio codec enables the encoding and transmission of data streams representing this variety of dimensions. As will be appreciated, the following description provides one paradigm for understanding the multi-dimensional experience available to the participants. There are many suitable ways of describing, characterizing and implementing the experience platform contemplated herein. - In general, services are defined at an API layer of the experience platform. The services are categorized into “dimensions.” The dimension(s) can be recombined into “layers.” The layers form to make features in the experience. The sentio codec enables encoding and transmission of the data streams representing the various dimensions and features.
- By way of example, the following are some of the dimensions that can be supported on a suitable experience platform, and the related data streams encoded by a suitable sentio codec. It will be appreciated that not all dimensions are necessarily available or needed on specific experience platforms or specific devices, and that the sentio codec can be implemented with general, all encompassing capabilities, or with only those capabilities needed for the specific implementation, or with a suitable subset.
- Video—is the near or substantially real-time streaming of the video portion of a video or film with near real-time display and interaction.
- Audio—is the near or substantially real-time streaming of the audio portion of a video, film, karaoke track, song, with near real-time sound and interaction.
- Live—is the live display and/or access to a live video, film, or audio stream in near real-time that can be controlled by another experience dimension. A live display is not limited to single data stream.
- Encore—is the replaying of a live video, film or audio content. This replaying can be the raw version as it was originally experienced, or some type of augmented version that has been edited, remixed, etc.
- Graphics—is a display that contains graphic elements such as text, illustration, photos, freehand geometry and the attributes (size, color, location) associated with these elements. Graphics can be created and controlled using the experience input/output command dimension(s) (see below).
- Input/Output Command(s)—are the ability to control the video, audio, picture, display, sound or interactions with human or device-based controls. Some examples of input/output commands include physical gestures or movements, voice/sound recognition, and keyboard or smart-phone device input(s).
- Interaction—is how devices and participants interchange and respond with each other and with the content (user experience, video, graphics, audio, images, etc.) displayed in an experience. Interaction can include the defined behavior of an artifact or system and the responses provided to the user and/or player.
- Game Mechanics—are rule-based system(s) that facilitate and encourage players to explore the properties of an experience space and other participants through the use of feedback mechanisms. Some services on the experience Platform that could support the game mechanics dimensions include leader boards, polling, like/dislike, featured players, star-ratings, bidding, rewarding, role-playing, problem-solving, etc.
- Ensemble—is the interaction of several separate but often related parts of video, song, picture, story line, players, etc. that when woven together create a more engaging and immersive experience than if experienced in isolation.
- Auto Tune—is the near real-time correction of pitch in vocal and/or instrumental performances. Auto Tune is used to disguise off-key inaccuracies and mistakes, and allows singer/players to hear back perfectly tuned vocal tracks without the need of singing in tune.
- Auto Filter—is the near real-time augmentation of vocal and/or instrumental performances. Types of augmentation could include speeding up or slowing down the playback, increasing/decreasing the volume or pitch, or applying a celebrity-style filter to an audio track (like a Lady Gaga or Heavy-Metal filter).
- Remix—is the near real-time creation of an alternative version of a song, track, video, image, etc. made from an original version or multiple original versions of songs, tracks, videos, images, etc.
- Viewing 360°/Panning—is the near real-time viewing of the 360° horizontal movement of a streaming video feed on a fixed axis. Also the ability to for the player(s) to control and/or display alternative video or camera feeds from any point designated on this fixed axis.
- Turning back to
FIG. 1 , theexperience platform 10 includes a plurality ofdevices 20 and adata center 40. The devices 12 may include devices such as aniPhone 22, an android 24, aset top box 26, adesktop computer 28, and anetbook 30. At least some of the devices 12 may be located in proximity with each other and coupled via a wireless network. In certain embodiments, a participant utilizes multiple devices 12 to enjoy a heterogeneous experience, such as using theiPhone 22 to control operation of the other devices. Multiple participants may also share devices at one location, or the devices may be distributed across various locations for different participants. - Each device 12 has an
experience agent 32. Theexperience agent 32 includes a sentio codec and an API. The sentio codec can include 1) a variety of codecs for each segment of experience described above, 2) a hybrid network stack with network intelligence, 3) data about available devices, and 4) a QoE decision engine that makes decisions on how to encode. It will be appreciated that QoE is achieved through various strategies that work differently for each given experience (say a zombie karaoke game vs. live stadium rock concert experience), and adapt in real-time to the network and other available resources, know the devices involved and take advantages of various psychological tricks to conceal imperfections which inevitably arise, particularly when the provided experience is scaled for many participants and devices. - The sentio codec and the API enable the
experience agent 32 to communicate with and request services of the components of thedata center 40. Theexperience agent 32 facilitates direct interaction between other local devices. Because of the multi-dimensional aspect of the experience, the sentio codec and API should fully enable the desired experience. However, the functionality of theexperience agent 32, including the sentio codec, is typically tailored to the needs and capabilities of the specific device 12 on which theexperience agent 32 is instantiated. In some embodiments, services implementing experience dimensions are implemented in a distributed manner across the devices 12 and thedata center 40. In other embodiments, the devices 12 have a verythin experience agent 32 with little functionality beyond a minimum API and sentio codec, and the bulk of the services and thus composition and direction of the experience are implemented within thedata center 40. -
Data center 40 includes anexperience server 42, a plurality ofcontent servers 44, and aservice platform 46. As will be appreciated,data center 40 can be hosted in a distributed manner in the “cloud,” and typically the elements of thedata center 40 are coupled via a low latency network. Theexperience server 42,servers 44, andservice platform 46 can be implemented on a single computer system, or more likely distributed across a variety of computer systems, and at various locations. - The
experience server 42 includes at least oneexperience agent 32, anexperience composition engine 48, and anoperating system 50. Theexperience agent 32 again includes a sentio codec with the various capabilities as described herein. In one embodiment, theexperience composition engine 48 is defined and controlled by the experience provider to compose and direct the experience for one or more participants utilizing devices 12. Direction and composition is accomplished, in part, by merging various content layers and other elements into dimensions generated from a variety of sources such as theservice provider 42, the devices 12, thecontent servers 44, and/or theservice platform 46. - The
content servers 44 may include avideo server 52, anad server 54, and ageneric content server 56. Any content suitable for encoding by the sentio codec of an experience agent can be included as an experience layer. These include well know forms such as video, audio, graphics, and text. As described in more detail earlier and below, other forms of content such as gestures, emotions, temperature, proximity, etc., are contemplated for encoding and inclusion in the experience via a sentio codec, and are suitable for creating dimensions and features of the experience. - The
service platform 46 includes at least oneexperience agent 32, a plurality ofservice engines 60, thirdparty service engines 62, and amonetization engine 64. In some embodiments, eachservice engine single experience agent 32 can supportmultiple service engines monetization engines 64 can be instantiated on one server, or can be distributed across multiple servers. Theservice engines 60 correspond to engines generated by the service provider and can provide services such as audio remixing, gesture recognition, and other services referred to in the context of dimensions above, etc. Thirdparty service engines 62 are services included in theservice platform 46 by other parties. Theservice platform 46 may have the third-party service engines instantiated directly therein, or within theservice platform 46 these may correspond to proxies which in turn make calls to servers under control of the third-parties. - Monetization of the
service platform 46 can be accomplished in a variety of manners. For example, themonetization engine 64 may determine how and when to charge the experience provider for use of the services, as well as tracking for payment to third-parties for use of services from the third-party service engines 62. -
FIG. 2 illustrates a block diagram of anexperience agent 100. Theexperience agent 100 includes an application programming interface (API) 102 and asentio codec 104. TheAPI 102 is an interface which defines available services, and enables the different agents to communicate with one another and request services. - The
sentio codec 104 is a combination of hardware and/or software which enables encoding of many types of data streams for operations such as transmission and storage, and decoding for operations such as playback and editing. These data streams can include standard data such as video and audio. Additionally, the data can include graphics, sensor data, gesture data, and emotion data. -
FIG. 3 illustrates a block diagram of one embodiment of asentio codec 200. Thesentio codec 200 includes a plurality of codecs such asvideo codecs 202,audio codecs 204,graphic language codecs 206,sensor data codecs 208, andemotion codecs 210. Thesentio codec 200 further includes a quality of experience (QoE)decision engine 212 and anetwork engine 214. The codecs, theQoE decision engine 212, and thenetwork engine 214 work together to encode one or more data streams and transmit the encoded data according to a low-latency transfer protocol supporting the various encoded data types. One suitable low-latency protocol and more details related to thenetwork engine 214 can be found in Vonog et al.'s U.S. patent application Ser. No. 12/569,876. - The
sentio codec 200 can be designed to take all aspects of the experience platform into consideration when executing the transfer protocol. The parameters and aspects include available network bandwidth, transmission device characteristics and receiving device characteristics. Additionally, thesentio codec 200 can be implemented to be responsive to commands from an experience composition engine or other outside entity to determine how to prioritize data for transmission. In many applications, because of human response, audio is the most important component of an experience data stream. However, a specific application may desire to emphasize video or gesture commands. - The sentio codec provides the capability of encoding data streams corresponding to many different senses or dimensions of an experience. For example, a device 12 may include a video camera capturing video images and audio from a participant. The user image and audio data may be encoded and transmitted directly or, perhaps after some intermediate processing, via the
experience composition engine 48, to theservice platform 46 where one or a combination of the service engines can analyze the data stream to make a determination about an emotion of the participant. This emotion can then be encoded by the sentio codec and transmitted to theexperience composition engine 48, which in turn can incorporate this into a dimension of the experience. Similarly a participant gesture can be captured as a data stream, e.g. by a motion sensor or a camera on device 12, and then transmitted to theservice platform 46, where the gesture can be interpreted, and transmitted to theexperience composition engine 48 or directly back to one or more devices 12 for incorporation into a dimension of the experience. - The sentio codec delivers the best QoE to a consumer on the device of their choice through current network. This is accomplished through a variety of mechanisms, selected and implemented based on the specific application and available resources. In certain embodiments, the sentio codec encodes multi-dimensional data streams in real-time, adapting to network capability. A QoE engine operating within the sentio codec a makes decisions on how to use different available codecs. The network stack can be implemented as hybrid, as described above, and in further detail with reference to Vonog et al.'s U.S. patent application Ser. No. 12/569,876.
- In addition to the above mentioned examples, various other modifications and alterations of the invention may be made without departing from the invention. Accordingly, the above disclosure is not to be considered as limiting and the appended claims are to be interpreted as encompassing the true spirit and the entire scope of the invention.
Claims (17)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/363,187 US20120134409A1 (en) | 2010-08-12 | 2012-01-31 | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES |
US15/087,657 US20160219279A1 (en) | 2010-08-12 | 2016-03-31 | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US37323610P | 2010-08-12 | 2010-08-12 | |
US13/136,870 US9172979B2 (en) | 2010-08-12 | 2011-08-12 | Experience or “sentio” codecs, and methods and systems for improving QoE and encoding based on QoE experiences |
US13/363,187 US20120134409A1 (en) | 2010-08-12 | 2012-01-31 | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/136,870 Continuation US9172979B2 (en) | 2010-08-12 | 2011-08-12 | Experience or “sentio” codecs, and methods and systems for improving QoE and encoding based on QoE experiences |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/087,657 Continuation US20160219279A1 (en) | 2010-08-12 | 2016-03-31 | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES |
Publications (1)
Publication Number | Publication Date |
---|---|
US20120134409A1 true US20120134409A1 (en) | 2012-05-31 |
Family
ID=45564811
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/136,870 Expired - Fee Related US9172979B2 (en) | 2010-08-12 | 2011-08-12 | Experience or “sentio” codecs, and methods and systems for improving QoE and encoding based on QoE experiences |
US13/363,187 Abandoned US20120134409A1 (en) | 2010-08-12 | 2012-01-31 | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES |
US15/087,657 Abandoned US20160219279A1 (en) | 2010-08-12 | 2016-03-31 | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/136,870 Expired - Fee Related US9172979B2 (en) | 2010-08-12 | 2011-08-12 | Experience or “sentio” codecs, and methods and systems for improving QoE and encoding based on QoE experiences |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/087,657 Abandoned US20160219279A1 (en) | 2010-08-12 | 2016-03-31 | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES |
Country Status (1)
Country | Link |
---|---|
US (3) | US9172979B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9172979B2 (en) | 2010-08-12 | 2015-10-27 | Net Power And Light, Inc. | Experience or “sentio” codecs, and methods and systems for improving QoE and encoding based on QoE experiences |
US9557817B2 (en) | 2010-08-13 | 2017-01-31 | Wickr Inc. | Recognizing gesture inputs using distributed processing of sensor data from multiple sensors |
Families Citing this family (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8390670B1 (en) | 2008-11-24 | 2013-03-05 | Shindig, Inc. | Multiparty communications systems and methods that optimize communications based on mode and available bandwidth |
US9401937B1 (en) | 2008-11-24 | 2016-07-26 | Shindig, Inc. | Systems and methods for facilitating communications amongst multiple users |
US9344745B2 (en) | 2009-04-01 | 2016-05-17 | Shindig, Inc. | Group portraits composed using video chat systems |
US9712579B2 (en) | 2009-04-01 | 2017-07-18 | Shindig. Inc. | Systems and methods for creating and publishing customizable images from within online events |
US8779265B1 (en) | 2009-04-24 | 2014-07-15 | Shindig, Inc. | Networks of portable electronic devices that collectively generate sound |
WO2012021173A2 (en) | 2010-08-12 | 2012-02-16 | Net Power And Light Inc. | System architecture and methods for experiential computing |
EP2630630A2 (en) | 2010-10-21 | 2013-08-28 | Net Power And Light, Inc. | System architecture and method for composing and directing participant experiences |
US9167296B2 (en) * | 2012-02-28 | 2015-10-20 | Qualcomm Incorporated | Customized playback at sink device in wireless display system |
US11159804B1 (en) * | 2012-09-13 | 2021-10-26 | Arris Enterprises Llc | QoE feedback based intelligent video transport stream tuning |
EP2999265B1 (en) * | 2013-06-13 | 2020-12-02 | Huawei Technologies Co., Ltd. | Method, device, and system for network handover |
US10271010B2 (en) | 2013-10-31 | 2019-04-23 | Shindig, Inc. | Systems and methods for controlling the display of content |
WO2015062087A1 (en) | 2013-11-01 | 2015-05-07 | 华为技术有限公司 | Network switching method, device and system |
US9952751B2 (en) | 2014-04-17 | 2018-04-24 | Shindig, Inc. | Systems and methods for forming group communications within an online event |
US9733333B2 (en) | 2014-05-08 | 2017-08-15 | Shindig, Inc. | Systems and methods for monitoring participant attentiveness within events and group assortments |
US9711181B2 (en) | 2014-07-25 | 2017-07-18 | Shindig. Inc. | Systems and methods for creating, editing and publishing recorded videos |
US10469630B2 (en) | 2014-12-05 | 2019-11-05 | Facebook, Inc. | Embedded RTCP packets |
US9729601B2 (en) | 2014-12-05 | 2017-08-08 | Facebook, Inc. | Decoupled audio and video codecs |
US9667801B2 (en) * | 2014-12-05 | 2017-05-30 | Facebook, Inc. | Codec selection based on offer |
US9729726B2 (en) | 2014-12-05 | 2017-08-08 | Facebook, Inc. | Seamless codec switching |
US9729287B2 (en) | 2014-12-05 | 2017-08-08 | Facebook, Inc. | Codec with variable packet size |
US10506004B2 (en) | 2014-12-05 | 2019-12-10 | Facebook, Inc. | Advanced comfort noise techniques |
US9734410B2 (en) | 2015-01-23 | 2017-08-15 | Shindig, Inc. | Systems and methods for analyzing facial expressions within an online classroom to gauge participant attentiveness |
US10133916B2 (en) | 2016-09-07 | 2018-11-20 | Steven M. Gottlieb | Image and identity validation in video chat events |
Citations (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020051493A1 (en) * | 2000-06-26 | 2002-05-02 | Shin Jae-Seob | Moving picture mailing system and method |
US20040128350A1 (en) * | 2002-03-25 | 2004-07-01 | Lou Topfl | Methods and systems for real-time virtual conferencing |
US20050007965A1 (en) * | 2003-05-24 | 2005-01-13 | Hagen David A. | Conferencing system |
US20050100100A1 (en) * | 2003-11-12 | 2005-05-12 | Sony Corporation | Apparatus and method for use in providing dynamic bit rate encoding |
US20060183547A1 (en) * | 2005-02-11 | 2006-08-17 | Mcmonigle Mace | Fantasy sports television programming systems and methods |
WO2007084994A2 (en) * | 2006-01-20 | 2007-07-26 | Qualcomm Incorporated | Method and apparatus for determining an encoding method based on a distortion value related to error concealment |
US20080039205A1 (en) * | 2006-08-11 | 2008-02-14 | Jonathan Ackley | Method and/or system for mobile interactive gaming |
US20080195744A1 (en) * | 2007-02-14 | 2008-08-14 | Microsoft Corporation | Adaptive media playback |
US7516255B1 (en) * | 2005-03-30 | 2009-04-07 | Teradici Corporation | Method and apparatus for providing a low-latency connection between a data processor and a remote graphical user interface over a network |
US20090118017A1 (en) * | 2002-12-10 | 2009-05-07 | Onlive, Inc. | Hosting and broadcasting virtual events using streaming interactive video |
US20090115843A1 (en) * | 2005-04-04 | 2009-05-07 | Gary Richard Sohmers | Interactive Television System and Method |
US20100149096A1 (en) * | 2008-12-17 | 2010-06-17 | Migos Charles J | Network management using interaction with display surface |
US20110116540A1 (en) * | 2009-11-18 | 2011-05-19 | General Instrument Corporation | Multimedia Content Handling in a Home-Network System |
US20110246908A1 (en) * | 2010-04-01 | 2011-10-06 | Microsoft Corporation | Interactive and shared viewing experience |
US8171154B2 (en) * | 2009-09-29 | 2012-05-01 | Net Power And Light, Inc. | Method and system for low-latency transfer protocol |
US20120151541A1 (en) * | 2010-10-21 | 2012-06-14 | Stanislav Vonog | System architecture and method for composing and directing participant experiences |
US20130019184A1 (en) * | 2011-07-11 | 2013-01-17 | Net Power And Light, Inc. | Methods and systems for virtual experiences |
US8903740B2 (en) * | 2010-08-12 | 2014-12-02 | Net Power And Light, Inc. | System architecture and methods for composing and directing participant experiences |
Family Cites Families (76)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5491743A (en) * | 1994-05-24 | 1996-02-13 | International Business Machines Corporation | Virtual conference system and terminal apparatus therefor |
US6144991A (en) | 1998-02-19 | 2000-11-07 | Telcordia Technologies, Inc. | System and method for managing interactions between users in a browser-based telecommunications network |
US8150757B1 (en) | 1999-07-01 | 2012-04-03 | Foundationip, Llc | Web-based infomediary for intellectual property transfer |
US6779016B1 (en) | 1999-08-23 | 2004-08-17 | Terraspring, Inc. | Extensible computing system |
US7590739B2 (en) | 1999-11-22 | 2009-09-15 | Akamai Technologies, Inc. | Distributed on-demand computing system |
US7298975B2 (en) | 2000-07-13 | 2007-11-20 | L-3 Communications Integrated Systems L.P. | Synchronous collapsed ring architecture for real-time signal switching and distribution |
CA2426913A1 (en) | 2000-10-20 | 2002-05-23 | Wavexpress, Inc. | Broadcast browser including multi-media tool overlay and method of providing a coverged multi-media display including user-enhanced data |
US7184984B2 (en) | 2000-11-17 | 2007-02-27 | Valaquenta Intellectual Properties Limited | Global electronic trading system |
US7172508B2 (en) | 2001-01-23 | 2007-02-06 | Burton Simon | Multi-person parimutuel betting games based on sporting events |
US7171485B2 (en) | 2001-10-17 | 2007-01-30 | Velcero Broadband Applications, Llc | Broadband network system configured to transport audio or video at the transport layer, and associated method |
US20030074554A1 (en) | 2001-10-17 | 2003-04-17 | Roach Wayne C. | Broadband interface unit and associated method |
US20030188320A1 (en) | 2002-04-02 | 2003-10-02 | Intervideo, Inc. | Method and system for a distributed digital video recorder |
US20090100452A1 (en) | 2002-05-01 | 2009-04-16 | Brandon Lee Hudgeons | Interactive multi-media system |
US7945636B2 (en) | 2002-05-15 | 2011-05-17 | In-Store Broadcasting Network, Llc | Providing a multi-tier enterprise level application |
US8387099B2 (en) | 2002-12-10 | 2013-02-26 | Ol2, Inc. | System for acceleration of web page delivery |
US7321384B1 (en) | 2003-06-03 | 2008-01-22 | Cisco Technology, Inc. | Method and apparatus for using far end camera control (FECC) messages to implement participant and layout selection in a multipoint videoconference |
US7428000B2 (en) | 2003-06-26 | 2008-09-23 | Microsoft Corp. | System and method for distributed meetings |
ATE356499T1 (en) | 2004-01-06 | 2007-03-15 | Cit Alcatel | PHYSICAL LAYER SESSION RESOURCE BROKER |
US7680694B2 (en) | 2004-03-11 | 2010-03-16 | American Express Travel Related Services Company, Inc. | Method and apparatus for a user to shop online in a three dimensional virtual reality setting |
CN101103630B (en) | 2004-10-05 | 2010-11-17 | 维克多曼克斯公司 | Method and system for authorizing multimedia multicasting |
KR100614930B1 (en) | 2004-12-03 | 2006-08-25 | 한국전자통신연구원 | Method of managing platform profile for digital home service |
CA2632579A1 (en) | 2004-12-13 | 2006-06-22 | Coldspark, Inc. | Electronic message delivery system including a network device |
KR20060083034A (en) | 2005-01-14 | 2006-07-20 | 정치영 | On-line shopping system using on-line game and avatar and on-line shopping method using thereof |
US20070094691A1 (en) | 2005-10-24 | 2007-04-26 | Gazdzinski Robert F | Method and apparatus for on-demand content transmission and control over networks |
US8347341B2 (en) | 2006-03-16 | 2013-01-01 | Time Warner Cable Inc. | Methods and apparatus for centralized content and data delivery |
US20070271580A1 (en) | 2006-05-16 | 2007-11-22 | Bellsouth Intellectual Property Corporation | Methods, Apparatus and Computer Program Products for Audience-Adaptive Control of Content Presentation Based on Sensed Audience Demographics |
US8239286B2 (en) | 2006-06-29 | 2012-08-07 | Microsoft Corporation | Medium and system for location-based E-commerce for mobile communication devices |
EP2475166A1 (en) | 2006-07-31 | 2012-07-11 | United Video Properties, Inc. | Systems and methods for providing media guidance planners |
US20100004977A1 (en) | 2006-09-05 | 2010-01-07 | Innerscope Research Llc | Method and System For Measuring User Experience For Interactive Activities |
US9762861B2 (en) | 2006-12-05 | 2017-09-12 | Excalibur Ip, Llc | Telepresence via wireless streaming multicast |
US20080139301A1 (en) | 2006-12-11 | 2008-06-12 | Ole-Ivar Holthe | System and method for sharing gaming experiences |
KR100857178B1 (en) | 2006-12-14 | 2008-09-05 | 주식회사 펄센 | Goods deal intermediating system using coordinating and method of the same |
US8656440B2 (en) | 2006-12-27 | 2014-02-18 | Verizon Patent And Licensing Inc. | Method and system of providing a virtual community for participation in a remote event |
WO2008091695A1 (en) | 2007-01-25 | 2008-07-31 | Samuel Pierce Baron | Virtual social interactions |
US7949890B2 (en) * | 2007-01-31 | 2011-05-24 | Net Power And Light, Inc. | Method and system for precise synchronization of audio and video streams during a distributed communication session with multiple participants |
US8453178B2 (en) | 2007-02-23 | 2013-05-28 | At&T Intellectual Property I, L.P. | System and method for presenting media services |
WO2008134745A1 (en) | 2007-04-30 | 2008-11-06 | Gesturetek, Inc. | Mobile video-based therapy |
US20090013263A1 (en) | 2007-06-21 | 2009-01-08 | Matthew Jonathan Fortnow | Method and apparatus for selecting events to be displayed at virtual venues and social networking |
US7769806B2 (en) | 2007-10-24 | 2010-08-03 | Social Communications Company | Automated real-time data stream switching in a shared virtual area communication environment |
US20100113140A1 (en) | 2007-11-02 | 2010-05-06 | Bally Gaming, Inc. | Gesture Enhanced Input Device |
US9462020B2 (en) | 2008-01-16 | 2016-10-04 | Qualcomm Incorporated | Intelligent client: multiple channel switching over a digital broadcast network |
US9067150B2 (en) | 2008-01-19 | 2015-06-30 | Lamplight Games | System and method for providing interactive content for multiple networked users in a shared venue using short messaging service communication |
US9503562B2 (en) * | 2008-03-19 | 2016-11-22 | Universal Electronics Inc. | System and method for appliance control via a personal communication or entertainment device |
US20110292181A1 (en) | 2008-04-16 | 2011-12-01 | Canesta, Inc. | Methods and systems using three-dimensional sensing for user interaction with applications |
KR20090113158A (en) | 2008-04-25 | 2009-10-29 | (주)에이치앤비솔루션 | A system for intelligent user request routing and reception based on the presence and the system for utilizing and managing collective intelligence based on presence using the same |
EP2304588A4 (en) | 2008-06-11 | 2011-12-21 | Teliris Inc | Surface computing collaboration system, method and apparatus |
JP4986243B2 (en) | 2008-07-04 | 2012-07-25 | Kddi株式会社 | Transmitting apparatus, method and program for controlling number of layers of media stream |
US8547342B2 (en) * | 2008-12-22 | 2013-10-01 | Verizon Patent And Licensing Inc. | Gesture-based delivery from mobile device |
US8572271B2 (en) | 2009-01-29 | 2013-10-29 | Qualcomm Incorporated | Multimedia management system for seamless multimedia content mobility |
US8260883B2 (en) | 2009-04-01 | 2012-09-04 | Wimm Labs, Inc. | File sharing between devices |
US8639046B2 (en) | 2009-05-04 | 2014-01-28 | Mamigo Inc | Method and system for scalable multi-user interactive visualization |
US20120127100A1 (en) | 2009-06-29 | 2012-05-24 | Michael Domenic Forte | Asynchronous motion enabled data transfer techniques for mobile devices |
US20110046980A1 (en) | 2009-07-06 | 2011-02-24 | Onerecovery, Inc. | Information security for recovery based social networking |
EP2486469B1 (en) | 2009-10-07 | 2017-12-13 | Elliptic Laboratories AS | Method and system for implementing a touchless user interface |
EP2490776A1 (en) | 2009-10-19 | 2012-08-29 | Koninklijke Philips Electronics N.V. | Device and method for conditionally transmitting data |
US10264029B2 (en) | 2009-10-30 | 2019-04-16 | Time Warner Cable Enterprises Llc | Methods and apparatus for packetized content delivery over a content delivery network |
US20110145817A1 (en) | 2009-12-16 | 2011-06-16 | Grzybowski Carl E | Adaptive virtual environment management system |
US7955175B1 (en) | 2009-12-17 | 2011-06-07 | Face It Applications LLC | Role based game play on a social network |
US20110163944A1 (en) | 2010-01-05 | 2011-07-07 | Apple Inc. | Intuitive, gesture-based communications with physics metaphors |
US8756532B2 (en) | 2010-01-21 | 2014-06-17 | Cisco Technology, Inc. | Using a gesture to transfer an object across multiple multi-touch devices |
US20110209089A1 (en) | 2010-02-25 | 2011-08-25 | Hinckley Kenneth P | Multi-screen object-hold and page-change gesture |
US20110244954A1 (en) | 2010-03-10 | 2011-10-06 | Oddmobb, Inc. | Online social media game |
US20110225515A1 (en) | 2010-03-10 | 2011-09-15 | Oddmobb, Inc. | Sharing emotional reactions to social media |
US20110271208A1 (en) | 2010-04-30 | 2011-11-03 | American Teleconferencing Services Ltd. | Location-Aware Conferencing With Entertainment Options |
US9245177B2 (en) | 2010-06-02 | 2016-01-26 | Microsoft Technology Licensing, Llc | Limiting avatar gesture display |
US9542091B2 (en) | 2010-06-04 | 2017-01-10 | Apple Inc. | Device, method, and graphical user interface for navigating through a user interface using a dynamic object selection indicator |
US9172979B2 (en) | 2010-08-12 | 2015-10-27 | Net Power And Light, Inc. | Experience or “sentio” codecs, and methods and systems for improving QoE and encoding based on QoE experiences |
US9557817B2 (en) | 2010-08-13 | 2017-01-31 | Wickr Inc. | Recognizing gesture inputs using distributed processing of sensor data from multiple sensors |
WO2012021901A2 (en) | 2010-08-13 | 2012-02-16 | Net Power And Light Inc. | Methods and systems for virtual experiences |
US20120060101A1 (en) | 2010-08-30 | 2012-03-08 | Net Power And Light, Inc. | Method and system for an interactive event experience |
US20120078788A1 (en) | 2010-09-28 | 2012-03-29 | Ebay Inc. | Transactions by flicking |
US9405444B2 (en) | 2010-10-01 | 2016-08-02 | Z124 | User interface with independent drawer control |
US9124757B2 (en) | 2010-10-04 | 2015-09-01 | Blue Jeans Networks, Inc. | Systems and methods for error resilient scheme for low latency H.264 video coding |
US10303357B2 (en) | 2010-11-19 | 2019-05-28 | TIVO SOLUTIONS lNC. | Flick to send or display content |
US9544543B2 (en) | 2011-02-11 | 2017-01-10 | Tangome, Inc. | Augmenting a video conference |
US8665307B2 (en) | 2011-02-11 | 2014-03-04 | Tangome, Inc. | Augmenting a video conference |
-
2011
- 2011-08-12 US US13/136,870 patent/US9172979B2/en not_active Expired - Fee Related
-
2012
- 2012-01-31 US US13/363,187 patent/US20120134409A1/en not_active Abandoned
-
2016
- 2016-03-31 US US15/087,657 patent/US20160219279A1/en not_active Abandoned
Patent Citations (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020051493A1 (en) * | 2000-06-26 | 2002-05-02 | Shin Jae-Seob | Moving picture mailing system and method |
US20040128350A1 (en) * | 2002-03-25 | 2004-07-01 | Lou Topfl | Methods and systems for real-time virtual conferencing |
US20090118017A1 (en) * | 2002-12-10 | 2009-05-07 | Onlive, Inc. | Hosting and broadcasting virtual events using streaming interactive video |
US20050007965A1 (en) * | 2003-05-24 | 2005-01-13 | Hagen David A. | Conferencing system |
US20050100100A1 (en) * | 2003-11-12 | 2005-05-12 | Sony Corporation | Apparatus and method for use in providing dynamic bit rate encoding |
US20060183547A1 (en) * | 2005-02-11 | 2006-08-17 | Mcmonigle Mace | Fantasy sports television programming systems and methods |
US7516255B1 (en) * | 2005-03-30 | 2009-04-07 | Teradici Corporation | Method and apparatus for providing a low-latency connection between a data processor and a remote graphical user interface over a network |
US20090115843A1 (en) * | 2005-04-04 | 2009-05-07 | Gary Richard Sohmers | Interactive Television System and Method |
WO2007084994A2 (en) * | 2006-01-20 | 2007-07-26 | Qualcomm Incorporated | Method and apparatus for determining an encoding method based on a distortion value related to error concealment |
US20080039205A1 (en) * | 2006-08-11 | 2008-02-14 | Jonathan Ackley | Method and/or system for mobile interactive gaming |
US20080195744A1 (en) * | 2007-02-14 | 2008-08-14 | Microsoft Corporation | Adaptive media playback |
US20100149096A1 (en) * | 2008-12-17 | 2010-06-17 | Migos Charles J | Network management using interaction with display surface |
US8171154B2 (en) * | 2009-09-29 | 2012-05-01 | Net Power And Light, Inc. | Method and system for low-latency transfer protocol |
US20110116540A1 (en) * | 2009-11-18 | 2011-05-19 | General Instrument Corporation | Multimedia Content Handling in a Home-Network System |
US20110246908A1 (en) * | 2010-04-01 | 2011-10-06 | Microsoft Corporation | Interactive and shared viewing experience |
US8903740B2 (en) * | 2010-08-12 | 2014-12-02 | Net Power And Light, Inc. | System architecture and methods for composing and directing participant experiences |
US20120151541A1 (en) * | 2010-10-21 | 2012-06-14 | Stanislav Vonog | System architecture and method for composing and directing participant experiences |
US8789121B2 (en) * | 2010-10-21 | 2014-07-22 | Net Power And Light, Inc. | System architecture and method for composing and directing participant experiences |
US20130019184A1 (en) * | 2011-07-11 | 2013-01-17 | Net Power And Light, Inc. | Methods and systems for virtual experiences |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9172979B2 (en) | 2010-08-12 | 2015-10-27 | Net Power And Light, Inc. | Experience or “sentio” codecs, and methods and systems for improving QoE and encoding based on QoE experiences |
US9557817B2 (en) | 2010-08-13 | 2017-01-31 | Wickr Inc. | Recognizing gesture inputs using distributed processing of sensor data from multiple sensors |
Also Published As
Publication number | Publication date |
---|---|
US20120039382A1 (en) | 2012-02-16 |
US20160219279A1 (en) | 2016-07-28 |
US9172979B2 (en) | 2015-10-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20160219279A1 (en) | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES | |
US8571956B2 (en) | System architecture and methods for composing and directing participant experiences | |
US8429704B2 (en) | System architecture and method for composing and directing participant experiences | |
US8549167B2 (en) | Just-in-time transcoding of application content | |
US20120127183A1 (en) | Distribution Processing Pipeline and Distributed Layered Application Processing | |
CN103493479B (en) | The system and method for the low latency H.264 anti-error code of Video coding | |
US9571534B2 (en) | Virtual meeting video sharing | |
US9407867B2 (en) | Distributed recording or streaming of a videoconference in multiple formats | |
US20120192087A1 (en) | Method and system for a virtual playdate | |
WO2012021174A2 (en) | EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES | |
US11481983B2 (en) | Time shifting extended reality media | |
WO2012105318A1 (en) | Input support device, input support method, and recording medium | |
US11457176B2 (en) | System and method for providing and interacting with coordinated presentations | |
US8860720B1 (en) | System and method for delivering graphics over network | |
Lugmayr et al. | E= MC2+ 1: a fully digital, collaborative, high-definition (HD) production from scene to screen | |
Stathis | Media Player Optimization for Streaming | |
Koentjoro | Exploring the ability of Virtual Set Systems in helping Small Broadcasters entering the Digital Television scene: Addressing the Technical, Licensing, and Operational Challenges of Fatwa TV | |
Borsum et al. | FascinatE D5. 1.1 AV renderer specification and basic characterisation of audience interaction | |
Zorrilla et al. | Next Generation Multimedia on Mobile Devices | |
Gaarder | Video streaming into virtual worlds | |
Saastamoinen | Challenges in Mobile Video Production | |
Macq et al. | FascinatE D1. 1.1 End user, production and hardware and networking requirements |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ALSOP LOUIE CAPITAL, L.P., CALIFORNIA Free format text: SECURITY AGREEMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:031868/0927 Effective date: 20131223 Owner name: SINGTEL INNOV8 PTE. LTD., SINGAPORE Free format text: SECURITY AGREEMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:031868/0927 Effective date: 20131223 |
|
AS | Assignment |
Owner name: NET POWER AND LIGHT, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNORS:ALSOP LOUIE CAPITAL, L.P.;SINGTEL INNOV8 PTE. LTD.;REEL/FRAME:032158/0112 Effective date: 20140131 |
|
AS | Assignment |
Owner name: PENINSULA TECHNOLOGY VENTURES, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:033086/0001 Effective date: 20140603 Owner name: ALSOP LOUIE CAPITAL I, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:033086/0001 Effective date: 20140603 Owner name: PENINSULA VENTURE PRINCIPALS, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:033086/0001 Effective date: 20140603 |
|
AS | Assignment |
Owner name: TWB INVESTMENT PARTNERSHIP II, LP, WASHINGTON Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: BROWN, JOHN SEELY, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: SINGTEL INNOV8 PTE. LTD., SINGAPORE Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: ORRICK INVESTMENTS 2011, LLC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: WANG, TA-HUI TY, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: SHIN, JEANNIE, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIO Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: PENINSULA TECHNOLOGY VENTURES, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: ALSOP LOUIE CAPITAL 1, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: ORRICK INVESTMENTS 2010, LLC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: PENINSULA VENTURE PRINCIPALS, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 Owner name: LOW, LAWRENCE B., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:036740/0252 Effective date: 20150930 |
|
AS | Assignment |
Owner name: PENSULA VENTURE PRINCIPALS, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: LOW, LAWRENCE B., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: WANG, TA-HUITY, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: SINGTEL INNOVS PTE. LTD., SINGAPORE Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: BROWN, JOHN SEELY, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: PENINSULA TECHNOLOGY VENTURES, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: THE BOARD OF TRUSTEES OF THE LELAND STANFORD UNIVE Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: ORRICK INVESTMENTS 2010, LLC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: ALSOP LOUIE CAPITAL 1, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: ORRICK INVESTMENTS 2011, LLC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: TWB INVESTMENT PARTNERSHIP II, LP, WASHINGTON Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 Owner name: SHIN, JEANNIE, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037333/0680 Effective date: 20151218 |
|
AS | Assignment |
Owner name: PENINSULA VENTURE PRINCIPALS, L.P., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: ORRICK INVESTMENTS 2010, LLC, CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: ALSOP LOUIE CAPITAL 1, L.P., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: TWB INVESTMENT PARTNERSHIP II, LP, WASHINGTON Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: BROWN, JOHN SEELY, CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: SINGTEL INNOV8 PTE. LTD., SINGAPORE Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: PENINSULA TECHNOLOGY VENTURES, L.P., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: ORRICK INVESTMENTS 2011, LLC, CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: SHIN, JEANNIE, CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: LOW, LAWRENCE B., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIO Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 Owner name: WANG, TA-HUITY, CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 037333 FRAME: 0680. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037394/0150 Effective date: 20151218 |
|
AS | Assignment |
Owner name: ORRICK INVESTMENTS 2011, LLC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: PENINSULA TECHNOLOGY VENTURES, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: ORRICK INVESTMENTS 2010, LLC, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIO Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: LOW, LAWRENCE B., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: SINGTEL INNOV8 PTE. LTD., SINGAPORE Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: ALSOP LOUIE CAPITAL 1, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: WANG, TA-HUI TY, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: SHINE, JEANNIE, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: BROWN, JOHN SEELY, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: PENINSULA VENTURE PRINCIPALS, L.P., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 Owner name: TWB INVESTMENT PARTNERSHIP II, LP, WASHINGTON Free format text: SECURITY INTEREST;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:037786/0090 Effective date: 20160219 |
|
AS | Assignment |
Owner name: NET POWER & LIGHT, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:038543/0831 Effective date: 20160427 Owner name: NET POWER & LIGHT, INC., CALIFORNIA Free format text: NOTE AND WARRANT CONVERSION AGREEMENT;ASSIGNORS:PENINSULA TECHNOLOGY VENTURES, L.P.;PENINSULA VENTURE PRINCIPALS, L.P.;ALSOP LOUIE CAPITAL 1, L.P.;REEL/FRAME:038543/0839 Effective date: 20160427 |
|
AS | Assignment |
Owner name: NET POWER & LIGHT, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:038398/0958 Effective date: 20160427 Owner name: NET POWER & LIGHT, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:NET POWER & LIGHT, INC.;REEL/FRAME:038398/0946 Effective date: 20160427 Owner name: NET POWER & LIGHT, INC., CALIFORNIA Free format text: NOTE AND WARRANT CONVERSION AGREEMENT;ASSIGNORS:PENINSULA TECHNOLOGY VENTURES, L.P.;PENINSULA VENTURE PRINCIPALS, L.P.;ALSOP LOUIE CAPITAL 1, L.P.;AND OTHERS;REEL/FRAME:038543/0966 Effective date: 20160427 Owner name: NET POWER & LIGHT, INC., CALIFORNIA Free format text: NOTE AND WARRANT CONVERSION AGREEMENT;ASSIGNORS:PENINSULA TECHNOLOGY VENTURES, L.P.;PENINSULA VENTURE PRINCIPALS, L.P.;ALSOP LOUIE CAPITAL 1, L.P.;AND OTHERS;REEL/FRAME:038543/0942 Effective date: 20160427 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: NET POWER AND LIGHT, INC., CALIFORNIA Free format text: MERGER;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:039809/0346 Effective date: 20160427 |
|
AS | Assignment |
Owner name: WICKR INC., ILLINOIS Free format text: MERGER;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:039810/0277 Effective date: 20160721 |
|
AS | Assignment |
Owner name: WICKR INC., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE ADDRESS PREVIOUSLY RECORDED AT REEL: 039810 FRAME: 0277. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:040262/0085 Effective date: 20160721 Owner name: WICKR INC., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ADDRESS FROM 71 S. WACKER DR. SUITE 2385 CHICAGO, ILLINOIS 60606 TO 459 18TH ST., #313, SAN FRANCISCO, CALIFORNIA 94107 PREVIOUSLY RECORDED ON REEL 039810 FRAME 0277. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER;ASSIGNOR:NET POWER AND LIGHT, INC.;REEL/FRAME:040267/0459 Effective date: 20160721 |