US 20080250459 A1
A handheld, wireless video receiver that receives compressed video, decompresses the compressed video, displays the decompressed video. The receiver is part of a system comprising methods, medium, and handheld, wireless devices which compress, enhance, encode, transmit, decompress and display digital video images in real time. Real time wireless videoconferences connect multiple handheld video devices. Real time compression is achieved by sub-sampling each frame of a video signal, filtering the pixel values, and encoding. Real time transmission is achieved due to high levels of effective compression. Real time decompression is achieved by decoding and decompressing the encoded data to display high quality images. The receiver can alter various setting including but not limited to the format for the compression, image size, frame rate, brightness and contrast. A zoom control can be used select a portion of interest of video being transmitted or being played back. The receiver may also have a touch sensitive display screen providing controls for video display and mobile telephone operation.
51. A handheld wireless video receiver, in a system for transmission and display of video and other media, the system comprising a wireless network, a media exchange, and a plurality of devices, each connected to the wireless network, wherein at least one device is capable of transferring a plurality of compressed video frames which make up a video stream, said handheld video receiver comprising:
a) a color display having a predetermined display width and display height and having a touch screen for receiving input from a user, wherein the display width is a predetermined first number of pixels and the display height is a predetermined second number of pixels,
b) an audio output,
c) a program memory, for storing computer programs,
d) a processor, for executing computer programs from the program memory,
e) a data memory, for storing data,
f) a user interface on the touch screen comprising controls for interfacing with the executing computer programs,
g) a wireless network interface for connecting to the wireless network via wireless a communications channel, wherein the handheld video receiver maintains connection to the wireless network when being moved from one location to another location or while being freely carried by the user,
h) a decompressor, cooperating with the processor,
i) a video camera for capturing images in real-time,
j) a microphone for capturing an audio portion in real-time, and
k) a compressor configured to compress captured images,
wherein the handheld video receiver has a weight of less than about six ounces,
wherein the handheld device has physical dimension allowing it to be held substantially in one hand of the user,
wherein the user interface is operated by one or more fingers of the user,
wherein each of the communications channels from the wireless network to the handheld video receiver has a predetermined bandwidth such that the maximum sustainable bandwidth measured in millions of bits per second,
wherein the compressed video frames are received over one of the communications channel,
wherein the decompressor decompresses the compressed video frames,
wherein the compressed video frames is stored in a video file,
wherein the data memory is configured to store a plurality of video files,
wherein at least a portion of the decompressed video frames is displayed on the color display in real time,
wherein the audio portion plays on the audio output,
wherein the handheld wireless video receiver receives one or more video files from the media exchange after the user agrees to pay a license fee determined by interaction of the user with the media exchange over the wireless network,
whereby user enters commands via the touch screen to play back any of the stored video files, and
whereby the video is displayed while the user carries the handheld wireless video receiver.
52. The handheld wireless video receiver of
i) a zoom interface for issuing zoom in, zoom out, and pan commands, wherein the magnification and position of the portion of the displayed media are changed in response to one or more of the zoom in, zoom out, and pan commands, and
ii) mobile telephone controls for entering numeric telephone numbers and for controlling mobile telephone calls.
53. The handheld wireless video receiver of
54. The handheld wireless video receiver of
55. The handheld wireless video receiver of
56. The handheld wireless video receiver of
wherein the data memory of the handheld wireless video receiver stores graphical user programs such as games,
wherein processor further executes the user programs from the data memory.
57. The handheld wireless video receiver of
wherein the display height is greater than or equal to 480 pixels.
58. The handheld wireless video receiver of
wherein the display width is greater than or equal to 320 pixels.
59. The handheld wireless video receiver of
wherein the display width is less than or equal to 640 pixels.
60. The handheld wireless video receiver of
wherein the handheld video receiver has a weight of less than about four ounces.
61. The handheld wireless video receiver of
wherein the magnification and position of the portion of the displayed media are changed in response to one or more of the zoom in, zoom out, and pan commands.
62. The handheld wireless video receiver of
wherein the controls further comprises commands relating to video quality and transmission,
wherein the handheld video receiver dynamically interprets control commands entered by the user via the touch screen,
whereby the user of the receiver views the optimum quality live video.
63. The handheld wireless video receiver of
wherein the data memory comprises a hard disk drive,
wherein the video file is stored on the hard disk drive, and
wherein the video file stored on the hard disk drive is played back.
64. The handheld wireless video receiver of
wherein the data memory comprises a non-volatile memory,
wherein the video file is stored on the non-volatile memory, and
wherein the video file stored on the non-volatile memory is played back.
65. The handheld wireless video receiver of
wherein the video frames are compressed using a color lossless compression method.
66. The handheld wireless video receiver of
wherein the processor is less powerful than the processors typically found in non-handheld devices,
wherein the processor and decompressor cooperate to decompress each of the video frames in a single pass,
whereby the processor speed requirements are reduced and the battery life is extended while maintaining high quality video with optimal effective compression ratios.
67. The handheld wireless video receiver of
wherein the wireless network is connected to the Internet, and
wherein the computer programs further comprise programs for receiving and displaying still graphic images and text downloaded from the network.
68. A method of exchanging digital video files between a media exchange and a handheld wireless video receiver, comprising the steps of:
a) capturing real-time, full-frame, full-color video from a video input device,
b) digitizing the captured video,
c) compressing the digital video with a compressor configured to preserve the level of spatial quality and motion quality of the compressed video that the handheld video receiver is capable of displaying,
d) storing the compressed video in a data memory in a device in communication with a wireless network,
e) transferring the compressed video to the media exchange via the network,
f) downloading the compressed video from the media exchange via the network to the handheld wireless video receiver after a user of the handheld wireless video receiver agrees to pay a fee for use of the downloaded video,
g) storing the downloaded video in a data memory of handheld wireless video receiver,
h) decompressing the downloaded video to a format that is optimal for display on a touch screen display on the handheld wireless video receiver,
i) displaying at least a portion of the decompressed video on the display,
j) playing an audio portion of the decompressed video on an audio output of the handheld wireless video receiver,
whereby the user downloads and plays back video from the media exchange.
69. A handheld wireless video receiver, comprising:
a) a computer-based circuit including a processor, a memory for storing computer programs and data, and a decompressor for decompressing compressed video frames,
b) a color display coupled to the circuit,
c) a user interface coupled to the circuit and configured to provide user interaction with computer programs operating in the processor thereby providing user control to one or more programmed functions, and
d) a wireless network interface coupled to the circuit for connecting to a wireless network via a communications channel, for receiving a stream of compressed video frames from the network,
wherein the compressed video frames are stored into memory, and the user interface controls playback by having the circuit retrieve the compressed video frames from memory, send them to the decompressor to thereby decompress the video frames, and display the decompressed video frames on the color display.
70. The handheld wireless video receiver of
a) a video camera coupled to the circuit for capturing video frames in real-time, and
b) a compressor included with the circuit for compressing the captured video frames,
wherein the compressed video frames are stored into memory, and the user interface controls playback by having the circuit retrieve the compressed video frames from memory, send them to the decompressor to thereby decompress the video frames, and display the decompressed video frames on the color display.
This application is a continuation of co-pending U.S. patent application Ser. No. 11/262,106, filed on Oct. 27, 2005, published Jun. 1, 2006, as U.S. patent application publication 2006/0114987, entitled “HANDHELD VIDEO TRANSMISSION AND DISPLAY,” which hereby is incorporated by reference.
U.S. patent application Ser. No. 11/262,106 is a continuation in part of co-pending U.S. patent application Ser. No. 09/467,721, filed on Dec. 20, 1999, and entitled “VARIABLE GENERAL PURPOSE COMPRESSION FOR VIDEO IMAGES (ZLN)”, now U.S. Pat. No. 7,233,619, which hereby is incorporated by reference.
This application and application Ser. No. 09/467,721 claim priority under 35 U.S.C. § 119(e) of U.S. provisional application Ser. No. 60/113,051, filed on Dec. 21, 1998, and entitled “METHODS OF ZERO LOSS (ZL) COMPRESSION AND ENCODING OF GRAYSCALE IMAGES”, which hereby is incorporated by reference.
My U.S. patent application Ser. No. 09/312,922, filed on May 17, 1999, and entitled “SYSTEM FOR TRANSMITTING VIDEO IMAGES OVER A COMPUTER NETWORK TO A REMOTE RECEIVER,” now U.S. Pat. No. ______, is also hereby incorporated by reference.
My U.S. patent application Ser. No. 09/433,978, now U.S. Pat. No. 6,803,931, filed on Nov. 4, 1999, and entitled GRAPHICAL USER INTERFACE INCLUDING ZOOM CONTROL REPRESENTING IMAGE AND MAGNIFICATION OF DISPLAYED IMAGE”, is also hereby incorporated by reference. A co-pending divisional application of U.S. Pat. No. 6,803,931, is U.S. patent application Ser. No. 10/890,079, filed on Jul. 13, 2004, published on Dec. 9, 2004 as publication number 2004/0250216, and entitled GRAPHICAL USER INTERFACE INCLUDING ZOOM CONTROL REPRESENTING IMAGE AND MAGNIFICATION OF DISPLAYED IMAGE”, and is also hereby incorporated by reference.
My U.S. patent application Ser. No. 09/470,566, now U.S. Pat. No. 7,016,417, filed on Dec. 22, 1999, and entitled GENERAL PURPOSE COMPRESSION FOR VIDEO IMAGES (RHN)”, describes a compression method known as the “RHN” method, and is also hereby incorporated by reference.
My co-pending U.S. patent application Ser. No. 09/473,190, filed on Dec. 20, 1999, and entitled “ADDING DOPPLER ENHANCEMENT TO GRAYSCALE COMPRESSION (ZLD)” is also hereby incorporated by reference.
My co-pending U.S. patent application Ser. No. 10/154,775, filed on May 24, 2002, published as US 2003/0005428, and entitled “GLOBAL MEDIA EXCHANGE” is also hereby incorporated by reference.
U.S. patent application Ser. No. 09/436,432, filed on Nov. 8, 1999, and entitled “SYSTEM FOR TRANSMITTING VIDEO IMAGES OVER A COMPUTER NETWORK TO A REMOTE RECEIVER,” now U.S. Pat. No. 7,191,462, is wholly owned by the inventor of the present invention.
1. Field of the Invention
This invention relates to handheld devices for video transmission, including video capture, wired and wireless file transfer and live streaming, and display. Embodiments of the invention relate to data compression, specifically to the compression and decompression of video and still images, and relate to graphical user interfaces for controlling video transmission and display.
2. Description of Prior Art
In the last few years, there have been tremendous advances in the speed of computer processors and in the availability of bandwidth of worldwide computer networks such as the Internet. These advances have led to a point where businesses and households now commonly have both the computing power and network connectivity necessary to have point-to-point digital communications of audio, rich graphical images, and video. However the transmission of video signals with the full resolution and quality of television is still out of reach. In order to achieve an acceptable level of video quality, the video signal must be compressed significantly without losing either spatial or temporal quality.
A number of different approaches have been taken but each has resulted in less than acceptable results. These approaches and their disadvantages are disclosed by Mark Nelson in a book entitled The Data Compression Book, Second Edition, published by M&T Book in 1996. Mark Morrision also discusses the state of the art in a book entitled The Magic of Image Processing, published by Sams Publishing in 1993.
Standard video signals are analog in nature. In the United States, television signals contain 525 scan lines of which 480 lines are visible on most televisions. The video signal represents a continuous stream of still images, also known as frames, that are fully scanned, transmitted and displayed at a rate of 30 frames per second. This frame rate is considered full motion.
A television screen has a 4:3 aspect ratio.
When an analog video signal is digitized each of the 480 lines is sampled 640 times, and each sample is represented by a number. Each sample point is called a picture element, or pixel. A two dimensional array is created that is 640 pixels wide and 480 pixels high. This 640×480 pixel array is a still graphical image that is considered to be full frame. The human eye can perceive 16.7 thousand colors. A pixel value comprised of 24 bits can represent each perceivable color. A graphical image made up of 24-bit pixels is considered to be full color. A single, second-long, full frame, full color video requires over 220 millions bits of data.
The transmission of 640×480 pixels×24 bits per pixel times 30 frames requires the transmission of 221,184,000 million bits per second. A Ti Internet connection can transfer up to 1.54 million bits per second. A high-speed (56 Kb) modem can transfer data at a maximum rate of 56 thousand bits per second. The transfer of full motion, full frame, full color digital video over a TI Internet connection, or 56 Kb modem, will require an effective data compression of over 144:1, or 3949:1, respectively.
A video signal typically will contain some signal noise. In the case where the image is generated based on sampled data, such as an ultrasound machine, there is often noise and artificial spikes in the signal. A video signal recorded on magnetic tape may have fluctuations due the irregularities in the recording media. Florescent or improper lighting may cause a solid background to flicker or appear grainy. Such noise exists in the real world but may reduce the quality of the perceived image and lower the compression ratio that could be achieved by conventional methods.
An early technique for data compression is run-length encoding where a repeated series of items are replaced with one sample item and a count for the number of times the sample repeats. Prior art shows run-length encoding of both individual bits and bytes. These simple approaches by themselves have failed to achieve the necessary compression ratios.
In the late 1940s, Claude Shannon at Bell Labs and R.M. Fano at MIT pioneered the field of data compression. Their work resulted in a technique of using variable length codes where codes with low probabilities have more bits, and codes with higher probabilities have fewer bits. This approach requires multiple passes through the data to determine code probability and then to encode the data. This approach also has failed to achieve the necessary compression ratios.
D. A. Huffman disclosed a more efficient approach of variable length encoding known as Huffman coding in a paper entitled “A Method for Construction of Minimum Redundancy Codes,” published in 1952. This approach also has failed to achieve the necessary compression ratios.
In the 1980s, arithmetic, finite coding, and adaptive coding have provided a slight improvement over the earlier methods. These approaches require extensive computer processing and have failed to achieve the necessary compression ratios.
Dictionary-based compression uses a completely different method to compress data. Variable length strings of symbols are encoded as single tokens. The tokens form an index to a dictionary. In 1977, Abraham Lempel and Jacob Ziv published a paper entitled, “A Universal Algorithm for Sequential Data Compression” in IEEE Transactions on Information Theory, which disclosed a compression technique commonly known as LZ77. The same authors published a 1978 sequel entitled, “Compression of Individual Sequences via Variable-Rate Coding,” which disclosed a compression technique commonly known as LZ78 (see U.S. Pat. No. 4,464,650). Terry Welch published an article entitled, “A Technique for High-Performance Data Compression,” in the June 1984 issue of IEEE Computer, which disclosed an algorithm commonly known as LZW, which is the basis for the GIF algorithm (see U.S. Pat. Nos. 4,558,302, 4,814,746, and 4,876,541). In 1989, Stack Electronics implemented a LZ77 based method called QIC-122 (see U.S. Pat. No. 5,532,694, U.S. Pat. No. 5,506,580, and U.S. Pat. No. 5,463,390).
These lossless (method where no data is lost) compression methods can achieve up to 10:1 compression ratios on graphic images typical of a video image. While these dictionary-based algorithms are popular, these approaches require extensive computer processing and have failed to achieve the necessary compression ratios.
Graphical images have an advantage over conventional computer data files: they can be slightly modified during the compression/decompression cycle without affecting the perceived quality on the part of the viewer. By allowing some loss of data, compression ratios of 25:1 have been achieved without major degradation of the perceived image. The Joint Photographic Experts Group (JPEG) has developed a standard for graphical image compression. The JPEG lossy (method where some data is lost) compression algorithm first divides the color image into three color planes and divides each plane into 8 by 8 blocks, and then the algorithm operates in three successive stages:
JPEG can be scaled to perform higher compression ratio by allowing more loss in the quantization stage of the compression. However this loss results in certain blocks of the image being compressed such that areas of the image have a blocky appearance and the edges of the 8 by 8 blocks become apparent because they no longer match the colors of their adjacent blocks. Another disadvantage of JPEG is smearing. The true edges in an image get blurred due to the lossy compression method.
The Moving Pictures Expert Group (MPEG) uses a combination of JPEG based techniques combined with forward and reverse temporal differencing. MPEG compares adjacent frames and, for those blocks that are identical to those in a previous or subsequent frame, only a description of the previous or subsequent identical block is encoded. MPEG suffers from the same blocking and smearing problems as JPEG.
These approaches require extensive computer processing and have failed to achieve the necessary compression ratios without unacceptable loss of image quality and artificially induced distortion.
Apple Computer, Inc. released a component architecture for digital video compression and decompression, named QuickTime. Any number of methods can be encoded into a QuickTime compressor/decompressor (codec). Some popular codec are CinePak, Sorensen, and H.263. CinePak and Sorensen both require extensive computer processing to prepare a digital video sequence for playback in real time; neither can be used for live compression. H.263 compresses in real time but does so by sacrificing image quality resulting in severe blocking and smearing.
Extremely high compression ratios are achievable with fractal and wavelet compression algorithms. These approaches require extensive computer processing and generally cannot be completed in real time.
Sub-sampling is the selection of a subset of data from a larger set of data. For example, when every other pixel of every other row of a video image is selected, the resulting image has half the width and half the height. This is image sub-sampling. Other types of sub-sampling include frame sub-sampling, area sub-sampling, and bit-wise sub-sampling.
If an image is to be enlarged but maintain the same number of pixels per inch, data must be filled in for the new pixels that are added. Various methods of stretching an image and filling in the new pixels to maintain image consistency are known in the art. Some methods known in the art are dithering (using adjacent colors that appear to be blended color), and error diffusion, “nearest neighbor”, bilinear and bicubic.
In the early 1990s, a number of pen based computers were developed. These portable computers were characterized by a display screen that could be also used as an input device when touched or stroked with a pen or finger. For example in 1991, NCR developed a “notepad” computer, the NCR 3125. Early pen-based computers ran three operating systems: DOS, Microsoft's Windows for Pen Computing and Go Corp.'s PenPoint. In 1993, Apple developed the Newton MessagePad, an early personal digital assistant (PDA). Palm developed the Palm Pilot in 1996. Later, in 2002, Handspring released the Treo which runs the Palm OS and features a Qwerty keyboard. In 2000, the Sony Clie, used the Palm OS and could play audio files. Later versions included a built-in camera and could capture and play Apple QuickTime™ video. Compaq (now Hewlett Packard) developed the iPAQ in 2000. The iPAQ and other PocketPCs run a version of Windows CE. Some PocketPC and PDA have wireless communication capabilities.
In 2001, Apple released a music player, called the iPod, featuring a small, internal hard disk drive that could hold over 1000 songs and fit in your pocket. The original iPod has a display, a set of controls, and ports for connecting to a computer, such as a Macintosh or PC, via Firewire, and for connecting to headphones. However, the original iPod did not have a color display, a built-in camera, built-in speakers, built-in microphone or wireless communications.
The first cellular telephones had simple LCD displays suitable for displaying only a limited amount of text. More recently, cell phones have been developed which have larger, higher resolution displays that are both grayscale and color. Some cell phones have been equipped with built-in cameras with the ability to save JPEG still photos to internal memory. In April 2002, Samsung introduced a cell phone with a built-in still photo camera and a color display. The Samsung SCH-X590 can store up to 100 photos in its memory and can transfer still photos wirelessly.
Cell phones can be used as wireless modems. Initially they had limited data bandwidth. Next, digital cell phones were developed. By early 2002, bandwidth was typically 60-70 Kbps. Higher bandwidth wireless networks are being developed.
Hand held devices are limited in size and weight. Many users are only willing to use a handheld device that weights a few ounces and can fit inside a typical shirt pocket, or even worn on their waist or arm. These size and weight limitation prevent handheld devices from having the electronic circuitry, processors, and batteries found in laptops and other larger computers. These limitations have made it impossible to provide full frame, full motion video display or live transmission on handheld devices.
The existing, commercially available hand held devices have not been able to support live or streaming video for a number of reasons. Uncompressed full-motion, full frame video requires extremely high bandwidth that is not available to handheld portable devices. In order to reduce the bandwidth, lossy compression such as MPEG has been used to reduce the size of the video stream. While MPEG is effective in desktop computers with broadband connections to the Internet, decoding and displaying MPEG encoded video is very processor intensive. The processors of existing handheld devices are slower or less powerful than those used in desktop computers. If MPEG were used in a handheld device, the processor would quickly drains the battery of most handheld devices. Further, the higher bandwidth wireless communications interfaces would also place a large strain on the already strained batteries. Live video transmission and reception would be even more challenging. For this reason, handheld device have not been able to transmit or receive streaming, or especially, live video.
What is needed is an enhanced handheld device that is capable of receiving streaming and live video. Further a handheld device that could capture and transmit live video would provide live coverage of events that would otherwise not be able to be seen. With handheld video devices that both transmit and receive live video, handheld wireless videoconferencing could become a reality. Also a video compression method that requires significantly reduced processing power and would be less draining on the battery of a handheld device is needed. Additionally since, handheld video display screens which are smaller than typical computer screens, a user of a handheld video receiver needs to be able control the portion of a video be transmitted to allow a smaller, higher quality video to be received and viewed on the handheld screen with dimensions smaller than the original video.
In accordance with the present invention a handheld device comprises a black and white or color video display screen, speakers or headphones for hearing audio associated with the video display, controls for user input, a memory for storing compressed video data, and a processor for running computer programs which decompress the compressed video data and play the video on the display screen, and the video's audio on speakers and/or headphones. Further, some embodiments of the present invention include a microphone and video camera for inputting audio and video. A plurality of handheld video devices are connected to a network for exchanging video file, streaming video from a pre-recorded video file or live transmission from one device to one or more devices in remote locations. The network connections can be wired or wireless.
One embodiment of the present invention comprises a video camera that can be removably mounted on an iPod-type device to add the video capture capability. Further the separate camera unit could include a microphone or speakers. Further, wireless communications could be added to the separate camera unit or as yet another removable unit.
Further, the present invention includes a method of compression of a video stream comprising steps of sub-sampling a video frame, and run-length encoding the sub-sampled pixel values, whereby the method can be executed in real time, and whereby the compressed representation of pixels saves substantial space on a storage medium and requires substantially less time and bandwidth to be transported over a communications link. The present invention includes a corresponding method for decompressing the encoded data.
Further, the present invention includes a zoom control that is graphically displayed on the display screen and receives input from either the touch screen or the controls of the handheld device. A user may use the zoom control to send remote control commands to a transmitting device to dynamically specify an area to be transmitted. Alternatively, the user may use the zoom control to magnify video that is being played from a file.
Accordingly, beside the objects and advantages of the method described above, some additional objects and advantages of the present invention are:
In the drawings, closely related figures have the same number but different alphabetic suffixes.
Video digitizing hardware typical has the options of storing the pixel values as a 32 bit pixel value 200 or a 24 bit pixel value 210, shown in
The 24-bit pixel value 210 is composed of a blue component 212, a green component 214, and a red component 216. There is no component for the alpha channel in the 24 bit pixel value 210. Regardless of the structure, the blue channel 202 is equivalent to the blue component 212, the green channel 204 is equivalent to the green component 214, and the red channel 206 is equivalent to the red component 216.
In the present invention, the 32 bit pixel value 200 alternative is preferred due to the consistent alignment of 32 bit values in most computer memories; however for simplicity of illustration the alpha channel 208 will be omitted in
If the video signal is digitized in color, the three color components may have different values. For example in
If the video signal being digitized is grayscale, the three color components will have the same values. For example in
The preferred embodiment of this invention uses the low order byte of the pixel value, which is typically the blue component as shown in
For additional compression, the filtered pixel value 299 can variably select any number of bits. For example, selection of the most significant four bits instead of all eight bits filters noise that may show up in the low order bits may be very suitable for an image such as one produced by an ultrasound medical device. An example of this is shown by ZL4 804 in
Speed of compression and decompression may be enhanced if the algorithms fit into computer memory native storage elements such as 8 bit bytes, 16 bit words, or 32 bit double words, or some other size for which the computer architecture is optimized.
A grayscale image may be stored at a higher bit level than the actual values require. This may occur when an image is generated by an imaging technology such as radar, ultrasound, x-ray, magnetic resonance, or similar electronic technology. For example an ultrasound machine may only produce 16 levels of grayscale, requiring 4 bits of data per pixel, but the image digitizing may be performed at 8 to 12 bits per pixel. In this example, the low order bits (4 to 8) respectively provide no significant image data.
In the present invention, a fast and efficient compression and encoding method is implemented by using unused bits to store a repeat count for repeated values.
The most significant N bits of the pixel value are selected where N is the number of significant bits (determined by data analysis or by user selection). If N is less than W, where W is a native machine data type such as 8 bit byte, 16 bit word, or 32 bit double word or some other size for which the computer architecture is optimized, then W-N equals the number of unneeded bits, U. A repeat count, C, can contain a value from 1 to CMAX where CMA is 2 to the power of U. For example, if U equals 4, C can be a number from 1 to 16. In practice the maximum value will be encoded as a zero because the high order bit is truncated. In the example, decimal 16 has a binary value “10000” will be stored as “0000”.
For example, when W is 8, value pairs for N and U could include without limitation (2,6), (3,5), (4,4), (5,3), and (6,2). When W is 16, value pairs for N and U could include without limitation (2, 14), (3, 13), (4, 12), (5, 11), (6, 10), (7, 9), (8, 8), (9, 7), (10, 6), (11, 5), (12, 4), (13, 3), and (14, 2). When W is 32, value pairs for N and U could include without limitation all combinations of values pairs for N and U where N+U equals 32 and N>1 and U>1. When W is not a multiple of 8, value pairs for N and U could include without limitation all combinations of values pairs for N and U where N+U equals W and N>1 and U>1.
The most significant N bits of each pixel are selected from the image to obtain value V.
In the encryption embodiment of this invention V may be used to select an encoded value, E, from the encoding table. E is also a N-bit value. The number of elements in the encode table 1100 (
In the other embodiments of this invention V is used as E.
E is saved as the prior value, P. For each subsequent pixel, the encoded value, E, is obtained and compared to the prior value, P. If the prior value, P, is the same as E, then a repeat counter, C, is incremented; otherwise the accumulated repeat count, C, for the prior value, P, is merged with P and placed in an array A that implements the encoded data 140 (
The encoding begins at an encode entry 402. In an encode initialization step 403, a prior value P is set to a known value, preferably decimal “255” or hexadecimal 0×FF, a repeat counter C is set to zero, an encoded length L is set to 0, and a completion flag “Done” is set to a logical value of false. Next, a get pixel step 404 obtains a pixel from the image being encoded. At a get value step 405, a value V is set to the N bit filtered pixel value 299 as derived from the pixel using one of the methods shown in
If the encode value E does not match the prior value P, then a check count overflow 412 decision is made. If the counter C is less than or equal to CMAX, then a new code step 414 is executed, otherwise a counter overflow step 420 is executed.
At step 414, the counter C is masked and bit-wise OR-ed with P shifted left by U bit positions and is placed in the A at the next available location as indexed by the encoded length L. Then, continuing inside flowchart step 414, L is incremented, the repeat count C is set to 1 and the prior value P is set to E. After step 414, a “check end of data” decision is made by checking to see if there are any more pixels in the image, and, if not, if the last value has been processed. Because this method utilizes a read ahead technique step 414 must be executed one more time after the end of data is reached to process the last run-length. If there is more data in the image, flow continues to a check of the completion flag “Done” at step 422. If the check indicates that the process is not completed, flow continues to step 404.
If the end of data is reached but the completion flag “Done” is still false, flow continues to a set done step 418. At step 418, the completion flag “Done” is set to logical true, and flow continues to decision 412 where the last run-length will be output and flow will eventually exit through step 414, decision 416, decision 422, and then terminate at encode exit 428.
It is possible for the repeat count C to become larger than CMAX requiring more bits than allocated by this method. This situation is handled by making the check count overflow 412 decision and executing the counter overflow step 420. At step 420, the counter C is masked and bit-wise OR-ed with P shifted left by U bit positions and is placed in the A at the next available location as indexed by the encoded length L. Then, continuing inside flowchart step 414, L is incremented, and the repeat count C is decrement by CMAX. After step 420, flow continues to the check count overflow 412 decision. Thus when the encode value E repeats more than CMAX times, multiple sets of repeat counts and encoded values are output to the encoded data 140 buffer.
This entire process is repeated for each image or video frame selected during optional image sub-sampling (see 110 in
Because the video signal being digitized is analog there will be some loss of information in the analog to digital conversion. The video digitizing hardware can be configured to sample the analog data into the image 430 with almost any width 440 and any height 450. The present invention achieves most of its effective compression by sub-sampling the data image with the width 440 value less than the conventional 640 and the height 450 value less than the convention 480. In a preferred embodiment of the invention, for use in a medical application with TI Internet transmission bandwidth, image dimensions are sub-sampled at 320 by 240. However an image dimension sub-sampling resolution of 80 by 60 may be suitable for some video application.
In addition, the present invention provides for a larger count when the bit filtering is larger. For example, the alternate ZLN format where each byte contains 4 data bits, ZL4 (where N is 4 and U is 4), allows for a four bits of repeat count. For example, in practice, ZL4 is superior to RHN on a typical ultrasound image containing 16 shades of gray.
The embodiment of the present invention shown in
The ZLN method of the present invention provides for variable formats. The values of N 300, U 301, and W 302 can be dynamically changed between frames. For ease of communication a format is named with the prefix “ZL” and a digit representing the value of N. For example, “ZL5” refers to a format where bit width of N is equal to 5. There are multiple values of U depending of the W. To also specify the bit width of U a hyphen and a number can be appended. For example, “ZL5-13” represents a format where N=5 and U=13. “ZL5-3” is a common format and may be imprecisely referred to as “ZL5.”
To decode the compressed array, the decoder has a decode table that corresponds with the encode table. For W*4 bit color pixels, the decode table contains the appropriate alpha, red, green, and blue values. For W*3 bit color pixels, the alpha value is not used. The compressed array is processed W bits at a time as X. The repeat count, C, is extracted from X by masking off the data value (C=X&(((2**N)−1)<<U)). The encoded value, E, is extracted from X by masking off the count (E=X&((2**U)−1)). The encoded value, E maybe used to index into the decryption. The decoded pixels are placed in a reconstructed image and repeated C times. Each element of the compressed array, A, is processed until its entire length, L, has been processed.
The decoding begins at a decode entry 900. In a “decode initialization” step 901, a repeat counter C is set to one, an encoded length L is set to the value obtained with the encoded data 140 (
In this illustrative decryption embodiment of the present invention, flow goes to a “decode lookup” step 908 where the value of E is used to index into the decode table 1110 (
The 909 decision always fails the first time ensuring that a place pixel step 910 is executed. The place pixel step 910 places the pixel value V in the next location of the decompressed image and decrements the repeat counter C and returns to the 909 decision. The pixel value V is placed repeatedly until C decrements to zero. Then the 909 decision branches flow to a “reset counter” step 914. At step 914 the repeat counter is reset to 1.
Flow continues to the “check length” 916 decision where the index I is compared to the encoded length L to determine if there are more codes to be processed. If I is less than L flow returns to step 902, otherwise the decode process terminates at a “decode exit” 918.
The entire decode process is repeated for each encoded frame image.
Pixels 1052, 1054, 1056, 1058 and 1060 are inserted due to the enlargement of the image. Their values are calculated by averaging the values of the two pixels above and below or to the left or the right of the new pixel. A preferred sequence is calculation of:
1. 1052 between 1010 and 1012
2. 1054 between 1010 and 1014
3. 1058 between 1012 and 1016
4. 1056 between 1054 and 1058
Pixel 1060 can be calculated on the interpolation for the subsequent row.
By using corresponding encoding and decoding tables the data can be encrypted and decrypted without using actual values. Encryption provides a level of security for the encoded data 140 while in storage or transit.
The encode table 1100 is 2 the power of N in length. If the target color image format is W*4 bit color, then the decode table 1110 has W bits for alpha, red, green, and blue each, respectively. If the target color image format is W*3 bit color, then the alpha value is not used. If the image is W bit grayscale then only the grayscale value is used to create the decompressed and decoded image.
The corresponding table elements are mapped to each other. For example, 0 could encode to 22 as long as the 22nd element of the decode table returns (θ×ff<<24|θ<<16|θ<<8|θ).
When these versions of the tables are used, the encode and decode processes and their speed of execution are substantially the same but the encoded data 140 (
FIGS. 12A through 12D—Compression and Decompression Devices
FIGS. 13A through 13J—Compressor Details, Encoding Circuit, and Bitwise Pixel Sub-Samplers
FIGS. 14A through 14C—Variable Selection of Bit-wise Sub-sampling
The settings 1660 include brightness 1661, contrast 1662, height 1663, width 1664, and frame rate 1665. The brightness 1661, contrast 1662, height 1663, and width 1664 setting alter the attributes of each frame as it is digitized in a frame sub-sampler 1620. The brightness 1661 and contrast 1662 settings alter the video digitizer 1310 (
The frame sub-sampler 1620 outputs a selected frame 1630 along path 1621. The transmitter pixel sub-sampler 1640 scans the selected frame 1630 getting each pixel from frame 1632 and outputs data values along path 1642 to a run length encoder 1650. The encoded data stream 1235 is then transmitted to the remote receiver 1610.
These embodiments illustrate the novel feature of the present invention of allowing a user at a remote receiver 1610 to control aspects of the transmitter 1600 or 1690 from a remote location, including brightness, contrast, frame dimensions, frame rate, image area, and the type of compression used.
FIGS. 19A through 19C-Handheld Video Transmission Networks
In another embodiment of the present invention, any node could act as a video server and transmit pre-recorded video to one or more other nodes.
These illustrations are exemplary. In practice, combined networks could consist of any number of nodes. Any of the nodes in the network could be a handheld video device.
FIGS. 20A through 20D—Handheld Video Devices
A first handheld device 2010 comprises a display 2012, manual controls 2014, a wireless port 2016, and a first wired connection 2051 a. While either the wireless port 2016 or the wired connection 2051 a could be present, only one of the two would be necessary to receive video from or transmit video to other nodes in the network 1910. In this example, the first handheld device is shown as an iPod-type device with an internal hard disk drive. The first handheld device 2010 further comprises a headphone 2020, connected via a speaker/microphone cable 2024, and a camera 2030, connected via a camera cable 2034. The headphone 2020 comprises a right speaker 2021, a microphone 2022, and a left speaker 2023. The camera 2030 has a lens 2032 and internal circuitry that converts the light that passes through the lens 2032 into digital video data.
In the best mode for this embodiment, the iPod-type device is implemented using a standard Apple iPod (enhanced with an audio input for the microphone and, optionally, with a wireless port, and appropriate software), and the camera 2030 is implemented using an iBot Firewire camera manufactured by Orange Micro, a lower performing Connectix USB camera, or similar camera. Alternatively, if the iPod-type device were only used of viewing video, the Apple iPod could be used without hardware modification. In another variation, the microphone could be build into the camera (not shown) instead of the headphones.
A second handheld device 2040 comprises a second display 2012 b, a second wireless port 2016 b, and a second wired connection 2051 b. While either the wireless port 2016 b or the wired connection 2051 b could be present, only one of the two would be necessary to receive video from or transmit video to other nodes in the network 1910. In this example, the second handheld device is shown as a device with a touch screen. The second handheld device 2040 further comprises a right built-in speaker 2021 b, a built-in microphone 2022 b, a left built-in speaker 2023 b, and a built-in camera 2030 b with lens 2032.
The configuration of the second handheld device 2040 has the advantage of eliminating the cables for the external headphone and camera of the first handheld device 2010 by having all elements built-in.
These two devices are exemplary. A two-device handheld videoconferencing network could have two identical handheld devices, such as the first handheld device 2010. Further, a single device with a camera (as shown) could transmit video for display on any number of hand held devices that do not have cameras or microphones.
The configuration of the integrated handheld device 2060 has the advantage of eliminating the cables for the external headphone and camera of the first handheld device 2010 by having all elements integrated into removably attached modules that form a single unit when attached. The user can configure the standard iPod based on the user's intended use. If only a wireless connection is needed, only the wireless module 2064 can be attached to the iPod; in this configuration video can be received and displayed but not transmitted. If only video transmission is necessary and a wired connection is convenient, the wireless module 2064 can be omitted. Either configuration provides a single integrated unit that can be carried in the user's pocket and can store and display videos.
Any of the handheld devices shown in
FIGS. 21A through 21C—Handheld Video Devices with Graphical Zoom Control
A graphical user interface (GUI) graphically corresponds to a video display window 2110 through which a single image or a stream of video frames is displayed. The GUI and the video display window 2110 are displayed on a display 2012 (or 2012 b or 2012 d). The GUI includes a zoom control 2100 having an inner region 2102 positioned within an outer region 2106. The zoom control 2100 is a graphical way for the user of a remote receiver 1610 (see
A user controls aspects and changes parameters of the image displayed within the video display window 2110 using the controls 2014 to enter input commands within the zoom control 2100 by selecting appropriate parts of the controls 2104 (or regions of the zoom control 2100 on a touch screen or with a pointing device). The controls 2014 can be a touch screen, touch pad, iPod-like scroll pad, remote control or other device, depending on the configuration of the handheld device.
The size of the inner region 2102 relative to the outer region 2106 represents the magnification of the portion of the image being displayed within the video display window 2110. A magnification factor 104 representing the current magnification of the image being displayed within the video display window 2110 from the original image is displayed within the inner region 2102. The magnification of the image being displayed is increased by tapping within the inner region 2102, or while in zoom control mode, pressing the “zoom in” button on a iPod-type control 2104 or cell phone control 2014 d. As the magnification is thus increased, the size of the inner region 2102 is decreased appropriately relative to the outer region 2106 and the magnification factor 104 is appropriately incremented. The magnification of the image being displayed is decreased by tapping outside of the inner region but inside of the outer region, or while in zoom control mode clicking the “zoom out” button on a iPod-type control 2104 or cell phone control 2014 d. As the magnification is thus decreased, the size of the inner region 102 is increased appropriately relative to the outer region 2106 and the magnification factor 104 is appropriately decremented.
The position of the inner region 2102 within the outer region 2106 represents the portion of the entire original image being displayed within the video display window 2110. The portion of the image being displayed within the video display window 2110 is changed by moving the inner region 2102 to the desired position within the outer region 2106 using the touch screen, a pointing device, or the controls 2014 or 2014 d. As the position of the inner region 2102 changes within the outer region 2106, the portion of the image displayed within the video display window 2110 changes appropriately.
The display 2012 including the video display window 2110 and a graphical user interface including the zoom control 2100, according to the present invention. The zoom control 2100 of the present invention preferably includes two regions 2102 and 2106. The outer region 2106 forms the outer edge of the zoom control 2100 and represents the entire available original image. The inner region 2102, is included and positioned within the outer region 2106 and represents a region of interest of the original image currently being displayed within the video display window 2110. Within the inner region 2102, a magnification factor 104 is optionally displayed, representing the current magnification being applied to the image displayed within the video display window 2110.
The magnification factor 104 is changed by using the touch screen or controls 2014 (or 2014 d) to zoom in or zoom out. By zooming in a number of times, the inner region 102 becomes continually smaller in size and the magnification factor 104 is incremented a number of times equal to the number of times that the control zoomed in.
A user zooms out on a specific portion of the image to decrease the magnification factor 104; the inner region 102 becomes appropriately larger in size and the magnification factor 104 is decremented. By zooming out a number of times, the inner region 102 becomes increasingly larger with each zoom out and the magnification factor 104 is decremented a number of times equal to the number of times the user zooms out, until the magnification factor is equal to 1.
The inner region 2102 also has a pan or positional feature within the outer region 2106, such that the position of the inner region 2102 within the outer region 2106 represents the portion of the entire original image that is being displayed within the video display window 2110. The position of the inner region 2102 is changed within the outer region 2106 by using the touch screen, a pointing device, or controls 2014 to move the inner region 2102 to the desired position within the outer region 2106. Accordingly, the inner region 2102 graphically represents what portion of the entire image is currently being displayed within the video display window 2110 and what magnification factor 104 is currently being used to make this selected portion of the original image fit within the video display window 2110.
The present invention will allow low cost, portable, video transmission of events of interest whenever and wherever they happen. These handheld wireless video transmitters will be able to provide news coverage of wars, natural disasters, terrorist attacks, traffic and criminal activities in a way that has never before been possible.
The present invention will enabled enhanced personal communication between friends, family, and co-workers in ways never before possible.
The present invention will enabled the transmission of video-based entertainment and education in ways never before possible. User will be able to use pocket-sized, handheld device to watch video that are downloaded from a global media exchange, streamed from a video server, or transmitted live from a performance, classroom, laboratory, or field experience.
The present invention would enable a physician or medical specialist to receive medical quality video any time in any location. For example, a critical emergency room ultrasound study could be monitored while it is being performed by less skilled emergency room personnel ensuring that the best medical image is acquired. A rapid diagnosis can be made and the results of a study can be verbally dictated for immediate transcription and use within the hospital.
Further, the present invention could be used to transmit medical quality video from a remote, rural location, including a battle ground. It could also be used to transmit guidance and advice from an expert physician into a remote, rural location.
Thus, the present invention can improve medical care, reduce the turnaround for analysis of medical studies, reduce the turnaround for surgery, and provide medical professionals with continuous access to medical quality imaging.
The removal of the least significant bits of pixel values results in high quality decompressed images when the original image is generated by an electronic sensing device, such as an ultrasound machine, which is generating only a certain number of bits of grayscale resolution. By variably altering the number of most significant bits, various filters can be implemented to enhance the image quality. Such a noise filter can be beneficial when the image is generated by an imaging technology such as radar, ultrasound, x-ray, magnetic resonance, or similar technology. Variations can be made to enhance the perceived quality of the decompressed image. Therefore, altering the number of data bits selected and altering the width of the repeat count is anticipated by this invention and specific values in the examples should not be construed as limiting the scope of this invention.
While a video stream is being viewed a viewer on the decoding end of the transmission can vary the settings for the compressor. Different tradeoffs between image spatial and temporal quality can be made. As the contents of the video signal change an appropriate format can be selected. Control signals can be sent back to the compressor via a communications link.
The preferred embodiment of this invention uses a number of techniques to reduce the time required to compress and decompress the data.
The methods require only a single sequential pass through the data. Both the compression steps 100 and the decompression steps 150 access a pixel once and perform all calculations.
When selecting the filtered pixel value 299, the preferred embodiment selects the low order byte from the 32 bit pixel value 200 or the 24 bit pixel value 210 so that an additional shift operation or addressing operation is avoided.
The shift operation is a fast and efficient way to convert a byte or word to the filtered pixel value 299.
The lossless compression of the sampled data achieved by the preferred embodiment of the present invention results in high quality video streams that have general purpose application in a number of areas including, without limitation, video conferencing, surveillance, manufacturing, rich media advertising, and other forms of video transmission, storage, and processing.
Once the analog signal is sub-sampled and filtered to select a filtered pixel value that eliminates some of the real world defects, the methods of the present invention compress and decompress the data with no irreversible data loss. Unlike JPEG and MPEG, the decompressed image never suffers from artificially induced blocking or smearing or other artifacts that are result of the lossy compression algorithm itself. As a result even a small sub-sample of the image remains clear and true to the perceived quality of the original image.
Superior Features over RHN Format
When compared against the RHN format, the format and methods of the present invention provide a number of advantages, including, but not limited to, faster speed and smaller size of encoded data, better performance for both medical and typical video images, and a typically closer representation of the original video signal.
Accordingly, the reader will see that handheld wireless devices are used to receive and display high quality video. The video can be displayed as it is received live and a graphical zoom control can be used to dynamically control the area of the source image that is to be transmitted in full resolution. In other embodiments, a handheld wireless device captures the video with an attached video camera and microphone and the device transmits the video images live as they are captured. A single handheld wireless video transmitter can transmit to multiple handheld wireless receivers. A plurality of handheld wireless video devices which capture, transmit, receive, and display video over a network are used for mobile video conferencing. In other embodiments the video data is transferred as a video file or streamed from a video server contain pre-recorded video files.
Further the compression and decompression steps of the present invention provides a means of digitally compressing a video signal in real time, communicating the encoded data stream over a transmission channel, and decoding each frame and displaying the decompressed video frames in real time.
Furthermore, the present invention has additional advantages in that:
Although the descriptions above contain many specifics, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the preferred embodiments of this invention. For example, the physical layout, cable type, connectors, packaging, and location of the video display or video camera can all be altered without affecting the basic elements of the claimed embodiments. Further, bit ordering can be altered and the same relative operation, relative performance, and relative perceived image quality will result. Also, these processes can each be implemented as a hardware apparatus that will improve the performance significantly.
Thus the scope of the invention should be determined by the appended claims and their legal equivalents, and not solely by the examples given.