WO2004102957A1

WO2004102957A1 - Combined imaging system

Info

Publication number: WO2004102957A1
Application number: PCT/SG2004/000083
Authority: WO
Inventors: Kevin Ng; Hai Seng Michael Liew; Woei Ling Leow; Mingpo Tham
Original assignee: Honeywell International Inc.
Priority date: 2003-05-13
Filing date: 2004-04-07
Publication date: 2004-11-25
Also published as: CN1820497A; EP1627525A4; JP2006528474A; SG138442A1; EP1627525A1

Abstract

Apparatus for producing combined images of a person, the apparatus including : a video camera for producing a video image of the person, a thermal-imaging camera for producing a thermal image of the person. The video camera has a first field of view and the thermal imaging camera has a second field of view, the second field of view at least overlapping with the first field of view. The video and thermal images are for real-time display on a screen of a monitor in an adjacent relationship. A method for capturing the images is also disclosed.

Description

Combined Imaging System

Field of the Invention

The present invention relates to a combined imaging system and refers particularly, though not exclusively, to a combined imaging system wherein a plurality of different forms of image of a person may be simultaneously displayed and recorded.

Background to the Invention

Security systems quite often take real-time video images of the face of a person for facial, recognition purposes when the person is in locations such as, for example, controlled environments. Similarly, thermal images of people are taken to determine thermal characteristics of those people for detecting high or low body temperature and thus the likelihood of that person having an infectious disease, be under high stress, or for security purposes. Controlled environments include locations such as, for example, airports, ship terminals, train stations, hospitals, office buildings, military bases, factories, commercial installation, and the like. These are locations where security may be an issue. Security includes not only weapons detection, identifying suspect persons, and the like, but also detection of those who may have a contagious disease.

Existing systems for thermal imaging do not identify the person. If a large number of people are involved, this may reduce the effectiveness of the thermal imaging system. It also makes it difficult to track back to identify and locate people some time after the thermal image was taken.

Furthermore, existing system for thermal imaging rely on an operator to determine if a person should be subject to further examination. An error in judgement by the operator, or a lapse in concentration by the operator, may have significant, adverse consequences.

Summary of the Invention

In accordance with one aspect of the invention there is provided apparatus for producing combined images of a person, the apparatus including: a first camera for producing a first image of the person and a second camera for producing a second image of the person. The second image is of a different category to the first image. The first camera has a first field of view and the second camera has a second field of view. The second field of view at least in part overlaps with the first field of view. Preferably, it is coincident with and not greater than the first field of view.

In a second form there is provided a method for providing a combined image of a person, the method including the steps:

(a) taking a first image of the person with a first camera when the person is in a first field of view of the first camera;

(b) taking a second image of the person with a second camera when the person is in a second field of view of the second camera, the second field of view at least in part overlapping with the first field of view, and the second image being of a different category to the first image; and (c) displaying the first image and the second image on a display screen in an adjacent relationship.

Preferably, the second field of view is coincident with and not greater than the first field of view. In both forms, the images may be for real-time display on a screen of a monitor in an adjacent relationship.

The first camera may be a colour video camera and the first image may be a colour video image; and the second camera may be a thermal imaging camera and the second image may be a thermal image. The thermal image may include a colour/temperature scale.

Preferably, the first camera and the second camera capture the first image and the second image respectively simultaneously and continuously when the person is within the second field of view and the first field of view. The first image is preferably of a face of the person.

There may be included a thermal imaging processing system for determining if a temperature of the person is above a predetermined maximum temperature and for generating an event if the temperature of the person is above the predetermined maximum temperature. Alternatively or additionally, the thermal imaging processing system may be for determining if the temperature of the person is below a predetermined minimum temperature and for generating an event if the temperature of the person is below the predetermined minimum temperature. The thermal imaging processing system may also be for determining if the temperature of the person is outside a predetermined temperature range and for generating an event if the temperature of the person is outside the predetermined temperature range. The predetermined temperature range may be from a predetermined minimum temperature to a predetermined maximum temperature. The generating and event may be raising an alarm, raising an alert, and so forth.

There may be a recording device to record and store the first image and the second image. The recording may be in colour. The first image and the second image may be stored separately or together. When recorded, recording data such as, start date and time, first date and time, and location may also be recorded.

Image may include a still image, continuous images at a predetermined number of frames per second, and image data.

In a final form there is provided apparatus for producing combined images of a person in a field of view, the apparatus including a digital video camera for producing a colour video image of the person, and a thermal imaging camera for producing a colour thermal image of the person, the colour video image and the colour thermal image being for adjacent real-time display on a screen of a monitor; and a recording system for recording in real time the colour video image, the colour thermal image, and recording data.

Description of the Drawings

In order that the invention may be better understood and readily put into practical effect, there shall now be described by way of non-limitative example only preferred embodiments of the present invention, the description being with reference to the accompanying illustrative drawings in which:

Figure 1 is an illustration of the overall system architecture; Figure 2 is a flow chart of a first form of process; Figure 3 is a flow chart of a second form of process; Figure 4 is a flow chart of a third form of process;

Figure 5 is a flow chart of the image processing system; and Figure 6 is a flow chart of the image recording system. Description of the Preferred Embodiments

To first refer to Figure 1 , there is shown a preferred form of system architecture using two cameras 10, 20 of different categories. The cameras 10, 20 may be selected from any category of known cameras such as, for example, video, thermal imaging, rear infrared, ultra violet, x-ray, and so forth.

As illustrated there is the first camera 10 for providing a first form of image of a person. The first form of image is preferably a colour video image and the first camera 10 is preferably a video camera for producing colour video. In such an instance the first camera 10 may be a digital video camera. By having the video camera in colour, identifying features of the person may be able to be captured. Identifying features may include hair colour, eye colour, clothing details, and so forth.

The first camera 10 has a lens 11 that is focused on a first field of view 12 in which a person whose image is to be captured will be located at the time their image is captured.

First image data from first camera 10 is passed along first line 13 to a first video server 14. First line 13 may be a wireless link, if required or desired. First video server 14 is connected to a central video server 30 by line 15. Line 15 may be a wireless link, over a LAN, WAN, the Internet, or other network. Preferably, the network is a TCP/IP Ethernet network. Central server 30 is operatively connected to a monitor 31. Monitor 31 has display screen 32.

There is also the second camera 20 for providing a second form of image of the person. The second form of image is preferably a thermal image and the second camera 20 is preferably a thermal imaging camera such as, for example, a thermal imaging camera as supplied by Mikron Infrared, Inc of Oakland, N.J., USA. Being a thermal image it will inherently be in colour.

The second camera 20 has a lens 21 that is focused on a second field of view 22 in which the person whose image is to be captured will be located when this image is captured. The second field of view 22 at least overlaps with the first field of view

12 so it can be confirmed they will be of the same person. Preferably, the second field of view 22 is coincident with and located within the first field of view 12. Second field of view 22 is more preferably of a size not greater than the size of the first field of view 12. In this way the two images align and, when displayed, are of the same general view of the same person. This allows an operator to more accurately determine they are of the same person. It also allows mapping of the subject in view from the first view to the second view, and from the second view to the first view. This may assist confirming they are of the same person.

Second image data from second camera 20 is passed along second line 23 to a second video server 24. Second line 23 may be a wireless link if required or desired. Second video server 24 is also connected to central video server 30, preferably by line 15.

Cameras 10, 20 may be in separate housings or may be contained within the one housing. It is preferred they be relatively close to minimise parallax error.

Cameras 10, 20 capture their images simultaneously and preferably continuously when a person is in the second field of view 22, and thus the first field of view 12. In normal mode cameras 10, 20 capture images at about 25 frames per second. However, the capturing of images may be at different intervals such as, for example, twenty four frames per second, ten frames per second, four frames per second, one frame per second and so forth. In certain circumstances, the images captured may be still images. The capturing of images should continue while the person is within the second field of view 22. The initialisation of the capturing of images may be by an operator, or may be by known motion sensing technology. Similarly, the ceasing of the capturing of images may be operator determined, or by known motion sensing technology.

To refer to Figure 2, when a person enters a controlled environment (40) they walk through an area with the second field of view 22, and thus the first field of view 12. Controlled environments include locations such as, for example, airports, ship terminals, train stations, hospitals, office buildings, military bases, factories, commercial installation, and the like. These are locations where security may be an issue. Security includes not only weapons detection, identifying suspect persons, and the like, but also detection of those who may have a contagious disease. Cameras 10, 20 capture (41) at least facial images of the person. The face of the person should be captured by both cameras 10, 20. Second camera 20 may capture more than the face, including the upper body, trunk , and so forth as required.

With additional reference to Figure 5, the first and second images captured by cameras 10, 20 respectively are passed on a frame-by frame basis (70) as digital image data to, and are processed by, servers 14, 24 respectively. The digital image data is then passed to central video server 30 and are displayed in real time on screen 32 of monitor 31. Preferably, the first and second images are displayed in an adjacent relationship such as, for example, side-by-side, one above the other, and so forth. In this way, an operator at central video server 30 can view screen 32 and identify the person by virtue of first image 33 as displayed on screen 32, and adjacent second image 34 as displayed on screen 32. Preferably, the operator can input x, y co-ordinates (71) to specify a region of interest for either or both images. This can assist mapping of the image of the person from one view to the other, and to assist confirmation the two views are of the same person.

Second image 34, being a thermal image, does not identify the person. First image 33, being a video image, enables the operator to identify the person whose image has been captured by cameras 10, 20. First image 33 may be analysed by known image recognition technology for security purposes, if desired.

As shown on Figure 1 , second image 34 has a colourΛemperature chart 35 displayed as part of the image. In Figure 2, if the operator detects that a colour for a temperature above a predetermined maximum temperature is present (42), it indicates the person may have a fever and thus a guard is alerted (43). The person is directed to a checkpoint for further checks (44). The predetermined maximum temperature maybe, for example 37.5°C. However, it may be any other possible temperature above normal body temperature (36.4°C) such as, for example, 37°C, 38°C, 38.5°C, and so forth.

If the operator detects that a colour for a temperature above the predetermined maximum temperature is not present the person is allowed to proceed (45).

The system may also check for an abnormally low temperature - a temperature below a predetermined minimum. This is shown in Figure 3. The procedure here is exactly the same as for Figure 2 and thus like processes have like reference numerals but with the prefix number 5 rather than 4. Here, the operator checks to see if the colourΛemperature chart 35 indicates a colour for a temperature, thus indicating the presence of a non-heat-generating article on the person. That may be, for example, an explosive device. The predetermined minimum temperature may be, for example, 36°C, 35.5°C, 35°C or the like. If part of the person is below the predetermined minimum temperature (52) there may be an explosive device or the like attached to or worn by the person. The operator alerts the guard (53) and the person is taken to a checkpoint for further processing (54).

The embodiment of Figure 2 may be combined with the embodiment of Figure 3 so that the operator checks for a maximum predetermined temperature, and a minimum predetermined temperature. That is, the check is for an acceptable range of temperatures from the minimum predetermined temperature to the maximum predetermined temperature. Any temperature outside that range is unacceptable and the guard alerted.

As the operator must maintain a high degree of concentration for extended periods, and as relative colours are subjective, Figure 4 shows a third embodiment where a decision support system is included. Again, like processes use like reference numerals with a prefix number of 6 rather than 4 or 5.

The differences are in process steps 63 and 63. Here, the image data for second image 34 is analysed by a thermal image processing system that determines if the temperature is greater that the predetermined maximum temperature and/or if the temperature is less than the predetermined minimum temperature - i.e. is outside the allowable temperate range. If, yes, an alarm is sounded and the person directed to a checkpoint for further checks (63). If no, the person is allowed to proceed (65).

As shown in Figure 5, the image processing system conducts its determination on a frame-by-frame basis (72). If the temperature is not outside the predetermined maximum and/or minimum, the image processing system moves to the next frame. If a yes result is achieved, an event is generated (73). The event may be an alert, alarm or the like. The system also enables central server 30 to record all first images 33 and second images 34 and to store the recorded images 33, 34 as image data. This is shown in Figure 6. Recording is preferably on a frame-by-frame basis. The image data is preferably date, time and location stamped before storage. This may be as a result of a prompt for an operator to input recording data (76). Storage should be for at least a predetermined time. The predetermined time will depend on the circumstances. For disease control the predetermined time should be greater than the incubation time of the disease; for an airport, it should be greater that the anticipated flight time; and so forth.

Preferably, the screen display includes a number (preferably three) interfaces to enable an operator to control recording. The interfaces may to control the start of recording, end of recording, and playback function all by mouse click. Upon the display system being activated, the interfaces are displayed. Upon the operator selecting the start for recording the record data is required to be entered. The record data may include operator ID, location, date and time. Location may be, for example, an airport departure gate, building entrance, and so forth.

Upon the record data being entered (76) the recording commences (77) and continues until the operator selects 'stop' (78). The end record data may then be entered (79). The end record data may be for example, operator ID and time.

In many circumstances the person may be required to produce some form of identity at the same time as the image data is captured by cameras 10, 20. The identity may be a security pass, airline boarding pass, access code, identity card, passport, or the like. The data pertaining to the identity may be captured by the system. It may also be displayed on screen 32 with the images 33, 34; and may also be recorded and stored with the image data for images 33, 34.

In this way it is possible to back track to previously recorded images if required, and identity individuals by identity and/or facial characteristics. This may be of importance for security and/or disease control.

Whilst there has been described in the foregoing description preferred embodiments of the present invention, it will be understood by those skilled in the technology that many variations or modifications in details of design, construction or operation may be made without departing from the present invention.

Claims

The Claims

1. Apparatus for producing combined images of a person, the apparatus including: a first camera for producing a first image of the person, a second camera for producing a second image of the person, the second image being of a different category to the first image; the first camera having a first field of view and the second camera having a second field of view, the second field of view at least in part overlapping with the first field of view.

2. Apparatus as claimed in claim 1 , wherein the second field of view is coincident with and not greater than the first field of view.

3. Apparatus as claimed in claim 1 , wherein the first camera is a colour video camera and the first image is a colour video image.

4. Apparatus as claimed in claim 1 , wherein the second camera is a thermal imaging camera and the second image is a thermal image.

5. Apparatus as claimed in claim 1 , wherein the first camera is a colour video camera and the first image is a colour video image; and the second camera is a thermal imaging camera and the second image is a thermal image.

6. Apparatus as claimed in claim 1 , wherein the first and second images are for real-time display on a screen of a monitor in an adjacent relationship.

7. Apparatus as claimed in claim 4, wherein the thermal image includes a colour/temperature scale.

8. Apparatus as claimed in claim 1 , wherein the first camera and the second camera capture the first image and the second image respectively simultaneously and continuously when the person is within the second field of view and the first field of view.

9. Apparatus as claimed in claim 8, wherein the first image includes a face of the person.

10. Apparatus as claimed in claim 6, further including a thermal imaging processing system for determining if a temperature of the person is above a predetermined maximum temperature and generating an event if the temperature of the person is above the predetermined maximum temperature.

11. Apparatus as claimed in claim 6, further including a thermal imaging processing system for determining if a temperature of the person is below a predetermined minimum temperature and for generating an event if the temperature of the person is below the predetermined minimum temperature.

12. Apparatus as claimed in claim 6, further including a thermal imaging processing system for determining if a temperature of the person is outside a predetermined temperature range and for generating an event if the temperature of the person is outside the predetermined temperature range.

13. Apparatus as claimed in claim 12, wherein the predetermined temperature range is from a predetermined minimum temperature to a predetermined maximum temperature.

14. Apparatus as claimed in claim 1 , further including a recording device to record and store the first image and the second image.

15. Apparatus as claimed in claim 1 , wherein the combined images include still images, continuous images at a predetermined number of frames per second, and image data.

16. Apparatus for producing combined images of a person in a field of view, the apparatus including a digital video camera for producing a colour video image of the person and thermal imaging camera for producing a colour thermal image of the person, the colour digital image and the colour thermal image being for adjacent real-time display on a screen of a monitor; and a recording system for recording in real time the colour video image, the colour thermal image, and recording data.

17. A method for providing combined images of a person, the method including the steps: (a) taking a first image of the person with a first camera when the person is in a first field of view of the first camera;

(b) taking a second image of the person with a second camera when the person is in a second field of view of the second camera, the second field of view overlapping at least in part with the first field of view, and the second image being of a different category to the first image; and

(c) displaying the first image and the second image on a display screen in an adjacent relationship.

18. A method as claimed in claim 17, wherein the second field of view is coincident with and not greater than the first field of view.

19. A method as claimed in claim 17, wherein the first camera is a colour video camera and the first image is a colour video image.

20. A method as claimed in claim 17, wherein the second camera is a thermal imaging camera and the second image is a thermal image.

21. A method as claimed in claim 17, wherein the first camera is a colour video camera and the first image is a colour video image; and the second camera is a thermal imaging camera and the second image is a thermal image.

22. A method as claimed in claim 17, wherein the first and second images are displayed in real time.

23. A method as claimed in claim 20 wherein the thermal image includes a colour/temperature scale.

24. A method as claimed in claim 17, wherein the first camera and the second camera capture the first image and the second image respectively simultaneously and continuously when the person is within the second field of view and the first field of view.

25. A method as claimed in claim 24, wherein the first image includes a face of the person.

26. A method as claimed in claim 17, wherein the display of the first image is able to be mapped to the display of the second image.

27. A method as claimed in claim 17, wherein the display of the second image is able to be mapped to the display of the first image.

28. A method as claimed in claim 21 , further including the steps: determining if a temperature of the person is above a predetermined maximum temperature; and generating an event if the temperature of the person is above the predetermined maximum temperature.

29. A method as claimed in claim 21 , further including the steps: determining if a temperature of the person is below a predetermined minimum temperature; and generating an event if the temperature of the person is below the predetermined minimum temperature.

30. A method as claimed in claim 21 , further including the steps: determining if a temperature of the person is outside a predetermined temperature range; and generating an event if the temperature of the person is outside the predetermined temperature range.

31. A method as claimed in claim 30, wherein the predetermined temperature range is from a predetermined minimum temperature to a predetermined maximum temperature.

32. A method as claimed in claim 17, further including a recording device to record and store the first image and the second image on a frame-by-frame-basis.

33. A method as claimed in claim 24, wherein the first image and the second image include a still image, continuous images at a predetermined number of frames per second, and image data.

34. A method for producing combined images of a person in a field of view, the method including the steps: (a) taking a colour video image of the person with a digital video camera;

(b) taking a thermal image of the person with a thermal imaging camera; (c) displaying the colour video image and the thermal image in adjacent relationship in real-time on a screen of a monitor; and

(d) using a recording system for recording in real time the colour video image, the thermal image, and recording data.