WO2005024676A1

WO2005024676A1 - Method, apparatus and related computer program for replying and creating e-mails

Info

Publication number: WO2005024676A1
Application number: PCT/EP2004/051936
Authority: WO
Inventors: Manuel Gonzalez; Jose Abad
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2003-09-10
Filing date: 2004-08-27
Publication date: 2005-03-17
Also published as: GB0321173D0

Abstract

A method of replying to an e-mail or amending a draft e-mail or creating an e-mail comprises printing out a received e-mail, handwriting reply markin: s on the printed e-mail using an electronic digital pen which both physically writes on the printed e-mail and captures digitally the reply markings and their position on the printed e-mail. A digital version of the reply markings is created and both the draft or received e-mail, when used with pre-existing emails, and the digital version of the reply markings arc sent to the sender of the received e-mail. Apparatus and software for carryin: out the method are also disclosed.

Description

METHOD, APPARATUS AND RELATED COMPUTER PROGRAM FOR REPLYING AND CREATING E-MAILS

Field of the Invention

This invention relates to replying to e-mails, to creating and sending e-mails, and to apparatus, methods and software relating to such activities.

Background of the Invention E-mails are a part of everyday life for many people, at work and at home. Replying to them is easy when one is sitting in front of one' s desktop, PC, for example at work. It is nearly as easy to create original e-mails. Replying to e-mails and creating new original e-mails is almost as straightforward using portable devices, such as laptops, personal digital assistants, palm-sized devices, and even mobile telephones (although that is perhaps not so easy due to the small screen size). However, not everyone has such devices. Furthermore, if e-mails are handled using' a portable device it is often still necessary to update a person 's main, base, PC e-mail record or else the record on one machine will not be complete and may be misleading. It is known to have portable electronic devices with wireless telecommunication capability dock with desktop computers and synchronise data so that they update each other.

One common way of working with e-mails is for a user, typically in a work environment but not necessarily so, to come to the end of their session on their PC, possibly because it is time to go home, and still have e-mails to answer, e-mails partially prepared in draft and not sent out, and the user still has new e-mails they know they must write. They know they will have some time later, for example at home or whilst commuting (e.g. on the train), when they can work on their e-mail workload. They print out the e-mails they want to answer, and the draft e-mails, and take the printed paper versions with them to amend using a pen later. They amend the paper versions using a pen, and even write new e-mails on paper so that their thoughts are in order for the next day. When they get back to their PC at work the next morning they take out their paper copies of the e-mails and call up the electronic versions on their PC' s screen and type in the changes, and then send the e-mails. Alternatively, they give the final amended e-mails to their secretary for them to type in the changes and new e-mails.

Some people avoid the above scenario by accessing their e-mails at home and amending them from there. Others download the e-mails of interest to a portable electronic device (e.g. laptop or PDA) and work on them electronically, and either send the revised/newly created e-mails directly from their portable electronic device, or electronically transfer the revised e-mails/new e-mails to their main PC for sending the next day.

Some people just work better, or more naturally, with pen and paper and so prefer to print e-mails out on paper for working on by hand - even whilst they arc in the office, possibly even whilst sitting in front of the desktop PC. Some people think more carefully that way. They may do this whilst they are in the office, and give the revised e-mail to their secretary for typing/amending on the computer, or they may do the typing themselves.

Summary of the Invention

According to a first aspect the invention comprises a method of replying to an e-mail comprising printing out a received e-mail, handwriting reply markings on the printed e-mail using an electronic digital pen which both physically writes on the printed e-mail and captures digitally the reply markings and their position on the printed e-mail to create a digital version of the reply markings, and, optionally, sending both the received e-mail and the digital version of the reply markings to the sender of the received e-mail. The digital version of the reply markings may comprise in part or completely a facsimile copy of the handwritten reply markings. Not all of the handwritten reply markings may be sent to the original sender of the received e-mail: for example some (or indeed all) of the handwritten markings may comprise instructions to a computer and may be interpreted by the computer, and acted upon, rather than simply being sent to the sender of the original, received, e-mail. The reply markings may comprise content markings that are transmitted to the sender of the original e-mail and/or functional operation markings that may, or may not, be sent as visible markings to the sender of the original e-mail. For example, the printed e-mail may have an area to be marked to indicate that the writing of the reply e-mail is complete, and putting a mark in this area may not result in that mark being transmitted to the sender of the original e-mail. Also, there may be an emphasis marking which indicates that a particular word, or part, of the original e-mail is to be highlighted in some way (e.g. made in bold type, in a different font, or in colour), and such an emphasis marking may not be sent to the sender of the original e-mail, instead an interpretation of its intent may be sent: a result pursuant to the marking, but not the marking itself. For example, indicating or encircling a word in a loop may in some embodiments be sent as an underlining of the word, or as an encircling of the word, but in other it may result in the word being send in bold type, instead of being underlined or encircled. As another example, putting a line through a word with the pen may be sent as just that: a line through the word, or it may be interpreted as deletion of the word and the word may not be present in the reply e-mail that is sent.

Sending both the received e-mail and the digital version of the reply markings to the send of the received e-mail may comprise combining the received e-mail and the digital version of the reply markings before they are sent and sending them as a single combined e-mail. Alternatively, they could be sent as separate e-mails and combined at the recipient's end of the telecommunications transmission path.

In some embodiments, we may not send the original e-mail back to the original sender: they may already have it stored. We may send only the changes/reply markings, possibly with instructions regarding how to overlay the markings on the original e-mail (or the original sender's machine may already have the overlaying/joining protocol). If an e-mail is sent in reply to an earlier e-mail, and the reply does not include the original e-mail because it expects the original e-mail to be present at the other end of the internet and to be able to overlay the reply upon the original, there is a chance that the original may not be present at the original sender's end after all for example, (it may have been deleted). In such circumstances a request from the original sender's computer back to the recipient' s computer for the original e-mail to be sent may be made automatically. This may help to reduce the volume of telecommunication traffic.

There may in some circumstances be a need to handle properly different versions of an email. This may be achieved in some embodiments by generating a PDF document using a received (or created) email and associating position-identifying pattern with the email content in the PDF document (i.e. storing the email content and the pattern together in the PDF document). The PDF document may then be used as the original document (received email or draft email) , and the PDF document would be the candidate for exchange over the appropriate communication channel. This can be used to achieve version control.

Another possibility is to have the received or draft electronic email held in an appropriate folder of the email creating/replying computer filed "as is" - without modification, or without significant modification. Modification of the existing receive email or draft email (or an attachment) may result in modification information or a modification notice being presented to the user. For example, in Outlook™ modification of a message/attachment causes a pop-up window to appear warning to the user and they need to confirm that they wish to proceed with those acts.

In addition to printing the original e-mail onto paper, a position-determining pattern may be printed onto the paper in the same printing operation, the pattern enabling the pen to determine its position, thereby enabling the reply markings to be captured digitally by the pen.

An alternative may be to use dead-reckoning to determine the position of the pen, the pen having movement sensors, such as gyroscopes. A way marker, or start point, may be printed on the printed e-mail to initialise the pen' s position relative to the page prior to making subsequent movements.

The identity and/or content of a particular received e-mail that is printed out is electronically linked to the specific sheet of paper (or other printed article) that carries the printed e-mail. In this way it is possible to allocate particular reply markings with a particular e-mail (electronic version of e-mail). This identity or content linking may be achieved by printing an identity reference onto the paper/printed article and associating the handwritten markings with the identity code, and also associating the electronic version of the e-mail with the identity code. The identity code may be provided by printing an identifiable and unique (or hitherto unused/unallocated) pen-readable position-determining pattern onto the paper, and using the pattern not only to allow the pen to determine where it is on the pattern, but also to link a subset of positions in an overall large pattern with a specific received e-mail. This association of a subset of pattern with a specific printed e-mail may be made at the time of printing of the e-mail. Instead of unused pattern, we may re-use pattern that has not been used for a predetermined length of time, or pattern selected by some other criteria.

According to a second aspect, the invention comprises a method of editing an existing e-mail held on a computer processor comprising printing out the existing e-mail and writing on the printed e-mail, or otherwise marking the printed e-mail, with a digital pen which both physically marks the printed e-mail and determines its movements over the printed e-mail to create a digital editorial record, and communicating the digital editorial record from the pen to said computer processor and using the digital editorial record to annotate and/or change the existing e-mail.

According to another aspect the invention comprises computer program product, possibly encoded on a readable data carrier, which, when running on a computer processor having access to an existing e-mail including the e-mail address of the sender of the existing e-mail, is adapted to use input signals from a digital pen representative of handwritten amendments, annotations, or markings made superimposed upon and/or adjacent a printed version of said existing e-mail, together with said existing e-mail and sender e-mail address, to create a reply e-mail.

The computer program product may be adapted to send the reply e-mail to the sender's e-mail address. The computer program product may be adapted to send the reply e-mail, possibly as a draft reply, or possibly as a copy of a reply that has actually been sent to the e-mail address of the sender of the existing e-mail, to a computer processor or display screen of a recipient user who received said existing e-mail.

Said computer program product, or programme, may be adapted to interpret specific functional input signals from said digital pen as functional instructions to perform specific associated functions on or relating to said existing e-mail when creating said reply e-mail.

Said functions may include one or more, or any combination of (including any number or all of), the following:-

(i) interpreting a handwritten marking or markings on the printed e-mail as instructions to show identified parts of the existing e-mail in differentiated format (e.g. in bold or in colour or in different font) from other parts of the existing e-mail, and pursuant to such an interpretation differentiating the identified parts in predetermined manner;

(ii) interpreting a handwritten marking or markings on the printed e-mail as instructions to send the reply e-mail to all recipients of the said existing e-mail, and pursuant to such a interpretation causing said reply e-mail to be sent to all recipients of said existing e-mail;

(iii) interpreting a handwritten marking(s) on the printed e-mail as instructions to send the reply e-mail to selected identified other e-mail addresses present in an e-mail address database accessible by said computer processor, and pursuant to such interpretation obtaining said selected identified e-mail address from said e-mail address database and causing said reply e-mail to be sent to said obtained e-mail addresses;

(iv) interpreting a predetermined marking on or adjacent a word in said printed e-mail as intending the deletion of said word and pursuant to such an interpretation said program causing said word to be deleted in said reply e-mail;

(v) interpreting a predetermined marking on said printed e-mail as intending the attachment of a specified file accessible to said computer processor, and pursuant to such an assessment attaching said specified file to said reply e-mail;

(vi) interpreting marking(s) on said printed e-mail as being intended to be reproduced as a faithful facsimile(s) of the marking(s), and pursuant to such an assessment either (i) adding such markings to said existing e-mail before or at the time of sending said reply e-mail to said reply e-mail address or to a user as a draft e-mail, or (ii) sending both said facsimile marks and said existing e-mail to a recipient to be combined at a computer processor of said recipient, or (iii) sending said facsimile marks as an e-mail to a computer processor of a recipient which already has said existing e-mail to enable them to be combined at said recipient' s computer proccssor;

(vii) interpreting specified marking(s) as instructions to flag said reply e-mail as being of high importance, and pursuant to such an interpretation being made flagging said reply e-mail accordingly;

(viii) interpreting specified marking(s) as instructions to move said existing e-mail to an identified folder, and said program causing said existing e-mail to be moved to said identified folder pursuant to such an identification being made;

(ix) interpreting marking(s) as intended to be operated upon by a character recognition program to convert handwritten markings onto word processable text, and pursuant to such an identification being made operating on said markings with such a character recognition program and converting said markings into word processable text in said reply e-mail;

(x) interpreting specified marking(s) as instructions to activate, deactive, or create rules for the automatic handling of emails. Said computer program product may reside on a user's desktop computer (e.g. a PC) or laptop or general purpose computing device, or on a user' s server, such as an e-mail server. Said computer program product may reside in said digital pen.

If the digital pen is "smart" then the new, more functionally enabled, digital pen will be able to operate with existing computcrs/c-mail systems without upgrade of the existing computers.

If the computer program resides in a user's computer of one form or another, then existing digital pens which simply gather data on the pen movement, and possibly on the position in a pattern, can be used without modification as capturcrs of the markings. Both options may be provided. Of course, some functionality may be provided in the pen and other functionality in external computers.

The computer program product in some embodiments may be adjusted to create original e-mails, not in reply to an earlier e-mail, in which case a requirement to access the e-mail address of the sender of an e-mail and to use that in creating and/or sending a reply will not apply.

According to another aspect the invention comprises a computer system comprising a computer processor having access to computer memory, a pen adapted to write on paper sheet and adapted to determine its position on the sheet, the computer processor being adapted to access e-mail data relating to a received e-mail and print out the received e-mail from a digital printer to create a printed e-mail, and to receive marking data from the pen indicative of the markings the pen makes on the printed e-mail, the processor being adapted to correlate marking data with e-mail data and to prepare and/or send a reply e-mail to the sender of the received e-mail, the reply e-mail including both marking data and received e-mail data, and the processor being adapted to determine from the received e-mail data the e-mail address to which the reply e-mail is to be sent.

The computer processor may comprise a PC, possibly a networked PC, possibly on a LAN. The computer processor may comprise a server, possibly a networked server, possibly an e-mail server.

According to another aspect the invention comprises a pen having marking capturing means adapted to capture handwritten markings in digital form, and also having a computer processor adapted to receive said digital form of said handwritten markings, and having a computer memory adapted to hold an electronic version of a received e-mail, including the e-mail address of the sender of the received e-mail, and a computer program installed on said processor, or held in computer memory of said pen and installable on said processor, said computer program being adapted to convert handwritten markings on a printed e-mail, that is equivalent to an e-mail held in the pen' s computer memory, into a digital version of said handwritten markings and to combine said digital version of said markings with said electronic version of said e-mail, or otherwise associate them, so as to enable said pen to send both the received e-mail and the digital version of the reply markings to the e-mail address of the sender of the received e-mail.

According to another aspect the invention comprises a printed page or pages having printed on it/them the content of an e-mail and a position-determining pattern capable of being read by a digital pen in order to determine the movement of the pen when it writes on the pattern.

The content and pattern may have been printed in the same print operation from the same printer, upon demand from a computer user. According to another aspect the invention comprises, in combination, a printed page or pages according to the preceding aspect of the invention and a digital pen adapted not only to write on the paper but also to record digitally handwritten markings made with the pen.

The content may be printed in a different ink from the pattern. The pen may be able to sec the ink used for the pattern, but not the ink used for the content. The pattern ink may absorb at an infra-red wavelength, whilst ink or inks used to print the content, and possibly ink used by the pen, do not absorb significantly at that wavelength.

With hindsight it is possible to see that Anoto Group AB have a digital pen and digital paper e-mail application for their "dot pattern" digital position-determining technology.

The Anoto e-mail pad allows a user to write out e-mails that they wish to send onto a pad of e-mail forms, using their special pen that knows its position. The user writes out in full the e-mail address to which they want to send the e-mail and writes in the e-mail. The user can e-mail themselves by ticking a box. However, Anoto's digital paper is pre-printed using offset printing at specialist printer factories and pre-ordered a long time in advance of using the paper. Anoto's e-mail system knows from the dot pattern on the paper that what is being written is to be sent to an e-mail application, and that what is handwritten in the e-mail address box on the e-mail form is an e-mail address.

Anoto does not envisage printing content, such as text from an existing e-mail, onto their e-mail form. Their e-mail forms arc for original, initial, e-mails, not replies to incoming e-mails. Anoto does not link word-processable content with a specific form: it links nothing electronic to the form, except the purchaser of the forms own e-mail address. This is not likely to vary over time, and between different e-mails, and so Anoto can pre-print hundreds or thousands of e-mail forms.

According to another aspect the invention comprises a method of creating and/or sending an e-mail by writing markings on a printed sheet using a pen which is able to determine its position on the sheet, and using the marking of (i) predetermined markings, and/or (ii) marking the sheet at predetermined positions on the sheet, to select a selected one of an associated computer resource from a plurality of associated computer resources potentially available to an e-mail creating and/or sending processor, the processor using the selected resource in the creation of the e-mail and/or in the sending of the e-mail.

This has the advantage that computer resources available to a processor can be brought into use to create an electronic version of an e-mail, and/or send it, by using a pen and handwriting markings on a printed sheet, and that the user of the pen selects which resources are to be used (and which are not to be used) in the creati on of a specific e-mail.

A user, for any specific e-mail, will probably not want to use all e-mail-applicable computer resources available to the e-mail processor - only a subset of them.

Said computer resources may include one or more, preferably a plurality of, computer programs or software routines which can be selected and brought into use, for a particular e-mail, using the pen.

Said computer resources may include a plurality of e-mail addresses, the pen being used to select an e-mail address or e-mail addresses to which the e-mail is to be sent. Said computer resources may include one or more, preferably a plurality of, pre-existing computer records, for example files, susceptible for incorporation into the e-mail and/or attachment to the e-mail.

Said computer resources may include one or more url addresses susceptible to being included in the e-mail as links to the site of the url.

The computer resources, in some embodiment of the invention, can be considered computer- implemented tools to enable a user to construct an e-mail, accessed/used by a user marking the printed sheet and a sheet-to-tool conversion program.

The printed sheet may be printed with a pen-position determining pattern. An association between a particular region in the pattern with a particular computer resource may be made al the time of printing of the sheet.

There may be a large virtual pattern space available to a computer processor to be printed, and a relatively small part, or parts, of that may be printed onto a specific sheet. Different positions in the pattern will code for different computer resources once the link between pattern and resource has been created in the computer processor.

An advantage of creating the link between regions of pattern/positions on the printed sheet and computer resources available for a user to select/implement using the pen is that al the lime of printing the sheet the computer processor knows what resources it has available and where they are held and may create a link table, linking specific pattern areas with specific resources.

Different printed sheets may have different computer resources available to them. A user, or a computer processor, for example the processor controlling the printing of the sheet, may select a subset of computer reserves to be printed on the sheet and to be actually available using the sheet computer, the subset to be printed being selected from a larger group of potentially available computer resources.

For example, if the computer controlling the printing of the sheet determines that the sheet is being printed by Mr X, the available e-mail addresses as computer resources for that sheet may be limited to the e-mail address book of Mr X. Similarly, the available attachments may be restricted to those to which Mr X has system access.

Similarly, the application software enabled by markings on the printed sheet may be a restricted subset of a larger potentially available group. This may be because of space limitations on the sheet and/or entitlement of a particular user to use particular application software.

Some application software may be enabled by all, or a great number, of printed sheets. For example, there will usually be software to convert pen movements into computer-resource selection acts, and that software located on the pen or on an off-pen computer, may be started up again upon the pen seeing pattern.

Similarly, many or all sheets may have a location or pattern coding for "send e-mail", and marking this region of pattern will cause the e-mail to be sent by running the appropriate "send e-mail" computer program application (computer resource).

In some embodiments a user at least in part selects which computer resources are to be accessible via the printed sheet at/or before the printing of the sheet. A user may design their own e-mail printed sheet and select the e-mail tools they want to be able to use on the sheet. A user may have a library of different e-mail printed sheets with different functionality codes into them, and may select an appropriate one printing for the type of e-mail they wish to create.

According to another aspect the invention comprises a computer processor having installed in it a paper e-mail instruction translation program adapted to convert handwritten markings made by a user on paper using a position-determining pen into computer commands to cause the creation of an electronic version of the paper e-mail, the translation program being adapted to interpret certain handwritten markings dependent upon their position or other recognition criteria as instructions to perform associated acts to use available computer resources in the creation of the electronic version of the e-mail, at least some of the markings being interpreted as a selection of specific resources to be used, from a larger group of potentially available resources.

The processor may access computer resources from computer memory available to it. The resources may include a program to (i) create facsimile representations of handwritten markings in the electronic version of the e-mail; and/or (ii) convert handwritten markings into typeset equivalent words; and/or (iii) to cause the e-mail to be sent to an individual e-mail address; (iv) cause one or more other application programs to be run to provide additional functionality controlled or, initiated by, the handwritten markings; (v) retrieve data from computer memory and to use retrieved data in the electronic version of the e-mail.

It will be appreciated that software/computer program product is intended to be protected, as is a method of efficiently utilising travelling time, or other ad-hoc time away from a computer, by reply to, or creating, e-mails using paper; as is a computer network. According to another aspect the invention comprises a computer program product, possibly encoded on a machine-readable data carrier, which, when running on processing means having access to an existing e-mail including the e-mail address of the sender of the existing e-mail, is adapted to use input signals from capture means representative of handwritten amendments, annotations, or markings made superimposed upon and/or adjacent a printed version of said existing e-mail, together with said existing e-mail and sender e-mail address, to create a reply e-mail.

According to another aspect the invention comprises an e-mail reply system comprising a computer processor having access to computer memory, and a pen comprising an ink source, an infra-red LED and an infra-red camera, the pen adapted to write on a paper sheet having a position determining pattern of dots printed thereupon using ink from the ink source and adapted to determine its position on the sheet by sensing the position determining pattern of dots using the infra-red camera; the computer processor being adapted to access e-mail data relating to a received e-mail and print out the received e-mail from a digital printer to create a printed e-mail, and to receive marking data from the camera indicative of the markings the ink from the pen makes on the printed e-mail, the processor being adapted to correlate marking data with e-mail data and to prepare and/or send a reply e-mail to the sender of the received e-mail, the reply e-mail including both marking data and received e-mail data, and the processor being adapted to determine from the received e-mail data the e-mail address to which the reply e-mail is to be sent.

According to another aspect the invention comprises a pen having marking capture means for capturing handwritten markings in digital form, and also having a processing means for receiving said digital form of said handwritten markings, and having a memory means for holding an electronic version of a received e-mail, including the e-mail address of the sender of the received e-mail, and a computer program installed on said processing means, or held in memory means of said pen and installable on processing means, said computer program being adapted to convert handwritten markings on a printed e-mail, that is equivalent to an e-mail held in the pen's memory means, into a digital version of said handwritten markings and to combine said digital version of said markings with said electronic version of said e-mail, or otherwise associate them, so as to enable said pen to send both the received e-mail and the digital version of the reply markings to the e-mail address of the sender of the received e-mail.

According to another aspect the invention comprises a pen having an infrared camera adapted to capture handwritten markings in digital form, and also having a computer processor adapted to receive said digital form of said handwritten markings, and having a computer memory adapted to hold an electronic version of a received e-mail, including the e-mail address of the sender of the received e-mail, and a computer program installed on said processor, or held in computer memory of said pen and installable on said processor, said computer program being adapted to convert handwritten markings on a printed e-mail, that is equivalent to an e-mail held in the pen's computer memory, into a digital version of said handwritten markings and to combine said digital version of said markings with said electronic version of said e-mail, or otherwise associate them, so as to enable said pen to send both the received e-mail and the digital version of the reply markings to the e-mail address of the sender of the received e-mail.

According to another aspect the invention comprises a method of creating and/or sending an e-mail by writing markings on a printed sheet using a pen which is able to determine its position on the sheet having a pattern of position determining dots printed thereupon by means of an infra-red camera arranged to capture digitally an image of an area of the paper, the camera communicating to a processor arranged to process the image in order to determine the position of the pen, and using the marking of: (i) predetermined markings, and/or

(ii) marking the sheet at predetermined positions on the sheet, to select a selected one of an associated computer resource from a plurality of associated computer resources potentially available to an e-mail creating and/or sending processor, the processor using the selected resource in the creation of the e-mail and/or in the sending of the e-mail.

Prior Art

The invention arose out of a consideration of the work of Anoto Group AB and others in relation to digital pattern paper and digital pens. It is convenient to discu ss the invention in that contextual background, but it will be appreciated that the invention is not restricted to use with any proprietary system.

The prior art Anoto digital pen and paper system is described on their website www.aτiυto functionality. com . However, since the content of websites can change with time it is to be made clear that the prior art admitted is that which was published on their website no later than the day before the priority date of this patent application. It is also appropriate to include in this application itself a brief review of the Anoto system.

Figure 1 shows schematically an A4 sheet 10 of Anoto digital paper. This comprises a part of a very large non-repeating pattern 12 of dots 14. The overall pattern is large enough to cover 60,000,000 square kilometres. The pattern 12 is made from the dots which arc printed using infra-red absorbing black ink. The dots 14 are spaced by a nominal spacing of 300μm, but are offset from their nominal position a little way (about 50μm), for example north, south, east or west, from the nominal position. In WO 01/126032, a 4x4 array of dots is described, and also a 6x6 array of dots, to define a cell. Each cell has its dots at a unique combination of positions in the pattern space so as to locate the cell in the pattern space. The dot pattern of an area of the dot pattern space codes for the position of that area in the overall dot pattern space. The contents of WO 01/126032 arc hereby incorporated by reference, with especial reference on the dot pattern and the pen.

The sheet 12 has a pale grey appearance due to the dots 14.

Figure 2 schematically shows a digital pen 20 adapted to write human readable ink in non-machine-rcadable IR transparent ink and to read a position dot pattern in infra-red. The pen 20 has a housing 22, a processor 24 with access to memory 26, a removable and replaceable ink nib and cartridge unit 28, a pressure sensor 29 adapted to be able to identify when the nib is pressed against a document, an infra-red LED emitter 30 adapted to emit infra-red light, an infra-red sensitive camera 32 (e.g. a CCD or CMOS sensor), a wireless telecommunications transceiver 34, and a removable and replaceable battery 36. The pen 20 also has a visible wavelength warning light 38 (e.g. a red light) positioned so that a user of the pen can see it when they are using the pen, and a vibration unit 40 adapted to vibrate and to cause a user to be able to feel vibrations through the pen.

Such a pen exists today and is available from Anoto as the Logitech IO™ pen.

The pen, when in use writing on a page/marking a page, sees a 6x6 array of dots 14 and its processor 24 establishes its position in the dot pattern from that image. In use the LED 30 emits infra-red light which is reflected by the page 12 and detected by the camera 32. The dots 14 absorb the infrared and so are detectable against the generally reflective background. Of course, the ink of the dots might be especially reflective in order to distinguish them (and the paper less reflective) , or they may fluoresce at a different wavelength from the radiation that excites them, the fluorescent wavelength being detected. The dots 14 are detectable against the background page.

The processor 24 processes data acquired by the camera 32 and the transceiver 34 communicates processed information from the processor 24 to a remote complementary transceiver (e.g. to a receiver linked to a PC). Typically that information will include information related to where in the dot pattern the pen is, or has been, and its pattern of movement.

There are times when the processor 24 cannot determine its position in pattern space (the overall virtual space defined by the very large dot pattern). For example, if the pen is moved too fast over the pattern the processor cannot process the images fast enough. Also the pen may not be able to see where it is in the dot pattern. This can happen if the page 14 is marked or defaced by colorants, or the pattern covered up with something, or the field of view of the pattern is obscured. The user putting their finger in the way is a common reason why the processor fails to recognise the position of the pen. In order to alert the user to the fact that the pen is not able to determine its position properly the processor 24 is adapted to illuminate the light 38 and cause the vibrator 40 to vibrate. The user gets visual and tactile feedback that the camera is not seeing the dot pattern properly/that the pen is unable to determine its position properly.

Brief Description of the Drawings Embodiments of the invention will now be described by way of example only, with reference to the accompanying drawings, of which; Figure 1 shows schematically a sheet of prior art Anoto digital paper;

Figure 2 shows schematically an existing digital pen;

Figure 3 shows schematically an embodiment of the invention for processing digital paper e-mails;

Figure 4 shows schematically another embodiment of a system according to an aspect of the invention for processing digital paper e-mails;

Figure 5 schematically illustrates an embodiment of a method of the invention of using the system; i

Figure 6 schematically illustrates a prior art Anoto e-mail template;

Figure 7 schematically illustrates an e-mail template of an embodiment of the invention, the email template being similar to one that may be accessed from Microsoft Outlook™;

Figure 8 is a schematic illustration of an e-mail of an embodiment of the invention that has been printed out as a digital paper document and has been annotated;

Figure 9 is a schematic view of a combined e-mail of an embodiment of the invention produced from printed e-mail illustrated in Figure 8; Figure 10 is a schematic illustration of one example embodiment of an aspect of the invention and comprises a printed digital copy of a received e-mail;

Figure 11 is a schematic illustration of an example an embodiment of an aspect of the invention and shows a printed received e-mail with a user message area and a sender message area positioned side by side;

Figure 12 is a schematic illustration of an embodiment of an aspect of the invention and shows an example of the printed received e-mail with a user message area printed one page and the sender message area printed on a different page;

Figure 13 is a schematic illustration of an embodiment of an aspect of the invention and shows an additional page that forms part of a printed received e-mail;

Figure 14 is a schematic illustration of an embodiment of an aspect of the invention and shows a page of a printed received e-mail that has expanded functionality;

Figure 15 is a schematic illustration of an embodiment of the invention and shows functional linking within a page of a printed e- mail; and

Figure 16 is a schematic illustration of an embodiment of the invention and shows further functional linking within a page of printed e-mail. Detailed Description of Some Embodiments of the Invention

Figures 1 and 2, relating to known Anoto digital pen and paper, have already been described.

Figure 3 is a schematic illustration of a system 100 for processing electronic e-mails and digital paper e-mails in accordance with at least one aspect of the invention. The system 100 comprises a workstation 102 including a personal computer (PC) 104 which is connected to a printer 110. The PC 104 is also be connected to the Internet 114 or a local network such as a LAN 109 having peripheral devices and/or other computers (e.g. PCs 104). The workstation 102 includes a user interface including a screen 105, a keyboard 106 and a mouse 108. The PC 104 has as a processor 107, a memory 112, and I/O software devices (not shown) by means of which the processor communicates with the screen 105 , the keyboard 104 and the mouse 106 and a communications port 1 11 by means of which it communicates with the Internet 1 14 or the local network 109. In another variant the server 116 could be connected to the LAN 109, and the LAN may be connected to the Internet.

A server 1 16 is connected to the Internet 1 14 via a telecommunications interface 1 1 8. The server 1 16 has an e-mail memory 1 17.

The system 100 illustrated in Figure 3 includes a digital pen 20. The digital pen has already been described with reference to Figure 2 and is able to send a digital version of markings made on digital paper to a receiver, for example a receiver associated with the PC 104.

Figure 4 illustrates a system 130 that is a variation of the system 100 illustrated in Figure 3. In this system the pen 130 is capable of sending e- mails directly to recipients via the Internet 114. The system 100 illustrated in Figure 3 may be used to print out an e-mail that has been received by the PC 104 via the Internet 1 14 or LAN 109, in a form that is suitable for annotation by the digital pen 20. The system 100 may also be used to print an e-mail that is in draft form (i.e. not necessarily an e-mail that has already been received). The received or draft e-mail is printed to produce a hardcopy 120 that has both the e-mail and a position pattern 12 (sec Figures 1 and 2) to enable a digital pen to identify its position on the page.

The user interface of the PC 104 allows a user to the view electronic versions of digital documents (such as e-mails) to be printed, using the software application, on the screen 105, and to make modifications to what is printed out prior to printing, or to associate certain regions of pattern with specific functionality prior to printing (discussed later). The user interface includes the keyboard 106 and mouse 108 and software (not shown) for processing inputs from them, as well as the screen 105 and software for producing the content, e.g. images and/or text, on the screen.

Figure 8 is a schematic illustration of an e-mail that has been printed out as a digital paper document 218. For clarity, only some of the icons that may be available from the icon bar 143 have been illustrated. The e-mail 218 has a typescript message 220 that was printed by the printer 110 at the same time as the rest of the e-mail and the Anoto-type digital dot pattern. The e-mail may be a received e-mail awaiting review or reply or a draft e- mail that is awaiting to be reviewed and sent. The user takes the printed e- mail 21 8 from the printer 1 10 for later review and annotation.

The user uses the digital pen 20 to annotate the printed e-mail 218. In the example illustrated the word "Brian" has been crossed through with a diagonal line 226 (i.e., a diagonal pen stroke) and the words "Ben and Jim" 228 have been written. The markings and their position on the printed e- mail 218 of these annotations 224, 226, 228 are recorded by the pen 20.

When the user has finished writing the e-mail he marks a send box 198 with the pen. Once send box 198 has been marked the pen 20 knows that the e- mail editing operation is complete and that pen movement data it has acquired is ready for transmission. The pen movement data may be transmitted immediately to a receiver, for example, a receiver that is linked to a PC, but in this example it is not transmitted until the pen is docked within a docking station linked to a PC. The PC may be a PC associated with the address of the intended recipient of the e-mail.

Alternatively, pen-acquired editing data may be sent to the PC 104 of the u ser so that the u ser can perform further editing work before sending the e- mail.

The handwritten message in the message area 192 may be transmitted to the recipient of the reply e-mail as a facsimile of what has been written or the handwritten message may be converted into a typeset using OCR or ICR (Optical Character Recognition or Intelligent Character Recognition).

It is also possible to reply to the received e-mail by fax. In this case the printed e-mail has a printed box 204 associated with the function of sending a fax. Instead of writing an e-mail address in the "To" box 194, the user writes a fax number.

In a first implementation the markings and their positions are transmitted from the pen 102 to the PC 104 of the user along with an identification code identifying the e-mail. The PC 104 then identifies and retrieves the electronic version of the received e-mail from the e-mail application and combines the markings with the received e-mail to produce a combined e- mail. Since the PC 104 has received the positions of the pen-made editing markings the markings are placed in the correct position in the message area of the combined e-mail to create a faithful facsimile of what was on the printed e-mail (both original e-mail content and extra handwritten editing). The electronic version of the received e-mail together with the annotations 226, 228 is then sent to the intended recipients via a LAN or the Internet 114. The recipients will then receive the combined e-mail which will have the same appearance as the digital paper copy produced by the user, i.e., the same appearance as the e-mail 218 illustrated in Figure 8.

Having said that the electronic version of the reply e-mail will have the same appearance as the paper version, or it might not (see later: some handwritten markings may be interpreted as instructions and acted upon, rather than being reproduced).

In a variation of this technique the original e-mail is not sent back to the original sender of the e-mail since the original sender may have stored the original e-mail. In this case only the changes/reply markings, and possibly instructions regarding how to overlay the markings with the original e-mail, are sent. The changes/reply markings are then combined with the original e-mail at the PC (or a local server) of the person who sent the original e- mail.

In another variation of this implementation the PC 104 is programmed with software such that certain markings made by the user are interpreted as having a particular function. For example the line markings 222, 226 when made on an area of the reply box on which there is a typeset word (in this case the words "was" and "Brian") will cause the word to be deleted from the combined e-mail. That is to say, not all of the handwritten markings are necessarily present in the electronic e-mail that is created using the digital pen. A further function may be to insert written text (e.g. letters, words or punctuation) 222, 228 when this text is within a predetermined area or distance adjacent to the crossed out text. In the example illustrated the word "am" 222 replaces the word "was" and the words "Ben and Jim" 228 replace the word "Brian". The replacement words may remain as facsimiles of the original handwritten words or the PC 104 may use OCR or ICR to covert the words into typeset words. Figure 9 is a schematic view of a combined e-mail 250 that has been created when the annotations have been treated this way.

The printed digital e-mail 218 has other areas with which a computer- implemented function is associated, the function being activated when the u ser marks or presses the area with the digital pen 20. For example, as shown in Figure 9, the surrounding of an area on which the word "Reply" 228 has been printed, for example by an oval 229, will have the function of making the sender of the original e-mail one of the recipients of the combined e-mail 250.

As another example, the area within the "copy to" box 230 may be used by the user to write in the e-mail addresses or names of other intended recipients. The PC 104 will know that the addresses/names have been written in the copy to box 230 and will send the combined e-mail to those recipients.

If only names are written in the "copy to" box 230 then the PC looks up the names in an electronic address book (which is stored in the PC memory 1 12, on a local server or on the Internet server 1 16) and uses the appropriate e-mail address found in the address book. The printed digital email 218 may also have areas, with which a computer implemented function is associated, for activating, deactivating or creating rules for the automatic handling of emails. The provision of such areas will allow the user to filter, sort or otherwise process incoming emails or stored emails offline (for example when they are out of the office on a train) by merely marking or pressing the areas. Examples of such rules include: deleting all emails from a specified sender or group of senders; deleting all emails that have been saved for more than a certain amount of time (e.g., emails that are older than, say, 6 months); searching for a stored email from a particular sender/group of senders; searching for a stored email relating to a particular subject (e.g., by using a key word search); sending all emails in the out box; and showing only urgent emails. The printed email 218 may include areas for the user to enter information to be used in association with such rules, for example the user may write a keyword in such an area for use with a keyword search. Specific illustrations of the use of rules may allow the user to 'delete all CNN news messages' or 'search for Cindy' s email of last week regarding product X' .

The pen movement data associated with implementing the rules may be sent to a receiver, for example, a receiver associated with a PC. In this case the processing of the emails according to the rules will have been implemented by the time the user is next online, e.g., on the user returning to their PC in the office after a holiday. Alternatively, the pen movement data may be stored in the pen and transmitted when the pen is docked with a docking station linked to the PC. In this case the user will have access to the processed emails within a short period of the docking the pen. Therefore, a person may come to the office and place their pen in the docking station and start greeting their colleagues and answering questions on their holiday. When the person has finished talking with their colleagues their processed emails will be ready and waiting for them. In both these cases the user can achieve a "head start" when next online. It will be appreciated that because the paper e-mail reply, or draft e-mail has been printed under the control of a particular computer, at the request of a particular person (print on demand by a user), there will be a specific e-mail address book associated with that person (whether that be their personal e-mail address book or their company' s company-wide address book. There is then the possibility of writing less than the full e-mail address onto the paper version of the e-mail and having a processor access the electronic e-mail address and find a match, and use the match as the e- mail address. For example, if a user knew that they wanted to e-mail Ben Brown at Hewlett-Packard Company, they may be able to write Ben@hp.com onto the paper e-mail knowing that there was only one "Ben" in the address book for Hewlett-Packard. Similarly, they could write brown @hp, or "ben brown": anything to uniquely identify an e-mail address from their address book. This is practical when the address book contains thousands, or even tens of thousands of entries, but perhaps not so practical if the address book contains a million or a billion entries: having an address book that is person or empl oying company/association specific limits the possible e-mail address in the library of pre-existing e-mail addresses to be searched.

A possibility which many embodiments will allow for is for the email address-identifying information/characters written onto the paper by the user, intended to identify a specific intended recipient for the email, not in fact uniquely identifying a single recipient from the pre-existing email address book of the email sending PC/computer. For example if the electronic address book of a user's PC has two "Bens" at HP, eg Ben. Brown @hp.com and .Bcn.Whitc@hp.com then writing ben @ hp docs not distinguish between them. In such circumstances the user's email computer, when asked to send the reply/revised email by the digital pen, may not send the email automatically but instead cause a message to be displayed on the sender's display screen informing them of the fact that the intended recipient was not uniquely identified. The message may present the sender with allowable options for possible recipients from the email address list and allow the user to select one, for example by clicking on it. The computer may then, after the identity of the recipient has been uniquely determined, send the email to the selected email address.

Instead of writing the " @" symbol, which perhaps might be thought to imply that resolution of the email address has already been achieved, a user may write the word "at", or some other connector symbol. This may reinforce in their thinking the fact that the email will only be sent to an existing email address from the pre-existing email address book accessible by the email sending device. Users could write something like "to:jose abad a HP WRT digital pen", or "to: jose digital pen", and the mapping from what terms are written onto the paper and what email address is selected from the email address list may include a correlation between memory-jogging terms and the actual email address. The user may be able to create their own mapping. For example, if the user interacts with Jose Abad because of Jose's involvement with digital pen technology, and the user mentally has Jose pictured as " Mr Digital Pen", then they may be able to create their own look up table, or other correlation mechanism between Jose Abad' s real email address (say jose_abad.Barcelona.hp.com) and the key words "digital pen", "josc", "abad", "Barcelona" (if the user know Jose is in Barcelona".

The user may be able to add to or delete correlating terms in the correlation table/mechanism.

In a second implementation the pen is configured to send the combined e- mail directly to the recipients (route 131 illustrated in Figure 4). In this implementation the pen has a computer memory 26 adapted to hold an electronic version of the received e-mail including the e-mail address of the sender of the received e-mail. The pen has a processor 24 on which is installed a computer program for converting the handwritten markings into a digital version of the markings and to combine the digital version with the electronic version of the received e-mail, or otherwise associate them with received e-mail. This enables the pen 20 to send both the received e- mail and the digital version of the reply markings to the e-mail address of the sender of the received e-mail.

The pen processor 24 may use a intelligent character recognition (ICR) or Optical Character Recognition (OCR) program. The ICR/OCR program may be used by the pen processor 24 to recognise the characters of a handwritten e-mail address or name. The processor 24 may then match these characters with a database of e-mail addresses held in the pen memory 26. In this way the pen is able to transmit the received e-mail and the digital version of the reply markings to e-mail addresses other than that of the sender of the received e-mail.

Another option, as discussed earlier, is that the generated e-mail is stored in the pen memory 26 so that the e-mail may be downloaded into the user's PC 104 via a docking device.

Figure 10 is a schematic illustration of a printed digital copy 300 of a received e-mail that is in a Microsoft Outlook™ format. The Microsoft Outlook™ format is used only for illustrative purposes and it will be appreciated that the invention is not restricted to use with any proprietary system. The printed digital e-mail 300 includes a general menu header bar 142 and an icon header bar 143. The icon header bar 143 has a number of icons. Various functions are associated with the respective icons. Examples of functions that are commonly represented by icons are: send 144, move to folder 146, save 147, print 148, cut 149, copy 150, paste 152, attach 155, flag as important 158, flag to action 158, font type 160, font size 162, pen colour 164, bold 165, italicise 168, underline 170, previous item 171 , next item 173, help 175 and delete 176. These functions are well known and will not be described here. The template has a "To" box 172 and a "Cc" box 174 where a user may enter e-mail addresses for intended recipients of the e- mail. The template also has a subject box 176, and a message box 180 for the user to write a message.

The printed e-mail 300 has a message 232 from the sender of the original e- mail in a sender message area 231. The message 232 may be annotated in the same way as has been described with reference to Figures 8 and 9.

The printed e-mail 300 also has a user message area 234, which is clear of human visible content, for the user to write a message, to draw a drawing 238 or make other markings. The user message area 234 may be beneath the sender message area 231 as is illustrated in Figure 10. The position of message areas 231 , 234 may be swapped so that the user message area 234 is above the sender message area 31 .

In the example above all of the page of the printed e-mail has position-determining pattern (e.g. dots) printed on it. In another example, areas 1 80 and 231 , may not have pattern, the user only being able to capture digitally marking made in boxes 234, which arc printed with pattern.

Figure 1 1 is a schematic illustration of an example of the printed e-mail 300 with the user message area 234 and the sender message area 231 positioned side by side. Figure 1 1 shows the sender message area 231 on the right hand side of the page and the user message 234 area on the left hand side of the page, however the message areas could equally be positioned the other way around. The positioning of the user message area 234 and the sender message area 231 in this side by side manner allows the user to write comments or graphics in the user message area 234 along side portions of text in the sender message area 231 that correspond to those portions of text. The user may even use annotations 236 that link or show the connection between the handwritten text or graphics in the user message area 234 and content in the sender message area 231. If linking between two sections of the printed e-mail is used pattern may be printed in both sections.

Of course, some digital pens do not rely on pattern to determine their position: some may have an inertial systems such as gyroscopes and accelerometers (for example).

Figure 12 is a schematic illustration of an example of the printed e-mail 300 with the user message area 234 printed on one page and the sender message area 231 printed on a different page.

The TD box illustrated as having an ID code XY1 2 may have an alphanumeric code, or may simply have position- determining pattern, used as an ID code.

Figure 13 is a schematic illustration of an additional page that forms part of the printed e-mail 300. The additional page may form part of any of the examples of the printed e-mail 300 shown in Figures 10 to 12 and may contain a user message area 234, a sender message area 231 , or a combination of both.

The format of the printed e-mail desired by the user may be chosen before the printed e-mail 300 is printed. The format may be chosen from a database of formats or may be custom designed by the user. The user may wish, for example, to intersperse user message areas 234 between sender message areas 31 to allow the user to write comments in a message area 234, that are relevant to the a portion of the sent message in an area that is adjacent to the sender message area 231. This has been illustrated in the printed e-mail 300 illustrated in Figure 13. Areas 300a have position- determining dot pattern: areas 300b do not.

Referring again to Figure 10, the user of the printed e-mail 300 has access to the functions represented by the icons on the icon bar 144 by using the printed e-mail in combination with the digital pen, as a graphical user interface with their computer. Before printing the e-mail 300 the user may customise the icon bar to include icons of his choice. This will provide the selected functionality on the printed e-mail. There may not be enough space on a printed e-mail to use all functionality available on-screen.

As an example, when a user has printed out an e-mail with a toolbar/functionality area, the user may circle, underline or otherwise identify or mark some text that he has written and then mark the bold icon 1 66 in the icon bar 143. The composite e-mail that is created using the original e-mail and the pen marking data will then have the marked text in bold text. In another example the user may circle or mark a section of text and graphics in the sender message area 231 , mark the copy icon 1 50, mark a point in the user message area 234 where the text is to be inserted and then mark the paste icon 152. This will cause the marked text to be pasted in the user message area 234. In these examples it is not necessary to make a human visible mark on the icon and instead of marking the pen 20 may be held over the icon for a specified length of lime (e.g. 3 seconds), thus allowing the icon to be recognised from the pattern area where the icon is printed.

For a printed e-mail 300, such as that illustrated in Figures 9 and 10, some of the icons in the icon bar 143 may have no use to a user using the paper e-mail away from information they need, for example the move to folder icon 146 in Figure 9 cannot be moved without an indication of the folder to which the e-mail is to be moved to, similarly the attachment icon 155 in Figure 10 cannot be used without a way of indicating on the printed e-mail which file is to be attached.

On the other hand, those icons may indeed be uscablc by a user. For example, a user may have finished editing an e-mail to go to Ben Brown at Hewlett-Packard and may know that after sending the e-mail they want it to be stored in the folder named "Project Banana". They may identify the "move to folder" 146 command on the paper e-mail (e.g. by circling it) and link that command with the file name (e.g. by writing the file name linked by a line to the command). This is schematically illustrated in Figure 15.

As another example, the attachment icon/command 155 can be used if the user knows the file/object to be attached. A user may know that a report is stored in file "HP2500" and so may indicate on the e-mail, for example by circling the attachment icon, or putting a cross in its box, or in some other way, that a command "attach" is being given by the pen and identify what is to be attached, e.g. by linking the icon 155 with a file identifier. This is schematically shown in Figure 1 6.

Figure 14 is a schematic illustration of a page of a printed e-mail 310 with expanded functionality. Certain icons have been given additional areas on the page (additional space), the additional areas having functions associated with them.

In one example the "bold" icon 166 has two boxes associated with it, one is a 'yes' box 366 the other is a 'no' box 371. A user may then circle or otherwise mark a section of text and press or mark the yes box 366 with the pen 20 to bold the marked text. To remove holding from then the text is marked and the 'no' box 367 is marked or pressed with the pen 20. In a similar way the front type 314 and font size 312 of marked text may be selected. A default box 316 can be used to set the formatting of marked text, e.g., marking or pressing the pen 20 on the default box 316 may cause the marked text to become a default font such as 12 point, Times font that has no holding or underlining.

In another example the "save" icon 148 has a number of areas associated with it that relate to predefined folders 358. Pressing or marking (e.g. crossing) one of the folder areas 358 with the pen 20 will cause the received e-mail (or the received e-mail with annotations/user markings) to be stored in the equivalent folder on the PC 104 (or local network server). The "attach" icon 155 has an associated box 355 within which the user may write a filename 356. When the e-mail is sent by the u ser to the PC 104, or to a local network server, the PC or local server searches for a file that has a name that matches the filename 356 that was written in the attach box 355 and attaches that file to the e-mail before the e-mail is sent to the intended recipicnt(s).

An "address book" box 318 may also be printed on to the e-mail 310. The electronic address book in a computer with which the paper e-mail will work has a list of names (and/or names of e-mail groups) that the user frequently uses. Associated with each name is a "To" box 320 a "Carbon copy (Cc)" 322 box and a "Blind copy" (Be) box 324. The user marks or presses the boxes 320, 322, 324 with the pen 20 according to his selection of intended recipient and the manner of delivery, i.e., as a normal, copy or blind copy delivery. A section of the printed e-mail 310 may also be provided with "Other" recipient boxes 324 for the user to write in the name or e-mail address of intended recipients that do not appear in the address book 318. These other recipient boxes 324 will also have associated "To" 320, "Carbon copy" 322 and "Blind copy" 324 boxes. It will be appreciated that other functional boxes may be printed on the e- mail 310. The user may customise the palette of icons before printing the printed e-mail (for example if he has an idea how he is going to respond to the received e-mail) so that some functional areas are printed and some are not. For example if the received e-mail is of a social nature then the user may wish that the save icon 148 has an area 359 that is associated with a folder entitled "social" on the PC 104 (or local server) but not an area 357 associated with the folders entitled "work" (and sub folders within the "work" folder) on the PC 104 or local server. Similarly, the user may limit the address book 318 so that it only has social contact names/e-mail addresses.

The user may have a number of different predetermined palettes stored in a database on the PC 104 (or local server) that he may select before printing the received e-mail. The PC 104 (or local server) may be configured so that it automatically produces a personal palette of icons according to the name of the user the e-mail was been sent to. For example the address book 31 8 may have only have the user' s personal address book and the save icon may only have associated with it folder areas 358 that are relevant for the user.

The printed e-mail 310 with the expanded functionality may have a message area 220 having a sender message area 231 and a user message area that are configured on the page in any of the ways that have been described with reference to figures 10 to 13.

As mentioned, we also envisage creating or modifying original e-mails, as well as replying to e-mails. Figure 5 schematically illustrates a first method of using the system 100 to generate an original e-mail. A software application has access to a database of pattern data for use with Anoto-type position-determining pattern functional documents. The database may also have user names and identification numbers, which are in use associated with each particular document at the time of printing of the document 120 and which may or may not be printed out with the document. Text or human-readable content is printed at the same time using the same printing routine as the pattern. This database may be on the PC 104 or elsewhere on the network, for example on a local file server, on the server 116 or elsewhere on the Internet.

An already existing, previously designed, document may be accessed from a database of such documents. Alternatively, the document may be designed by the user.

Figure 6 schematically illustrates an e-mail template 190 that may be accessed from a database of digital documents. The template 190 contains a message area 192 within which a user may write a message when a hard copy of the template i s printed. The template also has one or more "To" boxes 194 within which the user may write the e-mail address of the intended recipient of the e-mail. A subject box 196 is provided for a user to write in the subject of the e-mail. Also provided is a send box 1 98.

Optionally, at step 512, a user edits the template. An example of editing is the insertion of the e-mail address of the intended recipients of the e-mail into "To" box 194. As discussed earlier, less than the full e-mail address of the intended recipient need be entered if the processor to which pen-created markings are sent can translate written information uniquely onto an e-mail address in an e-mail address book computer accessible memory.

At step 514 a hard copy 120 of the template 190 is printed using the printer 1 10. The hard copy 120 is a sheet of paper 10 on which is printed the Anoto pattern 12 and the e-mail template 190. The printer 110 may be fed sheets of paper on which the pattern 12 has been pre-printed. Alternatively, the printer 1 10 prints the pattern 12 and the template 140 in a single operation.

The printed e-mail template 120 may then be taken by the user. Normally the user will use the printed e-mail template 190 at a remote location, for example on the train, at home, in an office where they have no access to their normal computer network, at a hotel, etc.

At step 518 the user marks the printed template 190 with the digital pen 20. This will normally include writing a message in the message area 192 and the address of one or more recipients in the "To" box 172 (unless this has been prc-fillcd at step 512).

At step 522 the pen 20 creates an e-mail. The pen does this by monitoring the position of the pen tip relative to the pattern 1 2 on the printed template 120 as the pen is moved in the writing process and storing the pen strokes in the pen's memory 26. When the user has finished writing the e-mail he marks the send box 198. Once send box 198 has been marked the pen 20 knows that the e-mail is complete and the data acquired is ready for transmission. The e-mail address can be interpreted using Optical Character Recognition (OCR) or Intelligent Character Recognition (ICR). The e-mail may be transmitted immediately to a receiver, for example, a receiver that is linked to a PC, or via a piconet link from the pen to a user's mobile telephone. The PC may be a PC associated with the address of the intended recipient of the e-mail, this is shown as route 131 in Figure 4.

Alternatively, the e-mail may be sent to the PC 104 of the user so that the user can perform further editing work. This is shown as route 101 in Figure 3. Instead of immediately transmitting the generated e-mail, another option is that the generated e-mail is stored in the pen memory 26 so that the e- mail may be downloaded into the user' s PC 104 via a docking device.

The handwritten message in the message area 192 may be transmitted to the recipient as a facsimile of what has been written or the handwritten message may be converted into a typeset using OCR or ICR.

It is also possible to reply to the received e-mail by fax. In this case the printed e-mail has a printed box 204 associated with the function of sending a fax. Instead of writing an e-mail address in the To box 194, the user writes a fax number.

The system 100 may also be used to print out 514 a template that has been generated by an application that is u sed for the conventional sending and receiving of e-mails. An example of such an application is Microsoft Outlook™. When using such a template it is necessary for the various functional areas of the template to be designated areas of pattern space and for each template used to be designated a unique identification code. This may be achieved by using a software plug-in.

Figure 7 schematically illustrates a typical e-mail template 140, similar to one that can be accessed from Microsoft Outlook™ (this time for sending an original e-mail instead of replying to one). The Microsoft Outlook™ template 40 is used only for illustrative purposes and it will be appreciated that the invention is not restricted to use with any proprietary system. The Outlook™ template 140 includes a general menu header bar 142 and an icon header bar 143 that have been described with reference to Figure 10. The template has a "To" box 172 and a "Cc" box 174 where a user may enter e-mail addresses for intended recipients of the e-mail. The template also has a subject box 176, and a message box 180 for the user to write a message. When the Outlook™ template 140 is to be used to produce a digital paper document 120, it may not be necessary to designate areas of pattern space to the menu bar 142 if the menu bar 142 is to serve no function when the template 140 is printed for use with a digital pen 20. Alternatively, if the functionality of one or more, or all of, the function icons are to be accessible via a paper printed e-mail pattern may be printed registered with them. Similarly, when the template 140 is going to be used as an original e-mail some of the icons on the icon bar 143 may not serve any function in a printed digital form and if no function is allocated may therefore not be allocated a pattern area. Alternatively, function, and pattern, may be allocated. Examples of such unused functions may include: previous item 171 , next item 173 , help 175. "Move to folder" and "delete" 176 functionality may, or may not, be provided. It is perfectly feasible, and desirable, to configure the e-mail application, for example by the software plug in, so that the printed template 140 does not include the menu bar 142 or icons that are not functional.

The idea of using a printed sheet of paper to be part of a control signal input process to cause a computer to apply a selected one of a plurality of running or installed and runable application programmes to operate on identified data has wider applicability than just use in association with emails.

We see one aspect of the invention being the idea of having a computer with computer-available resources (e.g. memory records such as documents in memory, or software applications that are runnable on the computer processor that can operate on data records or for example, Microsoft Outlook) using the computer to print out a sheet of paper with certain areas of the paper/markings on the paper (probably a Anoto-dotted pattern - but perhaps not, perhaps identified by a different sort of pen technology that knows where the pen is), with those specified areas on the sheet of paper being associated with selecting particular resources, for example application software on the computer, and linking or using the particular resource (e.g. selected software) with either handwritten data written on to the sheet of paper or other pre-existing data existing on the computer. Thus, the sheet of paper may essentially be viewed as a Graphical User Interface (in combination with a position- sensitive pen) so as to enable the user to interact with a computer to select from a plurality of different application software and/or different pre-existing record data and to get them to operate on each other using the pen as an input device.

This is broader than "e-mails". The user has a choice of different functionality software applications, preferably of three, four, five, or more software applications.

Claims

1. A method of replying to an e-mail comprising printing out a received e-mail, handwriting reply markings on the printed e-mail using an electronic digital pen which both physically writes on the printed e-mail and captures digitally the reply markings and their position on the printed e-mail to create a digital version of the reply markings, and sending an e- mail to the sender of the received e-mail having a content derived from both the received e-mail and the digital version of the reply markings.

2. The method of claim 1 comprising creating the digital version of the reply markings in part or completely as a facsimile copy of the handwritten reply markings.

3. The method of either of claims 1 or 2 comprising creating the digital version of the handwritten reply markings such that not all of said, physical, reply markings arc created as facsimile digital versions of the markings and sent to the original sender of the received e-mail.

4. The method of claim 3 wherein at least some of the handwritten markings are interpreted by a computer as functional instructions to the computer, and the computer acting thereupon.

5. The method of any preceding claim comprising making reply markings which comprise content markings which are sent to the sender of the original e-mail and/or functional operation markings that comprise instructions to a computer and which arc (i) sent, or (ii) not sent as visible markings to the sender of the original e-mail.

6. The method of any preceding claim further comprising sending both the received e-mail and the digital version of the reply markings to the sender of the received e-mail and comprising combining the received e-mail and the digital version of the reply markings before they are sent and sending them as a single combined e-mail.

7. The method of any one of claims 1 to 5 comprising sending the changes/reply markings, but not the original e-mail content to the sender of the original e-mail.

8. The method of any preceding claim comprising printing a position-determining pattern paper, in addition to the original e-mail, onto paper in the same printing operation, the pattern enabling the pen to determine its position, thereby enabling the reply markings to be captured digitally by the pen.

9. The method of any one of claims 1 to 7 comprising using a waymarker, or start point, on the printed e-mail to initialise the pen' s position relative to the page prior to making subsequent movements.

10. A method according to any preceding claim comprising linking in a computer or processor memory the identity and/or content of a particular received e-mail that is printed out to digitally captured reply markings made on the specific sheet of paper (or other printed article) upon which the printed e-mail is printed.

11. The method according to claim 10 comprising linking the identity or content by printing an identity code onto the paper/printed article and associating the handwritten markings with the identity code, and also associating the electronic version of the received e-mail with the identity code.

12. The method according to either of claims 10 or 11 comprising printing an identifiable and unique (or hitherto unused/unallocated) pen-readable position-determining pattern onto the paper, to act as the identity code, and using the pattern not only to allow the pen to determine where it is on the pattern, but also to link a subset of positions in an overall larger pattern with a specific received e-mail.

13. A method of editing an existing e-mail held on a computer processor comprising printing out the existing e-mail and writing on the printed e-mail, or otherwise marking the printed e-mail, with a digital pen which both physically marks the printed e-mail and determines its movements over the printed e-mail to create a digital editorial record, and communicating the digital editorial record from the pen to said computer processor and using the digital editorial record to annotate and/or change the existing e-mail.

14. A computer program product, possibly encoded on a machine- readable data carrier, which, when running on a computer processor having access to an existing e-mail including the e-mail address of the sender of the existing e-mail, is adapted to use input signals from a digital pen representative of handwritten amendments, annotations, or markings made superimposed upon and/or adjacent a printed version of said existing e-mail, together with said existing e-mail and sender e-mail address, to create a reply e-mail.

15. A computer program product according to claim 14 which is adapted to send the reply e-mail to the sender's e-mail address.

16. A computer program product according to either of claims 14 or 15 which is adapted to send the reply e-mail, as a draft reply, or as a copy of a reply that has actually been sent to the e-mail address of the sender of the existing e-mail, to a computer processor or display screen of a recipient user who received said existing e-mail.

17. A computer program product according to any one of claims 14 to 16 which is adapted to interpret specific functional input signals from said digital pen as functional instructions to perform specific associated functions on or relating to said existing e-mail when creating said reply e-mail.

18. A computer program product according to claim 17 wherein said functions include: interpreting a handwritten marking or markings on the printed e-mail as instructions to show identified parts of the existing e-mail in differentiated format from other parts of the existing e-mail, and pursuant to such an interpretation differentiating the identified parts in said reply e-mail in predetermined manner.

19. A computer program product according to claim 17 or claim 18 wherein said functions include: interpreting a handwritten marking or markings on the printed e-mail as instructions to send the reply e-mail to all recipients of the said existing e-mail, and pursuant to such a interpretation causing said reply e-mail to be sent to all recipients of said existing e-mail.

20. A computer program product according to any one of claims 17 to 19 wherein said functions include: interpreting a handwritten marking(s) on the printed e-mail as instructions to send the reply e-mail to selected identified other e-mail addresses present in an e-mail address database accessible by said computer processor, and pursuant to such interpretation obtaining said selected identified e-mail address from said e-mail address database and causing said reply e-mail to be sent to said obtained e-mail addresses.

21. A computer program product according to any one of claims 17 to 20 wherein said functions include: interpreting a predetermined marking on or adjacent a word in said printed e-mail as intending the deletion of said word and pursuant to such an interpretation said program causing said word to be deleted in said reply e-mail.

22. A computer program product according to any one of claims 17 to 21 wherein said functions include: interpreting a predetermined marking on said printed e-mail as intending the attachment of a specified file accessible to said computer processor, and pursuant to such an assessment attaching said specified file to said reply e-mail.

23. A computer program product according to any one of claims 17 to 22 wherein said functions include: interpreting marking(s) on said printed e-mail as being intended to be reproduced as a faithful facsimile(s) of the marking(s), and pursuant to such an assessment either:

(i) adding such markings to said existing e-mail before or at the time of sending said reply e-mail to said reply e-mail address or to a user as a draft e-mail, or

(ii) sending both said facsimile marks and said existing e-mail to a recipient to be combined al a computer processor of said recipient, or

(iii) sending said facsimile marks as an e-mail to a computer processor of a recipient which already has said existing e-mail to enable them to be combined at said recipient' s computer processor.

24. A computer program product according to any one of claims 17 to 23 wherein said functions include: interpreting specified marking(s) as instructions to flag said reply e-mail as being of high importance, and pursuant to such an interpretation being made flagging said reply e-mail accordingly.

25. A computer program product according to any one of claims 17 to 24 wherein said functions include: interpreting specified marking(s) as instructions to move said existing e-mail to an identified folder, and said program causing said existing e-mail to be moved to said identified folder pursuant to such an identification being made.

26. A computer program product according to any one of claims 17 to 25 wherein said functions include: interpreting marking(s) as intended to be operated upon by a character recognition program to convert handwritten markings onto word processable text, and pursuant to such an identification being made operating on said markings with such a character recognition program and converting said markings into word processable text in said reply e-mail.

27. A computer program product according to any one of claims 14 to 26 wherein said computer product program resides on a user' s desktop computer or laptop or general purpose computing device, or on a user' s server, such as an e-mail server.

28. A computer program product according to any one of claims 14 to 26 wherein said computer program product resides in said digital pen.

29. An e-mail reply system comprising a computer processor having access to computer memory, and a pen adapted to write on paper sheet and adapted to determine its position on the sheet; the computer processor being adapted lo access e-mail dala relating to a received e-mail and print out the received e-mail from a digital printer to create a printed e-mail, and to receive marking data from the pen indicative of the markings the pen makes on the printed e-mail, the processor being adapted to correlate marking data with e-mail data and to prepare and/or send a reply e-mail to the sender of the received e-mail, the reply e-mail including both marking data and received e-mail data, and the processor being adapted to determine from the received e-mail data the e-mail address to which the reply e-mail is to be sent.

30. A pen having marking capturing means adapted to capture handwritten markings in digital form, and also having a computer processor adapted to receive said digital form of said handwritten markings, and having a computer memory adapted to hold an electronic version of a received e-mail, including the e-mail address of the sender of the received e-mail, and a computer program installed on said processor, or held in computer memory of said pen and installable on said processor, said computer program being adapted to convert handwritten markings on a printed e-mail, that is equivalent to an e-mail held in the pen' s computer memory, into a digital version of said handwritten markings and to combine said digital version of said markings with said electronic version of said e-mail, or otherwise associate them, so as to enable said pen to send both the received e-mail and the digital version of the reply markings to the e-mail address of the sender of the received e-mail.

31. A printed page or document having printed on it the content of an e-mail and a position-determining pattern capable of being read by a digital pen in order to determine the movement of the pen when it writes on the pattern.

32. A printed page according to claim 31 wherein the content and the pattern have been printed in the same print operation from the same printer, upon demand from a computer user.

33. A printed page or pages according to either of claims 31 or 32 in combination with a digital pen adapted not only to write on the paper but also to record digitally handwritten markings made with the pen.

34. A method of creating and/or sending an e-mail by writing markings on a printed sheet using a pen which is able to determine its position on the sheet, and using the marking of:

(i) predetermined markings, and/or (ii) marking the sheet at predetermined positions on the sheet, to select a selected one of an associated computer resource from a plurality of associated computer resources potentially available to an e-mail creating and/or sending processor, the processor using the selected resource in the creation of the e-mail and/or in the sending of the e-mail.

35. The method of claim 34 wherein said computer resources comprise one or more of, preferably a plurality of, computer programs or software routines which arc selected and used, for a particular e-mail , using the pen.

36. The method of either of claims 34 or 35 wherein said computer resources comprise a plurality of e-mail addresses, the pen being used to select an e-mail address or e-mail addresses to which the e-mail is to be sent.

37. The method of any one of claims 34 to 36 wherein said computer resources comprise one or more of, preferably a plurality of, pre-existing computer records susceptible for incorporation into the e-mail and/or attachment to the e-mail.

38. The method of any one of claims 34 to 37 wherein said computer resources comprise one or more url addresses susceptible to being included in the e-mail as links to the site of the url.

39. The method of any one of claims 34 to 38 wherein said computer resources comprise computer-implemented tools to enable a user to construct an e-mail, said tools being accessed/used by a user marking the printed sheet and a shcct-to-tool conversion program.

40. The method of any one of claims 34 to 39 comprising printing the printed sheet with a pen-position determining pattern.

41. The method of claim 40 comprising making an association between a particular region in the pattern with a particular computer resource at the time of printing of the sheet.

42. The method of claim 41 comprising coding different positions in the pattern for different computer resources once an association between pattern and resource has been created in a computer processor.

43. The method of any one of claims 34 to 42 comprising selecting a subset of computer resource-enabling printed markings to be printed on the sheet and to be actually available using the sheet as a GUI for the computer, the subset to be printed being selected from a larger group of potentially available computer resources.

44. A computer processor having installed in it a paper e-mail instruction translation program adapted to convert handwritten markings made by a user on paper using a position-determining pen into computer commands to cause the creation of an electronic version of the paper e-mail, the translation program being adapted to interpret certain handwritten markings dependent upon their position or other recognition criteria as instructions to perform associated acts to use available computer resources in the creation of the electronic version of the paper e-mail, at least some of the markings being interpretable as a selection of specific resources to be used in the creation of the electronic version of the paper e- mail.

45. A processor according to claim 44 wherein the processor is arranged to access computer resources from computer memory available thereto.

46. A processor according to claim 45 wherein the resources include a program to:

(i) create facsimile representations of handwritten markings in the electronic version of the e-mail; and/or (ii) convert handwritten markings into typeset equivalent words; and/or (iii) to cause the e-mail to be sent to an individual e-mail address; and/or

(iv) cause one or more other application programs to be run to provide additional functionality controlled or, initiated by, the handwritten markings; and/or (v) retrieve data from computer memory and to use retrieved data in the electronic version of the e-mail.

47. A method of replying to an existing e-mail or of creating or editing an existing draft e-mail according to claim 1 or claim 13 or claim 34, the method comprising using digital paper printed with a pattern adapted to enable a digital pen to identify its position in the pattern, and a digital pen adapted to identify its position in the pattern, the method further comprising printing an electronic version of a received e-mail or draft c- mail along with said pattern onto sheet material to create a printed version of the received or draft e-mail, and editing the printed version of the received or draft e-mail using the digital pen, the digital pen capturing digitally editing markings made by the pen, and using the captured digital editing markings, along with the electronic version of the received or draft e-mail to create an electronic reply or amended e-mail; and associating the printed pattern printed onto the sheet material with the specific electronic version of the received or draft e-mail in computer memory and using the pen to recognise the pattern on the printed sheet material and associating the captured digital editing marking with the specific electronic version of the received e-mail by using the pattern as a link.

48. A computer program product, possibly encoded on a machine- readable data carrier, which, when running on a computer processor, is adapted to use input signals from a digital pen representative of handwritten amendments, annotations, or markings made upon a printed sheet in the preparation of an electronic e-mail and which computer program product is adapted to interpret a plurality of different specific functional input signals from said digital pen as a corresponding plurality of functional instructions to perform different specific associated functions when creating said e-mail, said plurality of different specific functional input signals being associated with a plurality of different specific positions on the printed sheet.

49. A computer program product according to claim 48 wherein said functions include at least one of, and optionally any combination of, including all of:

(i) interpreting a handwritten marking or markings on the printed sheet as instructions to show identified parts of an existing e-mail or word- proccssablc document printed on the sheet in differentiated format from other parts of the existing e-mail or word-processable document, and pursuant to such an interpretation differentiating the identified parts in said electronic e-mail in predetermined manner; (ii) interpreting a handwritten marking(s) on the printed sheet as instructions to send an e-mail to selected identified other e-mail addresses present in an e-mail address database accessible by said computer processor, and pursuant to such interpretation obtaining said selected identified e-mail address from said e-mail address database and causing said e-mail to be sent to said obtained e-mail addresses;

(iii) interpreting a predetermined marking on or adjacent a word in said printed sheet as intending the deletion of said word and pursuant to such an interpretation said program causing said word to be deleted in said e-mail;

(iv) interpreting a predetermined marking on said printed e-mail as intending the attachment of a specified file accessible to said compu ter processor, and pursuant to such an assessment attaching said specified file to said e-mail;

(v) interpreting specified marking(s) as instructions to flag said reply e-mail as being of high importance, and pursuant to such an interpretation being made flagging said e-mail accordingly;

(vi) interpreting specified marking(s) as instructions to move said e-mail to an identified folder, and said program causing said e-mail to be moved to said identified folder pursuant to such an identification being made;

(vii) interpreting marking(s) as intended to be operated upon by a character recognition program to convert handwritten markings onto word processable text, and pursuant to such an identification being made operating on said markings with such a character recognition program and converting said markings into word processable text in said e-mail.

50. A method of replying to an e-mail comprising using printing means to print a received e-mail, marking a printed e-mail using electronic marking means and digital capture at least one mark made by the marking means by capture means, the capture means also being arranged to capture the position of the at least one mark, the capture means communicating with processing means arranged to process information captured by the capture means to produce a digital version of the printed e-mail, sending an e-mail to a sender of the received e-mail having content derived from both the received e-mail and the digital version of the printed e-mail.

51. A method of replying to an e-mail comprising printing out a received e-mail on paper with a position determining pattern of dots printed thereupon, handwriting reply markings on the printed e-mail using an electronic digital pen, comprising an ink source, an infra-red emitting LED and an infra-red sensor, the ink source being arranged to physically writes on the printed e-mail and the infra-red sensor being arranged to capture digitally the reply markings and capture their position on the printed e-mail, using the position determining pattern, to create a digital version of the reply markings, and sending an e-mail to the sender of the received e- mail, having a content derived from both the received e-mail and the digital version of the reply markings.

52. An e-mail reply system comprising processor means for processing data having access to memory means for storing data, and marking means for writing on a paper sheet and adapted to determine its position on the sheet; the processing means being adapted to access e-mail data relating to a received e-mail and printing means being arranged to print out the received e-mail to create a printed e-mail, and to receive marking data from the marking means indicative of the markings the pen makes on the printed e-mail, the processing means being adapted to correlate marking data with e-mail data and to prepare and/or send a reply e-mail to the sender of the received e-mail, the reply e-mail including both marking data and received e-mail data, and the processing means being adapted to determine from the received e-mail data the e-mail address to which the reply e-mail is to be sent.

53. A method of creating and/or sending an e-mail by writing markings on a printed sheet using marking means which is able to determine its position on the sheet, and using the marking of:

(i) predetermined markings, and/or (ii) marking the sheet at predetermined positions on the sheet, to select a selected one of an associated computer resource from a plurality of associated computer resources potentially available to an e-mail creating and/or sending processing means, the processing means using the selected resource in the creation of the e-mail and/or in the sending of the e-mail.

54. A method of using a computer provided with computer-available resources comprising using the computer to print out a sheet of paper with certain areas of the paper associated with the selection of particular application software on the computer, and using the particular selected software to act upon either handwritten data written on to the sheet of paper or other pre-existing record data existing accessible to the computer, the sheet of paper comprising in combination with a position-sensitive pen, a user command input mechanism so as to enable the user to interact with the computer to select from the plurality of different application software and/or different pre-existing record data in order to cause selected application software to interact with associated record data.