|Publication number||WO1998032109 A1|
|Publication date||23 Jul 1998|
|Filing date||21 Jan 1998|
|Priority date||21 Jan 1997|
|Also published as||EP0996944A1|
|Publication number||PCT/1998/39, PCT/NL/1998/000039, PCT/NL/1998/00039, PCT/NL/98/000039, PCT/NL/98/00039, PCT/NL1998/000039, PCT/NL1998/00039, PCT/NL1998000039, PCT/NL199800039, PCT/NL98/000039, PCT/NL98/00039, PCT/NL98000039, PCT/NL9800039, WO 1998/032109 A1, WO 1998032109 A1, WO 1998032109A1, WO 9832109 A1, WO 9832109A1, WO-A1-1998032109, WO-A1-9832109, WO1998/032109A1, WO1998032109 A1, WO1998032109A1, WO9832109 A1, WO9832109A1|
|Inventors||Lange Teunis Roelof Pieter De|
|Applicant||B.V. Uitgeverij En Boekhandel W.J. Thieme & Cie.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (8), Referenced by (31), Classifications (6), Legal Events (10)|
|External Links: Patentscope, Espacenet|
The present invention relates to a self-tuition apparatus. Various types of self-tuition apparatus are known in practice. They can be divided into two classes. The first class comprises apparatus which can be used to put open-ended questions to a student on a screen. The student answers the open-ended question with the aid of suitable input means, for example a keyboard. The apparatus comprises a memory, in which are stored not only the open-ended questions but also the associ- ated correct answers. After the user has input his/her answer with the aid of the input means, the apparatus automatically compares the input answer with the correct answer belonging to the relevant open-ended question. If the two answers correspond, the apparatus generates a message for the user in order to notify him/her of this fact, and the appar- atus then presents a following open-ended question.
Another class of self-tuition apparatus relates to apparatus which can be used to present multiple-choice questions to a student. These multiple-choice questions are provided with various, for example four, multiple-choice answers. With the aid of the input means, the student selects the answer which he/she considers to be correct. In addition to the multiple-choice questions and the associated multiple- choice answers, the apparatus also stores an indication of which of the multiple-choice answers is the correct answer. The apparatus compares the answer input by the student with the correct answer and then noti- fies the student of whether he/she has answered the question correctly or incorrectly. Then, the apparatus automatically presents a following multiple-choice question with associated multiple-choice answers.
The self-tuition apparatus of both classes have the drawback that they are primarily suitable for testing the knowledge of a student, and are not very suitable as an aid for stimulating the learning process.
The object of the invention is to provide a self-tuition apparatus which permits a new form of question-answer interaction of a strongly formative nature, i.e. in which questions put by the apparatus form part of the learning process and serve to a lesser extent as a test of the student's knowledge.
To achieve this object, the present invention provides a self- tuition apparatus comprising input means for a user to input data, output means for outputting data to the user, memory means at least pro- . vided with an open-ended question memory containing open-ended questions and associated correct answers and a multiple-choice question memory containing multiple-choice questions, which are linked to predetermined open-ended questions, and associated multiple-choice answers, and a processor, which is connected to the input means, the output means and the memory means and is provided at least with first and second processing means, which first processing means are designed to: a) control a first user interface in the form of the display of at least one open-ended question via the output means; b) read a first answer input by the user via the input means; c) compare the input first answer with the correct answer; and if the first answer read at step b) corresponds to the correct answer d) create a first message to the user and continue with step a) until all the open-ended questions have been answered, and the second processing means being activated automatically, if the first answer read at step b) does not correspond to the correct answer, so as to carry out the following steps: e) controlling a second user interface in the form of the display of at least one multiple-choice question with associated multiple- choice answers via the output means; f) reading a second answer input by the user via the input means; g) comparing the second answer read at step f) with the correct answer; h) creating a second message to the user and continuing with step a) until all the open-ended questions have been answered. A self-tuition apparatus of this kind automatically decides the level at which questions are put to a student. If a student has learnt the study material well, he/she will answer most of the open-ended ques- tions correctly. If he/she does not know the answer to an open-ended question, the first processing means of the apparatus register this fact during the comparison step c) . The apparatus then automatically activates the second processing means and presents the student with a multiple-choice question with various multiple-choice answers . The relevant multiple-choice question corresponds to the open-ended question which the student answers incorrectly.
The apparatus thus automatically controls the various user interfaces and ensures that the various part-memories of the memory means are actuated.
The new question-answer interaction which is presented here is intended specifically to be used in the context of computer-aided self- tuition. Use can advantageously be made here of modern technological developments. It is becoming every more possible to provide one computer per student, on which use can be made of multimedia presentation facilities, such as a sound card, CD-ROM, and a very large memory capacity for information relating to individual students . The computers can be incorporated into a network. Preferably, authors of the questions, i.e. both the open-ended questions and the multiple-choice questions, operate in a context in which they themselves provide material for computer-aided self-tuition. To do this, the authors may use, for example, a windows word processor and specific graphical or linguistic tools. Preferred embodiments of the apparatus according to the invention are defined in the subclaims.
The invention will be explained below with reference to a number of figures, which are intended only to illustrate the invention and not to limit the scope thereof. In the figures: Figure 1 shows a structure of a self-tuition apparatus according to the invention;
Figure 2 shows a flow chart of one of the operating possibilities of the apparatus in accordance with Figure 1.
A self-tuition apparatus according to the invention comprises a central processor 200, which is denoted in Figure 1 by CPU. The central processor 200 is connected to suitable input means 201 and suitable output means 202. The input means 201 may relate to one or more desired options from the following possibilities: a keyboard, a mouse, a touch screen, as well as other known input means. The output means 202 relate to one or more options from output means which are known per se, such as a monitor, a display screen, audio means, etc.
The self-tuition apparatus furthermore comprises a memory 203, which is likewise connected to the central processor 200. In its most basic form, the memory 203 comprises at least an open-ended question memory 204, in which open-ended questions and at least the associated correct answers are stored. Optionally, it is also possible to store expected incorrect answers here. This will be explain- ed further below. The questions and answers may relate to written text, but also to pictures, for example for children who are not yet able to read.
The basic parts of the memory 203 also include a multiple-choice question memory 205 with associated multiple-choice answers, a memory 206 which contains an algorithm for absolute comparison of the input answer with one or more correct answers, and a memory 211 which comprises hints/messages for the user.
In further, more detailed embodiments of the apparatus according to the invention, the memory 203 also comprises a memory 207, which contains an algorithm for absolute comparison of an answer input by a student with one or more expected incorrect answers, a memory 208 which contains an algorithm for recognizing errors which can be ignored, a memory 209 which contains an algorithm for relative comparison of the answer input by a student with the correct answer, and a memory 210 which is provided with an algorithm for relative comparison of the answer input by the student with expected incorrect answers.
The following terms are of importance to the explanation of the principle of the question-answer interaction: - correct answer = correct response to the question asked; expected error = response by the student which corresponds to incorrect reasoning expected by the author of the questions; error of detail = response by the student which virtually corresponds to a correct answer or an expected incorrect answer; error = response by the student which does not correspond to one of the preceding possibilities.
Figure 2 shows a flow chart to illustrate the operation of a possible, very detailed functioning of the apparatus according to the invention. In step 101, the apparatus is started up, for example by the student .
The apparatus responds by presenting an open-ended question from memory 204 via the output means 202, step 102.
In step 103, the apparatus waits until the student has input an answer .
Once the student has input an answer, the apparatus compares the answer input with the correct answer. To do this, the apparatus employs the correct answers which are stored in the memory 204, together with the associated open-ended questions, and the algorithm stored in the memory 206 for absolute comparison with correct answers. Comparison algorithms of this kind are known to the person skilled in the art and do not need to be explained further here. If the answer input by the student is correct, a positive message is given to the student in step 108. Then, the program returns to step 102 and a following open-ended question, if available, is presented to the student via the output means 202.
However, if step 104 finds that the student has not input a correct answer with the aid of the input means 201, the program checks in step 105 whether the student has made an error of detail with respect to the correct answer. To do this, use can be made of known algorithms for graphical or linguistic recognition. The algorithm for graphical recognition is adapted to fit the technical features of the available input means and output means of which it is strongly dependent. The algorithm for linguistic recognition is fully integrated in the system and is therefore described in more detail here. The input of a linguistic answer can be made via a keyboard or via a microphone with a system for phonological conversion to text. The system is very suitable for the latest technological advanced form of input as the algorithm for lin- guistic recognition can filter features which are not significant. Such an algorithm for linguistic recognition filters the answer input by the student, checking for example for double spaces, spaces at the end of the answer, and the like. Then, the answer input by the student is compared in absolute terms with the correct answer with regard to the fol- lowing points: upper-case letters and lower-case letters, punctuation marks and diacritical marks. If the comparison shows that the student has only made an error of detail, the program proceeds to step 106. In step 106 it is established whether the student has made what is known as an error of detail which can be ignored. This may relate, for example, to the incorrect use of upper-case letters and lower-case letters. If the program establishes that the error of detail made can in fact be ignored, the program continues its route to step 108, which has already been explained above. If the error of detail made is not an error of detail which can be ignored, the program proceeds to step 107, in which it provides the student, via the output means 202, with messages which can be used to help with correcting the error of detail. Then, the program returns to step 102.
However, if it is found from the algorithm for graphical or linguistic recognition used that the student has not made an error of detail (step 105), the program proceeds with step 109. In step 109, the program makes use of an algorithm for absolute comparison of the answer input by the student with one or more expected incorrect answers. As stated earlier, an algorithm of this kind is stored in the memory 207. Incorrect answers of this kind are based, for example, on incorrect reasoning which is nevertheless obvious and is expected by the author of the questions. If the use of the algorithm for absolute comparison with the one or more expected incorrect answers shows that the student has in fact input an expected incorrect answer using the input means 201, the apparatus notifies the student of this via the output means 202, via step 110, in which the error of reasoning is pointed out to the student. The program then gives the student a hint on the correct answer, step 113.
In step 114, the program waits until the student has input an answer, after having received a hint about it.
After the student has input an answer, the program checks whether the input answer is correct. This is performed in step 115, which is identical to step 104. If step 115 shows that the student has input a correct answer, a positive message follows, step 116. After step 116, the program returns to step 102, if there are still open-ended questions left to be asked. However, if the program establishes in step 115 that a correct answer has still not been given, the program proceeds to step 124, in which a multiple-choice question with various multiple-choice answers is presented to the student instead of an open-ended question. This step 124 will be explained below. As an alternative, step 117, which is ex- plained further below, can also follow here instead of step 124.
If step 109 shows that the answer input by the student is not identical to an expected incorrect answer, the program checks in step 111 whether the difference between the answer input by the student and one or more expected incorrect answers is merely an error of detail. To do this, use can be made of the algorithm for absolute comparison with one or more expected incorrect answers from memory 207.
If it is established that the error is indeed an error of detail, the program checks in step 112 whether this is an error of detail which can be ignored. Step 112 is comparable to step 106, except that the check here is for a difference from an expected incorrect answer, and not from the correct answer. If this is in fact an error of detail which can be ignored, the program proceeds with step 110, which has already been explained above. If it is not an error of detail which can be ignored, the program proceeds with step 117, which step is also carried out if it is established in step 111 that this is not an error of detail.
It should also be noted that in steps 106 and 112 for establish- ing an error of detail which can be ignored use is made of an algorithm for recognizing errors which can be ignored, which algorithm is stored here in a separate memory 208.
In step 117, the program carries out a relative comparison between the answer input by the student and the correct answer. To do this, use is made of an algorithm for relative comparison with correct answers, which algorithm is stored in memory 209. Algorithms of this kind are known in practice. Linguistic error recognition algorithms of this kind work at letter level. They have been used since approximately 1980, for example in educational programs for typing and word processing of the types QUERTY, Vingervlug, WP-Trainer, AZtekst and TypPlus. Error recognition algorithms of this kind establish very accurately whether the student has input too many letters via the input means 201, whether there are letters missing, and other types of discrepancies. The algorithms provide a pattern which can be used to present the spelling cor- rection and to assess the relationship between the answer input by the student and the correct answer. This assessment may take the form, for example, of a number.
It is then established in step 118 whether this number, which is a measure of the relative error observed, is less than or equal to a predetermined threshold value.
If so, the program continues the procedure by giving a hint as to the correct answer via the output means 202, step 119. In step 120, the program then waits until the student has input an answer on the basis of the hint given. Once this has been done, the program proceeds with step 115, which has already been explained above.
If it is found in step 118 that the relative error is greater than the predetermined threshold, the program continues with step 121, in which the program carries out a relative comparison with one or more expected incorrect answers. To do this, use is made of the algorithm for relative comparison with incorrect expected answers which is stored in memory 210. This error recognition algorithm may be of similar type to the error recognition algorithm which is stored in memory 209 and acts at letter level. In most cases, the algorithms from the memories 209 and 210 will be identical and only have to be stored once.
In step 122, the program checks whether the calculated relative error generated by step 121 is less than or equal to a predetermined threshold value. If so, the program continues with step 123, in which the student is given a hint about the correct answer. Then, the program proceeds to step 120, which has already been explained above.
If the program establishes that the relative error in step 122 is greater than the predetermined threshold value, the program continues with step 124, in which it presents the student with a multiple-choice question with various multiple-choice answers, via the output means 202. A presentation of this kind may take the usual form, in which, for example, one multiple-choice question with associated multiple-choice answers is shown to the user on a monitor.
In step 125, the program then waits until the user has input an answer with the aid of the input means 201. As is known, the answer here consists in selecting one of the multiple-choice answers presented.
In step 126, the program checks automatically whether the answer input by the student is correct. To do this, use can be made, for example, of an algorithm for absolute comparison with correct answers, which is stored in the memory 206.
If the answer is correct, a positive message follows via the output means 202, step 127. However, if the answer input by the student is incorrect, there follows a negative message via the output means 202, in step 128. After step 127, the program returns to step 102. After step 128, the program preferably returns to step 125 for as many times as necessary for the student to select the correct answer.
It will be clear that the flow diagram shown in Figure 2 is a detailed embodiment of the functioning of the apparatus according to the invention. In its most simple form, the steps of the program relating to checking for errors of detail, steps 105-107, the steps relating to checking the answer input by the student with respect to one or more expected incorrect answers, steps 109-116, and the steps relating to car- rying out a relative comparison of the answer input by the student with the correct answer or with one or more expected incorrect answers, steps 117-123, are omitted. Thus in its most simple form, step 124, in which a multiple-choice question with various multiple-choice answers is presented to the student, directly follows step 104, if it is established that the answer input by the student is incorrect.
Checking for errors of detail, comparing with one or more expected incorrect answers and carrying out relative comparisons with the correct answer or with one or more expected incorrect answers relates to options which refine the apparatus. With regard to the self-tuition apparatus presented above, the author of the open-ended questions and the multiple-choice questions, the student and any instructor have different roles.
The author thinks up an open-ended question and an associated multiple-choice question, as well as one or more correct answers, ex- pected incorrect answers, for which a sensible explanation can be given, and multiple-choice answers.
The student is initially presented with these questions as open- ended question via the output means 202. It is known that open-ended questions test the knowledge of students at the most difficult level. The apparatus helps the student in all kinds of different ways to give the correct answer, for example using (linguistic) spelling tips, indications of punctuation marks, automatic filling-in of non-core words, (graphically) showing contours, masking background, etc. Only if the student does not succeed in giving the correct answer at the level of the open-ended question (step 104) is the question presented as a multiple-choice question with various multiple-choice answers. With the multiple-choice question, the student has the option, for example, of selecting an answer as many times as the number of multiple-choice answers presented. It should be noted that whatever route the student follows, he/she will always ultimately be aware of the correct answer.
The apparatus of the present invention is preferably provided with means for recording the input answers and the route which the stu- dent took for each question. It is possible in a simple manner to store in the memory 203 at what level and via which route the student gave the correct answer. The apparatus is preferably provided with means for compiling graphs or the like on the basis of the above factors, which graphs can be called up by the instructor and can be shown on the output means 202. The routes followed by the student in responding to the questions can, for example, be "played back" again in the presence of the instructor, so that the instructor is helped in his/her discussion of the teaching material with the student. The apparatus according to the invention has various advantages.
For example, the author only has to take into account the answers which he/she finds useful in furthering the teaching process. He/she only has to compile correct answers or answers which are incorrect for an obvious reason. Moreover, in the multiple-choice questions the author does not have to invent extra diversionary answers to avoid the guessing effect. This is because the student is always asked at the level of the open-ended questions to input the correct answer.
Furthermore, each route travelled by the student for each question leads to a meaningful answer. The questions asked by the apparatus form part of the teaching material and can be assessed. There is less need for tests.
The interaction between the student and the apparatus has both a high maximum result, namely a correct answer immediately after an open- ended question has been asked, as well as a positive minimum result, namely finding the correct answer via the multiple-choice questions. The student learns even from this. By recording the routes followed by the student, an instructor can establish where the student has problems with the teaching material. By calling up the relevant routes, the student can offer help in solving these problems. The apparatus discussed thus forms a teaching aid, since it is able to switch automatically between various layers of questions and associated answers in interaction with a student.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|WO1995008810A1 *||20 Sep 1994||30 Mar 1995||Empower Information Technology Pty Limited||Auditing system|
|US4958284 *||6 Dec 1988||18 Sep 1990||Npd Group, Inc.||Open ended question analysis system and method|
|US5180309 *||4 Dec 1990||19 Jan 1993||United States Of America As Represented By The Secretary Of The Navy||Automated answer evaluation and scoring system and method|
|US5421730 *||27 Nov 1991||6 Jun 1995||National Education Training Group, Inc.||Interactive learning system providing user feedback|
|US5577919 *||25 Jul 1995||26 Nov 1996||Collins; Deborah L.||Method and apparatus for automated learning and performance evaluation|
|US5590057 *||20 Dec 1993||31 Dec 1996||Atlantic Richfield Company||Training and certification system and method|
|US5616033 *||3 Aug 1994||1 Apr 1997||Kerwin; Patrick A.||Speed learning system computer based training|
|US5618182 *||30 Sep 1994||8 Apr 1997||Thomas; C. Douglass||Method and apparatus for improving performance on multiple-choice exams|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|WO2000038132A1 *||8 Feb 1999||29 Jun 2000||Accenture Properties (2) B.V.||A system, method and article of manufacture for a simulation enabled focused feedback tutorial system|
|WO2000038135A1 *||8 Feb 1999||29 Jun 2000||Accenture Properties (2) B.V.||A system, method and article of manufacture for a runtime program regression analysis tool for a simulation engine|
|WO2000038144A1 *||8 Feb 1999||29 Jun 2000||Accenture Properties (B) B.V.||A system, method and article of manufacture for a goal based educational system with support for dynamic tailored feedback|
|WO2000038148A1 *||8 Feb 1999||29 Jun 2000||Accenture Properties (2) B.V.||A system, method and article of manufacture for a goal based educational system with support for dynamic personality feedback|
|US6493690||10 Feb 2000||10 Dec 2002||Accenture||Goal based educational system with personalized coaching|
|US6542880||22 Dec 1998||1 Apr 2003||Indeliq, Inc.||System, method and article of manufacture for a goal based system utilizing a table based architecture|
|US6549893||31 Jul 2000||15 Apr 2003||Indeliq, Inc.||System, method and article of manufacture for a goal based system utilizing a time based model|
|US6611822||5 May 1999||26 Aug 2003||Ac Properties B.V.||System method and article of manufacture for creating collaborative application sharing|
|US6658398||10 Feb 2000||2 Dec 2003||Indeliq, Inc.||Goal based educational system utilizing a remediation object|
|US6745170||22 Jun 2001||1 Jun 2004||Indeliq, Inc.||Goal based educational system with support for dynamic characteristic tuning|
|US6782374||22 Dec 1998||24 Aug 2004||Accenture Global Services Gmbh||System, method and article of manufacturing for a runtime program analysis tool for a simulation engine|
|US6970858||8 Feb 1999||29 Nov 2005||Accenture, Llp||Goal based system utilizing an activity table|
|US6993513||22 Aug 2001||31 Jan 2006||Indeliq, Inc.||Interactive simulations utilizing a remote knowledge base|
|US7054848||8 Feb 1999||30 May 2006||Accenture, Llp||Goal based system utilizing a time based model|
|US7065512||8 Feb 1999||20 Jun 2006||Accenture, Llp||Dynamic toolbar in a tutorial system|
|US7065513||8 Feb 1999||20 Jun 2006||Accenture, Llp||Simulation enabled feedback system|
|US7089222||8 Feb 1999||8 Aug 2006||Accenture, Llp||Goal based system tailored to the characteristics of a particular user|
|US7117189||8 Feb 1999||3 Oct 2006||Accenture, Llp||Simulation system for a simulation engine with a help website and processing engine|
|US7152092||5 Nov 2001||19 Dec 2006||Indeliq, Inc.||Creating chat rooms with multiple roles for multiple participants|
|US7156665||8 Feb 1999||2 Jan 2007||Accenture, Llp||Goal based educational system with support for dynamic tailored feedback|
|US7194444||8 Feb 1999||20 Mar 2007||Indeliq, Inc.||Goal based flow of a control presentation system|
|US7280991||5 May 2000||9 Oct 2007||Indeliq, Inc.||Creating collaborative simulations for creating collaborative simulations with multiple roles for a single student|
|US7386524||22 Jun 2001||10 Jun 2008||Accenture Global Services Gmbh||Simulation enabled focused feedback tutorial system|
|US7428518||8 Feb 1999||23 Sep 2008||Accenture Global Services Gmbh||Simulation enabled accounting tutorial system|
|US7433852||8 Feb 1999||7 Oct 2008||Accenture Global Services Gmbh||Runtime program regression analysis tool for a simulation engine|
|US7536363||8 Feb 1999||19 May 2009||Accenture Global Services Gmbh||Goal based system, utilizing a table based architecture|
|US7621748||16 May 2007||24 Nov 2009||Accenture Global Services Gmbh||Computer enabled training of a user to validate assumptions|
|US7660778||8 Feb 1999||9 Feb 2010||Accenture Global Services Gmbh||Runtime program analysis tool for a simulation engine|
|US7856410||22 Jun 2001||21 Dec 2010||Accenture Global Services Limited||Simulation enabled retail management tutorial system|
|US9530329||9 Apr 2015||27 Dec 2016||Laurence RUDOLPH||System and method for conducting multi-layer user selectable electronic testing|
|US9792829||14 Nov 2016||17 Oct 2017||Laurence RUDOLPH||System and method for conducting multi-layer user selectable electronic testing|
|International Classification||G09B7/06, G09B7/04|
|Cooperative Classification||G09B7/06, G09B7/04|
|European Classification||G09B7/04, G09B7/06|
|23 Jul 1998||AK||Designated states|
Kind code of ref document: A1
Designated state(s): AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GE GH GM GW HU ID IL IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT UA UG US UZ VN YU ZW
|23 Jul 1998||AL||Designated countries for regional patents|
Kind code of ref document: A1
Designated state(s): GH GM KE LS MW SD SZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN ML MR NE SN TD TG
|20 Aug 1998||DFPE||Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)|
|23 Dec 1998||121||Ep: the epo has been informed by wipo that ep was designated in this application|
|9 Jul 1999||WWE||Wipo information: entry into national phase|
Ref document number: 1998900775
Country of ref document: EP
|18 Oct 1999||WWE||Wipo information: entry into national phase|
Ref document number: 09341924
Country of ref document: US
|25 Nov 1999||REG||Reference to national code|
Ref country code: DE
Ref legal event code: 8642
|3 Mar 2000||NENP||Non-entry into the national phase in:|
Ref country code: JP
Ref document number: 1998534171
Format of ref document f/p: F
|3 May 2000||WWP||Wipo information: published in national office|
Ref document number: 1998900775
Country of ref document: EP
|1 Aug 2000||WWW||Wipo information: withdrawn in national office|
Ref document number: 1998900775
Country of ref document: EP