redictingMechanicalReasoningAbility(ReasoningwiththeNavy)预测机械推理能力(与海军的推理)课件

1、AutoTutor: An Intelligent Tutoring System with Mixed Initiative DialogArt GraesserUniversity of MemphisDepartment of Psychology & the Institute for Intelligent SystemsSupported on grants from the NSF, ONR, ARI, IDA, IES, US Census Bureau, and CHI SystemsInterdisciplinaryApproachComputer SciencePsych

2、ologyComputational LinguisticsEducationOverviewBrief comments on my research on question asking and answeringPrimary focus is on AutoTutor - a collaborative reasoning and question answering systemOverview of my Research on QuestionsPsychological ModelsQuestion asking (PREG, ONR, NSF, ARI)Question an

3、swering (QUEST, ONR)Computer ArtifactsTutor (AutoTutor, Why/AutoTutor, Think like a commander, NSF, ONR, ARI, CHI Systems)Survey question critiquer (QUAID, US Census, NSF)Point & Query software (P&Q, ONR)Query-based information retrieval (HURA Advisor, IDA)AutoTutor Collaborative reasoning and quest

4、ion answering in tutorial dialogThink Like a Commander Vignettes1 Trouble in McLouth2 Save the Shrine3 The Recon Fight4 A Shift In Forces5 The Attack Begins6 The Bigger Picture7 Looking Deep8 Before the Attack9 Meanwhile Back at the RanchThemes:Keep Focus on Mission? Highers Intent?Model a Thinking

5、Enemy?Consider Effects of Terrain?Use All Assets Available?Consider Timing?See the Bigger Picture?Visualize the BattlefieldAccurately? - Realistic Space-Time ForecastDynamically? - Entities Change Over TimeProactively? - What Can I Make Enemy DoConsider Contingencies and Remain Flexible?What does Au

6、toTutor do?Asks questions and presents problems Why? How? What-if? What is the difference? Evaluates meaning and correctness of the learners answers (LSA and computational linguistics)Gives feedback on answersFace displays emotions + some gesturesHintsPrompts for specific informationAdds information

7、 that is missedCorrects some bugs and misconceptionsAnswers student questionHolds mixed-initiative dialog in natural languagePedagogical Design Goals Simulate normal human tutors and ideal tutorsActive construction of student knowledge rather than information delivery systemCollaborative answering o

8、f deep reasoning questions Approximate evaluation of student knowledge rather than detailed student modelingA discourse prosthesisFeasibility of Natural Language Dialog in Tutoring Learners are forgiving when the tutors dialog acts are imperfect.They are even more forgiving when the bar is set low d

9、uring instructions.There are learning gains.Learning is not correlated with liking.Low ExpectedPrecisionHigh ExpectedPrecisionLow Common GroundYESMAYBEHigh Common GroundMAYBENODEMOHuman TutorsAnalyze hundreds of hours human tutorsResearch methods in college studentsBasic algebra in 7th gradeTypical

10、unskilled cross-age tutorsStudies from the Memphis labsGraesser & Person studiesStudies from other labsChi, Evens, McArthur Characteristics of students that we wish were betterStudent question askingComprehension calibrationSelf-regulated learning, monitoring, & and error correctionPrecise, symbolic

11、 articulation of knowledgeGlobal integration of knowledgeDistant anaphoric referenceAnalogical reasoning Application of principles to a practical problemPedagogical strategies not used by unskilled tutorsSocratic method (Collins, Stevens)Modeling-scaffolding-fading (Rogoff)Reciprocal training (Brown

12、, Palincsar)Anchored Learning (Bransford,Vye, CTGV)Error diagnosis & repair (Anderson, van Lehn, Lesgold)Building on prerequisites (Gagne)Cascade techniques (van Lehn, Schank)Sophisticated motivational techniques (Lepper)What can AutoTutor (and most human tutors) handle?Correct InformationErrors and

13、 MisconceptionsTutor expects and Student expressesYesYesTutor correctsTutor expects butStudent does not expressYesTutor helps to fill inYesNot manifestedTutor does not expect but Student expressesNo (or rarely)Tutor ignoresNo (or rarely)Tutor ignoresAutoTutorLanguage ExtractionSpeech actclassifierLa

14、tentSemanticAnalysisCurriculumScriptTalkinghead withgesturesDialog ManagementProblem selectionManaging One AutoTutor TurnShort feedback on the students previous turnAdvance the dialog by one or more dialog moves that are connected by discourse markersEnd turn with a signal that transfers the floor t

15、o the studentQuestionPrompting hand gestureHead/gaze signalDialog MovesPositive immediate feedback: “Yeah” “Right!”Neutral immediate feedback: “Okay” “Uh huh”Negative immediate feedback: “No” “Not quite”Pump for more information: “What else?”Hint: “How does tossing the pumpkin affect horizontal velo

16、city?” Prompt for specific information: “Vertical acceleration does not affect horizontal _.”Assert: “Vertical acceleration does not affect horizontal velicity.”Correct: “Air resistance is negligible”Repeat: “So, once again, how does tossing the pumpkin affect horizontal velocity?”Summarize: “So to

17、recap, succinct summary.”Discourse Marker RelationsSample dialogTutor: This might help. If the runner speeds up will the pumpkin land behind the runner, next to the runner, or in front of the runner?The pumpkin will land behind the runner.Tutor: Right. What is the horizontal force applied to the pum

18、pkin after it leaves the runners hands?There is no horizontal force applied to the pumpkin after it leaves the runners hands.Tutor: Right. expectation(1): 0.836expectation(2): 0.851expectation(3): 0.584misconception (1) 0.371misconception (2): 0.605As zero force is acting on it, it will haveconstant

19、 velocity hence it will land on his head.51.66.44Expectation-1 The pumpkin has the same horizontal velocity as the runner.Expectation-2 Zero force is needed to keep an object going with constant velocity.Expectation-3 Vertical forces on the pumpkin do not affect its horizontal velocity. The horizont

20、al velocity of the pumpkin is the same as the runner.The horizontal velocity of the pumpkin is the same as the runner.99.66.87Expectation-1 The pumpkin has the same horizontal velocity as the runner.Expectation-2 Zero force is needed to keep an object going with constant velocity.Expectation-3 Verti

21、cal forces on the pumpkin do not affect its horizontal velocity. How does Why/AutoTutor select the next expectation?Dont select expectations that the student has covered cosine(student answers, expectation) threshold Frontier learning, zone of proximal developmentSelect highest sub-threshold expecta

22、tionCoherenceSelect next expectation that has highest overlap with previously covered expectation Pivotal expectationsHow does AutoTutor know which dialog move to deliver?Dialog Advancer Network (DAN) for mixed-initiative dialog15 Fuzzy production rules Quality of the students assertion(s) in preced

23、ing turnStudent ability levelTopic coverageStudent verbosity (initiative)Hint-Prompt-Assertion cycles for expected good answersDialog Advancer NetworkHint-Prompt-Assertion Cycles to Cover Good Expectations Cycle fleshes out one expectation at a timeExit cycle when: cos(S, E ) TS = student input E =

24、expectation T = thresholdHintPromptAssertionHintAssertionPromptWho is delivering the answer?STUDENT PROVIDES INFORMATIONPumpHintPromptAssertionTUTOR PROVIDES INFORMATION Correlations between dialog moves and student abilityQuestion TaxonomyQUESTION CATEGORYGENERIC QUESTION FRAMES AND EXAMPLES1. Veri

25、fication Is X true or false? Did an event occur? Does a state exist?2. Disjunctive Is X, Y, or Z the case?3. Concept completion Who? What? When? Where?4. Feature specification What qualitative properties does entity X have?5. QuantificationWhat is the value of a quantitative variable? How much? How

26、many? 6. Definition questions What does X mean?7. Example questions What is an example or instance of a category?). 8. ComparisonHow is X similar to Y? How is X different from Y?9. InterpretationWhat concept/claim can be inferred from a static or active data pattern?10. Causal antecedentWhat state o

27、r event causally led to an event or state?Why did an event occur? Why does a state exist?How did an event occur? How did a state come to exist?11. Causal consequenceWhat are the consequences of an event or state?What if X occurred? What if X did not occur?12. Goal orientationWhat are the motives or

28、goals behind an agents action? Why did an agent do some action?13. Instrumental/proceduralWhat plan or instrument allows an agent to accomplish a goal? How did agent do some action?14. EnablementWhat object or resource allows an agent to accomplish a goal?15. ExpectationWhy did some expected event n

29、ot occur?Why does some expected state not exist?16. JudgmentalWhat value does the answerer place on an idea or advice?What do you think of X? How would you rate X?Speech Act ClassifierAssertions Questions (16 categories)DirectivesMetacognitive expressions (“Im lost”)Metacommunicative expressions (“C

30、ould you say that again?”)Short Responses95% Accuracy on tutee contributionsA New Query-based Information Retrieval System(Louwerse, Olney, Mathews, Marineau, Hite-Mitchell, Graesser, 2003)Input context: Text and Screen Select Highest Matching DocumentSyntactic ParserLexiconsSurface cuesFrozen expre

31、ssionsWord particles of question category Input speech actClassify speech act QUESTs 16 question categories, assertion, directive, otherAugment retrieval cuesSearch documents via LSAEvaluations of AutoTutorLEARNING GAINS (effect sizes) .42Unskilled human tutors(Cohen, Kulik, & Kulik, 1982) .75AutoTu

32、tor (7 experiments)(Graesser, Hu, Person)1.00Intelligent tutoring systems PACT (Anderson, Corbett, Koedinger)Andes, Atlas (VanLehn)2.00 (?)Skilled human tutorsLearning Gains (Effect Sizes)Spring 2002 EvaluationsConceptual Physics(VanLehn & Graesser, 2002)Four conditionsHuman tutorsWhy/AtlasWhy/AutoT

33、utorRead control86 College StudentsMeasures in Spring EvaluationMultiple Choice TestPretest and posttest (40 multiple choice questions in each)Essays graded by 6 physics experts 4 pretest and 4 posttest essaysExpectations versus misconceptionsWholistic gradesGeneric principles and misconceptions (fi

34、ne-grained)Learner perceptionsTime on TasksEffect Sizes on Learning Gains (pretest to posttest, no differences among tutoring conditions)Fall 2002 EvaluationsConceptual Physics(Graesser, Moreno, et al., 2003)Three tutoring conditionsWhy/AutoTutorRead textbook controlRead nothing63 subjectsMultiple C

35、hoice Scores 2002-3 EvaluationsComputer Literacy(Graesser, Hu, et al., 2003)2 Tutoring ConditionsAutoTutorRead nothing4 Media Conditions PrintSpeechSpeech+HeadSpeech+Head+Print96 subjectsDeep Reasoning QuestionsLATENT SEMANTIC ANALYSISSignal Detection AnalysesRecall, Precision, and F-measureWhat Exp

36、ectations are LSA-worthy?Compute correlation between:Experts ratings of whether essay answers have expectation E Maximum LSA cosine between E and all possible combinations of sentences in essayA high correlation means the expectation is LSA-worthyExpectations and Correlations (expert ratings, LSA)Af

37、ter the release, the only force on the balls is the force of the moons gravity (r = .71)A larger object will experience a smaller acceleration for the same force (r = .12)Force equals mass times acceleration (r = .67)The boxes are in free fall (r = .21)OTHER EMPIRICAL EVALUATIONSAssessment of Dialog

38、ue ManagementBystander Turing testParticipants rate whether particular dialog moves in conversations were generated by AutoTutor or by skilled human tutors.Bystander saysComputer said itBystander saysHuman said itReality:Computer said itHit .51Miss.49Reality:Human said itFalse alarm.53CR.47ASL Model

39、 501 Eye TrackerQUESTION4%ANSWER7%OFF20%TALKING HEAD40%DISPLAY29%(MAINLY KEYBOARD) Percentage of Time Allocated to Interface Components What Conversational Agents Facilitate Learning?Correlation matrix for DVsLikeCompCredQualitySyncComp.50*Cred.51*.33*Quality.54*.59*.49*Sync.56*.54*.31*.53*Learning.

40、03.07.02.04.03AutoTutor CollaborationsUniversity of Pittsburgh (VanLehn) ONR, physics intelligent tutoring systems, Why2University of Illinois, Chicago (Wiley, Goldman)NSF/ROLE, plate tectonics, eye tracking, critical stanceOld Dominion and Northern Illinois University (McNamara, Magliano, Millis, W

41、iemer-Hastings) IERI, science text comprehension.MIT Media LabNSR/ROLE, Learning Companion, emotion sensors (Picard, Reilly)BEAT, gesture, emotion and speech generator (Cassell, Bickmore)CHI Systems (Zachary, Ryder)Army SBIR, Think Like a CommanderInstitute for Defense Analyses (Fletcher, Toth, Fost

42、er)ONR/OSD, Human Use Regulatory Affairs Advisor, research ethics, web site with agentCollaboration with MIT Media LabAffect Computing LabFrustrationAngerConfusionEureka highsContemplation flow experienceInferring emotions from sensorsBlue EyesMouse-glove pressure and sweatButt Dialog moves sensitiv

43、e to emotionsForthcoming AutoTutor developmentsLanguage and discourse enhancementsWeave in deeper semantic processing componentsNatural language generation for prompts Improved animated conversational agent3-d simulation for enhancing the articulation of explanationsImprove authoring toolsEvolution

44、of the content of curriculum scripts through tutoring experienceThe Long-term VisionFuture human-computer interfaces will be conversational: Just like people talking face to face.Avatars will tutor and mentor learners on the web: students, soldiers, citizens, customers, elderly, special populations, low and high literacy, low and high motivationLearning modules will be