Artificial Intelligence in Education

Artificial Intelligence in Education

C.-K. Looi et al. (Eds.)

IOS Press, 2005

© 2005 The authors. All rights reserved.

Toward supporting hypothesis formation and

testing in an interpretive domain

VincentAleven1andKevinAshley2

1

Human-ComputerInteractionInstitute

CarnegieMellonUniversity

2

IntelligentSystemsProgram,LearningResearchandDevelopmentCenter

UniversityofPittsburgh

Abstract.TheresearchfieldofAI&Educationhaslongbeeninterestedincognitiveprocessesinwhichstudentsformulateandtesthypothesesbyconsideringtheminlightofspecificcases.However,fewifanyofthesystemsthathavebeenbuilttargetdomainswhichareill-structuredandinwhichdeterminingwhetherahypothesizedruleandproposedoutcomeareconsistentwithpastdecisionsisamatterofinterpretation,ratherthandeductiveinference.Thegoalsofourprojectareto(1)developanAImodelofhypothesisformationandtestinginaninterpretivedomain,USSupremeCourtoralargumentsand(2)touseitinanintelligenttutoringsystemtoguidelawstudentsinlearningthatprocess.Asafirststeptowardthesegoalswewillconductanexperimenttoevaluatewhetherself-explanationpromptsfacilitatelearningbystudyingargumenttranscripts.

Introduction

TheresearchfieldofAI&Educationhaslongbeeninterestedinprocessesofinquirylearninginwhichstudentsformulatehypothesesandtestthemagainstspecificcases([1-5]).Theseprocessesastheyoccurinill-structureddomains,however,havereceivedlittleattention, one exception being the workonCATO[6-8].OralargumentsbeforetheUnitedStates Supreme Court offer prime examples ofhypothesisformulationandtestinginanill-structured domain. In these arguments, advocatesframehypothesesfordecidingacaseandtheJusticeschallengethem,oftenbyposinghypotheticalscenariosthattestthehypotheses’limits.Whiletheseprocessesinthelegaldomainbearsomeresemblancetothecorresponding processes in science or mathematics, in the legaldomaindeterminingwhetherahypothesizedruleandproposedoutcomeareconsistentwithpastdecisionsandplausiblehypotheticalsismuchmoreamatterofinterpretation.

The goals ofourprojectare(1)todevelopacomputationalmodelofthereasoning

processes exemplified inUSSupremeCourtoralargumentand(2)tousethemodelasthebasisforanintelligenttutoringsystemthatwillengagestudentsinanappropriatelysimplifiedversionoftheseprocesses.

Aplannedexperiment

Asafirststep,wewillrunanexperimenttofindoutwhetherspecificpromptsforself-explanationhelpstudentsgainadeeperunderstanding,astheystudytranscriptsofSupremeCourtoralarguments.Thecognitivescienceliteraturesupportsthatstudyingexamples is an effective learning strategy at the early stages ofacquiringacognitiveskill[9]andthatself-explanationpromptscanhelpstudentsgainadeeperunderstandingofthesubjectmatter([10-12]).However,theeffectivenessofpromptshasnotyetbeenshowninill-structured domains as complex as the legal reasoning exemplified in Supreme Court oral

V.AlevenandK.Ashley/TowardSupportingHypothesisFormationandTesting

Table1:ExcerptoftranscriptoforalargumentmadebeforetheUSSupremeCourtinCaliforniav.

Carney,105S.Ct.2066(1985),withself-explanationpromptsadded

ArgumentTranscript

Self-ExplanationPrompts

733

QUESTION:Well,whatifthevehicleisinoneofthesemobilehomeparksandhookeduptowaterandelectricitybutstillhasitswheelson?

MR.HANOIAN:[*9]Ifitstillhasitswheelsanditstillhasitsengine,itiscapableofmovementanditiscapableofmovementveryquickly.

QUESTION:Eventhoughthepeoplearelivinginitasahomeandarepayingrentforthetrailerspace,andsoforth?

QUESTION: Well, there are places wherepeoplecanplugintowater,andelectricity,anddo.Therearemanyplaces,forexample,inthestateIcamefromwherepeoplegoandspendthewinterinamobilehome.Andyouthinktherewouldbenoexpectationofprivacyinsuchcircumstances?

1.DoyouthinkH'sresponseiseffective?

2.WhyaretheJusticesaddingthesefeaturestothehypothetical?

MR.HANOIAN:Well,Iamnotsuggestingthatthereisnoexpectationofprivacyinthosecircumstances,YourHonor.

3.Whydoesitmatterwhethertherewouldbeexpectationsofprivacy?

4.Ifitwasclearthatthereis,orshouldbe,ahighexpectationofprivacyinthecurrentfactsituation,wouldthatfavorH'sposition?

5.Nothingissaidfromwhichwecaninferhowthisparticularhypotheticalshouldbedecided.Doesthatmatter?Thatis,whatgoodisittousehypotheticalswhoseoutcomeisunknown?Wouldn'titbebettertocitepastcases,whoseoutcomewedoknow?

6.Byconcedingthatthereareexpectationsofprivacyinthehypotheticalscenariossketchedbythejudges,doesHnotreducehischancesofwinningthecaseathand?

7.DoesHconcedethatthemobilehomeparkhypotheticalshouldhavetheoppositeresultasthecaseathand?

8.HowwouldHdistinguishthecurrentcasefromthemobilehomeparkhypothetical?

arguments.Inlightoftheevidencethatpromptsdonotbenefitallstudentsequally([10,11]), it is important to ask how effective prompts are in such challenging domains.

Table1showsexcerptsfromoralargumentsmadeinthecaseofCaliforniav.Carney,105S.Ct.2066(1985),withself-explanationpromptsinserted.Thiscaseinvolvedthelegalityunderthe4thAmendmentoftheUSConstitutionofawarrantlesssearchofamotorhomelocatedinadowntownSanDiegoparkinglot.PolicesuspecteddefendantCarneyoftradingmarijuanaforsexacts.AftertheyquestionedaboyleavingCarney’smotorhome,agentsenteredthemotorhomewithoutawarrantorCarney’sconsent,observedmarijuana,andarrestedCarney.Thecasepittedtwoconflictingprinciples:theState’srighttodealeffectivelywiththeexigentpossibilitythatevidenceofacrimewilldisappearversusthecitizen’sconstitutionallyprotectedexpectationofautonomyandprivacyinhishome.Intheoralargument,theState’sattorney,Mr.Hanoian,proposedabright line test: if the vehicle/home is capable of self-locomotion, then no warrant isrequiredtosearchit.AsshowninTable1,hethenhastorespondtotheJustice’schallengehypothetical:whatresultwouldhistestproducewhenappliedtoasummermotorhomewithwheelsthatishookeduptoutilities?Mr.Hanoianrespondsthatsuchavehiclestillmightbemovedinahurry,butconcedestheownerswouldhavesomeexpectationofprivacy. Some of the self-explanation promptsfocusontheeffectivenessofthatresponse.OthersfocusontheJustices’strategiesandpossiblereasonsforposinghypotheticals.Discussion

Inordertoevaluatetheeffectoftheself-explanationprompts,thestudywillcomparethelearningresultsofstudentsstudyingargumenttranscriptswithandwithoutself-explanation

734V.AlevenandK.Ashley/TowardSupportingHypothesisFormationandTesting

prompts.Apilotstudyinvolvingtwolawstudents,afirst-yearstudentandasecond-yearstudent,providedsomeevidencethatthepromptsareuseful.ThestudentswentthroughtheCarneytranscripttwice,thefirsttimewithoutself-explanationprompts,thesecondtimewith.Eachtime,theywereaskedtoansweranumberofquestionsabouttheargumentexchange they had just studied.Wesawadifferenceinthequalityoftheanswersbetweenthefirst-yearandthesecond-yearstudent,indicatingthatthematerialischallenging.Further,wesawthattheanswersofthefirst-yearstudentimproved,afterstudyingthetranscriptwiththeself-explanationprompts.Ofcourse,suchevidenceispreliminary,duetothe“smallN”.Also,theimprovementintheanswerscouldbeattributedsimplytothefactthatthestudentwentthroughthetranscripttwice.Thisconfoundwillbeavoidedintheactualexperimentbyhavingacontrolgroup.Wearecurrentlyworkingondevelopingasuitabletaskbywhichwecanmeasureanyimprovementinstudents’argument-makingcapabilities, a preliminary challenge for any research in an ill-structured domain.

Weexpectthestudytoyieldinformationabouthowstudentsunderstandandmake

arguments. This information will helpusstarttobuildanargumentmodelanddevelopanintelligenttutoringsystem.Thestudywillalsocontributetocognitivesciencebytestingwhetherspecificself-explanationpromptscanhelpstudentstolearntoengageinaprocessofhypothesisformationandtestinginanill-structureddomain.Acknowledgements

ThisresearchissponsoredbyNSFAwardIIS-0412830.ThecontentsofthepaperaresolelytheresponsibilityoftheauthorsanddonotnecessarilyrepresenttheofficialviewsoftheNSF.

References

Bunt,A.,C.Conati,andK.Muldner,ScaffoldingSelf-ExplanationtoImproveLearninginExploratoryLearningEnvironments,inProceedingsofthe7thInternationalConferenceonIntelligentTutoringSystems,ITS2004,J.Lester,R.M.Vicario,andF.Paraguaçu,Editors.2004,Springer:Berlin.

[2].Collins,A.andA.L.Stevens,GoalsandStrategiesofInquiryTeachers,inAdvancesinInstructional

Psychology,R.Glaser,Editor.1982,LawrenceErlbaum:Hillsdale,NJ.p.65-119.

[3].Murray,T.,L.Winship,andN.Stillings,EvaluationoftheSimForestInquiryLearningEnvironment:

InquiryCyclesandCollaborativeTeachingPractices,inAERA.2004:SanDiego,CA.

[4].Shute,V.J.andR.Glaser,ALarge-ScaleEvaluationofanIntelligentDiscoveryWorld:Smithtown.

InteractiveLearningEnvironments,1990.1:p.51-77.

[5].Woolf,B.P.,etal.,TrackingStudentPropositionsinanInquirySystem,inProceedingsofthe11th

InternationalConferenceonArtificialIntelligenceinEducation,AIED2003,U.Hoppe,F.Verdejo,andJ.Kay,Editors.2003,IOSPress:Amsterdam.p.21-28.

[6].Aleven,V.,UsingBackgroundKnowledgeinCase-BasedLegalReasoning:AComputationlModeland

anIntelligentLearningEnvironment.ArtificialIntelligence,2003.150:p.183-238.

[7].Aleven,V.andK.D.Ashley,TeachingCase-BasedArgumentationThroughaModelandExamples:

EmpiricalEvaluationofanIntelligentLearningEnvironment,inProceedingsofthe8thInternational

ConferenceonArtificialIntelligenceandEducation,AI-ED'97,B.duBoulayandR.Mizoguchi,Editors.1997,IOSPress:Amsterdam.p.87-94.

[8].Ashley,K.D.,R.Desai,andJ.Levine,TeachingCase-BasedArgumentationConceptsUsingDialectic

Argumentsvs.DidacticExplanations,inProceedingsoftheSixthInternationalConferenceonIntelligentTutoringSystems,ITS2002,S.A.Cerri,G.Gouardères,andF.Paraguaçu,Editors.2002,Spinger:Berlin.p.585-595.

[9].Atkinson,R.K.,etal.,Learningfromexamples:Instructionalprinciplesfromtheworkedexamples

research.ReviewofEducationalResearch,2000.70(2):p.181-214.

[10].Chi,M.T.H.,etal.,ElicitingSelf-ExplanationsImprovesUnderstanding.CognitiveScience,1994.18:p.

439-477.

[11].Renkl,A.,etal.,LearningfromWorked-OutExamples:theEffectsofExampleVariabilityandElicited

Self-Explanations.ContemporaryEducationalPsychology,1998.23:p.90-108.

[12].Schworm,S.andA.Renkl,Learningbysolvedexampleproblems:Instructionalexplanationsreduceself-explanationactivity,inProceedingofthe24thAnnualConferenceoftheCognitiveScienceSociety,W.D.GrayandC.D.Schunn,Editors.2002,LawrenceErlbaum:Mahwah,NJ.p.816-821.[1].