新未来工作趋势报告+Microsoft+New+Future+of+Work+Report+2023

MicrosoftNew

FutureofWork

Report2023

Asummaryofrecentresearchfrom

Microsoftandaroundtheworldthat

canhelpuscreateanewandbetter

futureofworkwithAI.

2

MicrosoftNewFutureofWorkReport

aka.ms/nfw

EditorsandAuthors

•Editors:

JennaButler

(PrincipalAppliedResearchScientist),

SoniaJaffe

(PrincipalResearcher),

NancyBaym

(SeniorPrincipalResearchManager),

MaryCzerwinski

(PartnerResearchManager),

ShamsiIqbal

(PrincipalApplied&DataScientist),

Kate

Nowak

(PrincipalAppliedScientist),

SeanRintel

(SeniorPrincipalResearcher),

AbigailSellen

(VPDistinguishedScientist),Mihaela

Vorvoreanu(DirectorAetherUXResearch&EDU),

BrentHecht

(PartnerDirectorofAppliedScience),and

JaimeTeevan

(Chief

ScientistandTechnicalFellow)

•Authors:NajeebAbdulhamid,JudithAmores,ReidAndersen,KagonyaAwori,MaxamedAxmed,danahboyd,JamesBrand,GeorgBuscher,DeanCarignan,MartinChan,AdamColeman,ScottCounts,MadeleineDaepp,AdamFourney,DanGoldstein,Andy

Gordon,AaronHalfaker,JavierHernandez,JakeHofman,JennyLay-Flurrie,VeraLiao,SiânLindley,SathishManivannan,CharltonMcilwain,SubigyaNepal,JenniferNeville,StephanieNyairo,JackiO'Neill,VictorPoznanski,GonzaloRamos,NaguRangan,LaceyRosedale,DavidRothschild,TaraSafavi,AdvaitSarkar,AvaScott,ChiragShah,NehaShah,TenyShapiro,RylandShaw,Auste

Simkute,JinaSuh,SiddharthSuri,IoanaTanase,LevTankelevitch,MengtingWan,RyenWhite,LongqiYang

Referencingthisreport:

•Onsocialmedia,pleaseincludethereportURL(

https://aka.ms/nfw2023

).

•Inacademicpublications,pleaseciteas:Butler,J.,Jaffe,S.,Baym,N.,Czerwinski,M.,Iqbal,S.,Nowak,K.,Rintel,R.,Sellen,A.,

Vorvoreanu,M.,Hecht,B.,andTeevan,J.(Eds.).MicrosoftNewFutureofWorkReport2023.MicrosoftResearchTechReportMSR-TR-2023-34(

https://aka.ms/nfw2023

),2023.

3

MicrosoftNewFutureofWorkReport

aka.ms/nfw

Welcometothe2023MicrosoftNewFutureofWorkReport!

Inthepastthreeyears,therehavebeennotonebuttwogenerationalshiftsinhowworkgetsdone,bothofwhichwereonly

possiblebecauseofdecadesofresearchanddevelopment.ThefirstshiftoccurredwhenCOVIDmadeusrealizehowpowerfulremoteandhybridworktechnologieshadbecome,aswellashowmuchsciencewasavailabletoguideusinhowto(andhownotto)usethesetechnologies.Thesecondarrivedthisyear,asitbecameclearthat,atlonglast,generativeAIhadadvancedtothepointwhereitcouldbevaluabletohugeswathsoftheworkpeopledoeveryday.

WebegantheNewFutureofWorkReportseries

in2021

,attheheightoftheshifttoremotework.Thegoalofthatreportwastoprovideasynthesisofnew–andnewlyrelevant–researchtoanyoneinterestedinreimaginingworkforthebetterasa

decades-oldapproachtoworkwaschallenged.ThesecondNewFutureofWorkReport,published

in2022

,focusedonhybridworkandwhatresearchcouldteachusaboutintentionallyre-introducingco-locationintopeople’sworkpractices.Thisyear’sedition,thethirdintheseries,continueswiththesamegoal,butcentersonresearchrelatedtointegratingLLMsintowork.

Throughout2023,AIandthefutureofworkhavefrequentlybeenonthemetaphorical–andoftenliteral–frontpagearoundtheworld.TherehavebeenmanyexcellentarticlesaboutthewaysinwhichworkmaychangeasLLMsareincreasingly

integratedintoourlives.Assuch,inthisyear’sreportwefocusspecificallyonareasthatwethinkdeserveadditionalattentionorwherethereisresearchthathasbeendoneatMicrosoftthatoffersauniqueperspective.Thisisareportthatshouldbereadasacomplementtotheexistingliterature,ratherthanasasynthesisofallofit.

Thisisararetime,oneinwhichresearchwillplayaparticularlyimportantroleindefiningwhatthefutureofworklookslike.Atthisspecialmoment,scientistscan’tjustbepassiveobserversofwhatishappening.Rather,wehavetheresponsibilitytoshapeworkforthebetter.Wehopethisreportcanhelpourcolleaguesaroundworldmakeprogresstowardsthisgoal.

-JaimeTeevan,ChiefScientistandTechnicalFellow

4

MicrosoftNewFutureofWorkReport

aka.ms/nfw

ThisreportemergesfromMicrosoft’sNewFutureofWorkinitiative

Microsofthashelpedshapeinformationworksinceitsfounding.However,aconfluenceofrecentcircumstances–remotework,

hybridwork,LLMs–havecreatedanunprecedentedopportunity

forthecompanytoreimaginehowAIandotherdigitaltechnologiescanmakeworkbetterforeveryone.

Sinceitsinception,theNewFutureofWork(NFW)initiativehasbroughttogetherresearchersfromabroadrangeof

organizationsanddisciplinesacrossMicrosofttofocusonthe

mostimportanttechnologiesshapinghowpeoplework.The

initiativeisworkingtocreatethenewfutureofwork–onethatisequitable,inclusive,meaningful,andproductive–insteadof

predictingorwaitingforit.Itdoesthisbyconductingprimary

researchandsynthesizingexistingresearchtosharewiththeresearchcommunity.Thisreportisoneofthemanypublic

resourcesithasproduced.

ThereadercanfindtheNewFutureofWorkinitiative’smany

otherresearchpapers,practicalguides,reportsandwhitepapersattheinitiative’swebsite:

https://aka.ms/nfw

.

https://aka.ms/nfw

MicrosoftNewFutureofWorkReport

aka.ms/nfw

Reportoverview

ThisreportprovidesinsightintoAIandworkpractices.Inityouwillfindcontentrelatedto:

•LLMsforInformationWork:HowdoLLMsaffectthespeedandqualityofcommoninformationworktasks?LLMscanboostproductivityforinformationworkers,buttheyalsorequirecarefulevaluationandadaptation.

•LLMsforCriticalThinking:HowcanLLMshelpusbreakdownandbuildupcomplextasks?LLMscanhelpustacklecomplextasksbyprovokingcriticalthinking,enablingmicroproductivity,andshiftingthebalanceofskills.

•Human-AICollaboration:HowcanwecollaborateeffectivelywithLLMs?EffectivecollaborationwithLLMsdependsonhowweprompt,complement,relyon,andauditthem.

•LLMsforComplexandCreativeTasks:HowcanLLMstackletasksthatgobeyondsimpleinformationretrievalorgeneration?LLMscansupportcomplexandcreativetasksby,forinstance,enhancingmetacognition.

•Domain-SpecificApplicationsofLLMs:HowareLLMsbeingusedandaffectingdifferentdomainsofwork?Wefocusspecificsonsoftwareengineering,medicine,socialscience,andeducation.

•LLMsforTeamCollaborationandCommunication:HowcanLLMshelpteamsworkandcommunicatebetter?LLMscanhelpteamsimproveinteraction,coordination,andworkflowsbyprovidingreal-time,retrospectivefeedbackandleveragingholisticframeworks.

•KnowledgeManagementandOrganizationalChanges:HowisAIchangingthenatureanddistributionofknowledgeinorganizations?LLMsmight,forinstance,finallyeliminateknowledgesilosinlargecompanies.

•ImplicationsforFutureWorkandSociety:WhatimplicationswillAIhaveforthefutureofworkandsociety?WecanshapeAI’simpactbyaddressingadoptiondisparities,fosteringinnovation,leadinglikescientists,andrememberingthatthefutureofworkisinourcontrol.

MicrosoftNewFutureofWorkReport

aka.ms/nfw

LabexperimentsshowLLMscansubstantiallyimproveproductivityon

commoninformationworktasks,althoughtherearesomequalifiers

LLM-basedtoolscanhelpworkerscompleteavarietyoftasksmorequicklyandincreaseoutputquality.

•StudieshavefoundthatpeoplecompletesimulatedinformationworktasksmuchfasterandwithahigherqualityofoutputwhenusinggenerativeAI-basedtools,

•Peopletook37%lesstimeoncommonwritingtasks(NoyandZhang2023)

•BCGconsultantsproduced>40%higherqualityononesimulatedconsultingproject(Dell’Acquaetal.2023).

•Userswerealso2xfasteratsolvingsimulateddecision-makingproblemswhenusingLLM-basedsearchovertraditionalsearch(Spathariotietal.2023).

•Forsometasks,increasedspeedcancomewithmoderatelylowercorrectness.

•WhentheLLMmademistakes,BCGconsultantswithaccesstothetoolwere19percentagepointsmorelikelytoproduceincorrectsolutions(Dell’Acquaetal.2023).

•Spathariotietal.(2023)developasimpleUX-basedinterventionscanworkwellathelpingpeoplenavigatethesetradeoffs.

•Usersmayneedhelpnegotiatingthetradeoffsinvolvedtomaximizeproductivitygains

•Howtask-levelgainstranslatetojob-levelgainswilldependonwhether

gainsextendtoothertasksandhowthetoolsareintegratedintoworkflows

Qualityofoutput(Treated=usingChatGPT)(Noy&Zhang2023)

Estimatesandconfidenceintervalsforaveragelog(time)bycondition,(Spathariotietal.2023)

Dell’Acqua,F.,etal.(2023).

NavigatingtheJaggedTechnologicalFrontier:FieldExperimentalEvidenceoftheEffectsofAIonKnowledgeWorkerProductivityandQuality

.SSRNWorkingPaper4573321.Noy,S.,&Zhang,W.(2023).

ExperimentalEvidenceontheProductivityEffectsofGenerativeArtificialIntelligence

.SSRNpreprint.

MicrosoftStudy:Spatharioti,S.E.,etal.(2023).

ComparingTraditionalandLLM-basedSearchforConsumerChoice:ARandomizedExperiment

.arXivpreprint.

6

TaskcompletiontimesforlabstudiesofCopilotforM365(Cambonetal2023)

MicrosoftNewFutureofWorkReport

aka.ms/nfw

CopilotforM365savestimeforavarietyoftasksinlabstudiesandsurveys

UsersalsoreportCopilotreducestheeffortrequired.Effectsonqualityaremostlyneutral

Microsoft’sAIandProductivityReportsynthesizesresultsfrom8earlystudies,mostfocusedontheuseofM365Copilot

forinformationworkertasksforwhichLLMsaremostlikelytoprovidesignificantvalue(Cambonetal.,2023).

•Tasksincludedmeetingsummarization,informationretrieval,andcontentcreation

•StudyparticipantswithCopilotcompletedexperimenter-designedtasksin26-73%asmuchtimeasthosewithoutCopilot

•AsurveyofenterpriseuserswithaccesstoCopilotalsoshowedsubstantialperceivedtimesavings

•73%agreedthatCopilothelpedthemcompletetasksfaster,and85%saiditwouldhelpthemgettoagoodfirstdraftfaster.

•Manystudiesfoundnostatisticallysignificantormeaningfuleffectonquality

•However,inthemeetingsummarizationstudywhereCopilotuserstookmuchlesstime,theirsummaries

included11.1outof15specificpiecesofinformationintheassessmentrubricversusthe12.4of15foruserswhodidnothaveaccesstoCopilot.

•Intheotherdirection,thestudyofM365DefenderSecurityCopilotfoundsecuritynoviceswithCopilotwere44%moreaccurateinansweringquestionsaboutthesecurityincidentstheyexamined.

•AstudyoftheOutlook“Soundlikeme”featurefoundCopilotuserslikemanyaspectsoftheemailsitgeneratedmorethanhuman-writtenones,butcouldsometimestellthedifferencebetweenCopilotwritingversushumanwriting.

•OfenterpriseCopilotusers,68%ofrespondentsagreedthatCopilotactuallyimprovedqualityoftheirwork.

•UsersalsoreportedtasksrequiredlesseffortwithCopilot

•IntheTeamsMeetingStudy,participantswithaccesstoCopilotfoundthetasktobe58%lessdrainingthanparticipantswithoutaccess

•AmongenterpriseCopilotusers,72%agreedthatCopilothelpedthemspendlessmentaleffortonmundaneorrepetitivetasks

7

MicrosoftStudy:Cambonetal(2023),

EarlyLLM-basedToolsforEnterpriseInformationWorkersLikelyProvideMeaningfulBooststoProductivity

.MSFTTechnicalReport.

MicrosoftNewFutureofWorkReport

aka.ms/nfw

TheevidencepointstoLLMshelpingtheleastexperiencedthemost

Mostlyearlystudieshavefoundthatneworlow-skilledworkersbenefitthemostfromLLMs.

•InstudyingthestaggeredrolloutofagenerativeAI-basedconversational

assistant,Brynjolfssonetal.(2023)foundthatthetoolhelpednoviceandlow-skilledworkersthemost.

•Theyfoundsuggestiveevidencethatthetoolhelpeddisseminatetacitknowledgethattheexperiencedandhigh-skilledworkersalreadyhad.

•Inalabexperiment,participantswhoscoredpoorlyontheirfirstwritingtask

improvedmorewhengivenaccesstoChatGPTthanthosewithhighscoresontheinitialtask(seegraph,NoyandZhang2023).

•Pengetal.(2023)alsofoundsuggestiveevidencethatGithubCopilotwasmorehelpfultodeveloperswithlessexperience.

•InanexperimentwithBCGemployeescompletingaconsultingtask,thebottom-halfofsubjectsintermsofskillsbenefitedthemost,showinga43%improvementinperformance,comparedtothetophalfwhoseperformanceincreasedby17%(Dell’Acquaetal.,2023).

•RecentworkbyHaslbergeretal.(2023)highlightssomecomplexitiesandnuanceinthesetrends,includingcasesinwhichLLMsmightincreaseperformance

disparities.

GreentrianglesrepresentthosewhogotaccesstoChatGPTforthesecondtask.Theirscoresacrossthetwotasksareless

correlated.(Noy&Zhang2023)

Brynjolfsson,E.,etal.(2023).

GenerativeAIatWork

.NBERWorkingPaper31161.

Haslberger,M.etal.(2023)

NoGreatEqualizer:ExperimentalEvidenceonAIintheUKLaborMarket

.SSRNWorkingPaper4594466,

8

Dell’Acqua,F.,etal.(2023).

NavigatingtheJaggedTechnologicalFrontier:FieldExperimentalEvidenceoftheEffectsofAIonKnowledgeWorkerProductivityandQuality

.SSRNWorkingPaper4573321.Noy,S.,&Zhang,W.(2023).

ExperimentalEvidenceontheProductivityEffectsofGenerativeArtificialIntelligence

.SSRNWorkingPaper4375283.

MicrosoftStudy:Peng,S.,etal.(2023).

TheImpactofAIonDeveloperProductivity:EvidencefromGitHubCopilot

.arXivpreprint2302.06590.

MicrosoftNewFutureofWorkReport

aka.ms/nfw

Criticalthinking:LLM-basedtoolscanbeusefulprovocateurs

ReconceptualizingAIsystemsas“provocateurs”inadditionto“assistants”canpromotecriticalthinking

inknowledgework

•AsAIisappliedtomoregenerativetasks,humanworkisshiftingto“criticalintegration”ofAIoutput,requiringexpertiseandjudgement(Sarkar2023).

•Movingbeyondjusterrorcorrection,AIprovocateurswouldchallengeassumptions,encourageevaluation,andoffercounterarguments.

•InteractiondesignofprovocativeAIneedstostrikeabalancebetweenusefulcriticismandoverwhelmingpeople.

•Frameworksthatstructurecriticalthinkingobjectives(e.g.,Bloom’s

taxonomy)andToulmin’smodeloperationalizeargumentanalysis,whichcouldinformprovocativeAIdesign(Kneupper1978).

•Interactivetechnologiesthatsparkdiscussionandengageuserscontributetocriticalthinkingdevelopment(Sunetal.2017;Leeetal.2023).

ImageofBloom’sTaxonomy(Bezjak,S.,etal.2018)

MicrosoftStudy:Sarkar,A.(2023).

ExploringPerspectivesontheImpactofArtificialIntelligenceontheCreativityofKnowledgeWork:BeyondMechanisedPlagiarismandStochasticParrots

ProceedingsoftheACMSymposiumonHuman-ComputerInteractionforWork(CHIWORK2023).

Kneupper,C.W.(1978).Teachingargument:AnintroductiontotheToulminmodel.CollegeCompositionandCommunication29,3..

9

Sun,N.,etal.(2017).Criticalthinkingincollaboration:Talkless,perceivemore.Proceedingsofthe2017CHIConferenceExtendedAbstractsonHumanFactorsinComputingSystems.

Lee,S.,etal.(2023).FosteringYouth’sCriticalThinkingCompetencyAboutAIthroughExhibition.Proceedingsofthe2023CHIConferenceonHumanFactorsinComputingSystems.

Bezjak,S.etal,(2018).

OpenScienceTrainingHandbook

MicrosoftNewFutureofWorkReport

aka.ms/nfw

AIcanenhancemicroproductivitypractices

AIcanbeharnessedtoaugmenthumancapabilitiesthroughnoveltaskmanagementstrategies

•Theconceptof“microproductivity”,inwhichcomplextasksaredecomposedintosmallersubtasksand

performedin“micromoments”bythepersonmostskilledtodoso,canbeenhancedthroughautomation(Teevan2016).

•Forexample,Kokkalisetal.(2013)demonstratedthathighleveltasksbrokenintomultistepactionplansthroughcrowdsourcingresultinpeoplecompletingsignificantlymoretasks(47.1%task

completion)comparedtothecontrolconditionofnoplans(37.8%).ThesebenefitswerescaledbyapplyingNLPalgorithmstoautomaticallycreateactionplansforalargervarietyoftasksbasedonatrainingsetofsimilartasks,andtheplanswerefurtherrefinedthroughhumanintervention.

•Kauretal.(2018)showedthatusingafixedvocabularytobreakdowncommentsinadocumentintoaseriesofsubtasksresultedina28%increaseinsubtasksthatcanbehandedofftocrowdsourcingorautomation,leavingasmallerpercentageofsubtasksleftforthedocumentauthor.

•AIcanhelpwithautomaticidentificationofmicromomentsandmicrotasks,improvingoverallqualityandefficiency.

•Contextualidentificationofmicromomentsbasedonprecedingactivitiesandlocationcanyieldupto80.7%precision(Kangetal.2017);suchmicromomentscanbeusedforlearning(Caietal.2017),

creationofaudiobooks(Kangetal.2017),editingdocuments(Augustetal.2020),andcoding(Williamsetal.2018).

•Whiteetal.(2021)demonstratedhowmachinelearningcanbeleveragedtoautomaticallydetectmicrotasksfromuser-generatedtasklistsresultinginapositiveprecisionof75%,andforecast

duration,withthebestclassifierperformancefortaskswithdurationof5minutes.

Decomposinghighleveltasksintoconcretesteps(plans)makesthemmoreactionableresultinginhighertaskcompletionrates.Online

crowdsdothedecomposition,algorithmsidentifyandreuseexistingplans.(Kokkalis2013)

MicrosoftStudy:Teevan,J.(2016).

Thefutureofmicrowork

.XRDS23,2.

Kokkalis,N.,etal.2013.TaskGenies:

AutomaticallyProvidingActionPlansHelpsPeopleCompleteTasks

.ACMTransactionsonComputer-HumanInteraction20,5.

Kaur,H.etal.2018.

CreatingBetterActionPlansforWritingTasksviaVocabulary-BasedPlanning

.ProceedingsoftheACMonHuman-ComputerInteraction.2,CSCW.

Kang,B.etal.(2017).Zaturi:

WePutTogetherthe25thHourforYou.CreateaBookforYourBaby

.InProceedingsofthe2017ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW‘17).Cai,C.J.,Ren,A.,&Miller,R.C.(2017).

WaitSuite:ProductiveUseofDiverseWaitingMoments

.ACMTransactionsonComputerHumanInteraction24,1.

10

MicrosoftStudy:August,T.,etal.(2020).

CharacterizingtheMobileMicrotaskWritingProcess

.22ndInternationalConferenceonHuman-ComputerInteractionwithMobileDevicesandServices(MobileHCI‘20).

MicrosoftStudy:Williams,A.,(2019).

Mercury:EmpoweringProgrammers'MobileWorkPracticeswithMicroproductivity

.Proceedingsofthe32ndAnnualACMSymposiumonUserInterfaceSoftwareandTechnology

MicrosoftStudy:White,R.W.,etal.(2021).

MicrotaskDetection.

ACMTrans.Inf.Syst.39,2.

MicrosoftNewFutureofWorkReport

aka.ms/nfw

Analyzingandintegratingmaybecomemoreimportantskillsthansearchingandcreating

WithcontentbeinggeneratedbyAI,knowledgeworkmayshifttowardsmoreanalysisandcriticalintegration

•Informationsearchaswellascontentproduction(manuallytyping,writingcode,designingimages)isgreatlyenhancedbyAI,sogeneralinformationworkmayshifttointegratingandcriticallyanalyzingretrievedinformation

•WritingwithAIisshowntoincreasetheamountoftextproducedaswellastoincreasewritingefficiency(Biermannetal.2022,Leeetal2022)

•Withmoregeneratedtextavailable,theskillsofresearch,

conceptualization,planning,promptingandeditingmaytakeonmore

importanceasLLMsdothefirstroundofproduction(e.g.,Mollick2023).

•Skillsnotdirectlytocontentproduction,suchasleading,dealingwithcriticalsocialsituations,navigatinginterpersonaltrustissues,and

demonstratingemotionalintelligence,mayallbemorevaluedintheworkplace(LinkedIn2023)

Thecriticalintegration“sandwich”:whenAIhandlesproduction,humancritical

thinkingisappliedateitherendoftheprocesstocompleteknowledge

workflows(Sarkar,2023).

Biermann,O.C.,etal.(2022).

FromTooltoCompanion:StorywritersWantAIWriterstoRespectTheirPersonalValuesandWritingStrategies

.Proceedingsofthe2022ACMDesigningInteractiveSystemsConference(DIS'22).Mina,L.,etal.(2022).

CoAuthor:DesigningaHuman-AICollaborativeWritingDatasetforExploringLanguageModelCapabilities

.Proceedingsofthe2022CHIConferenceonHumanFactorsinComputingSystems(CHI'22).

11

Mollick,E.(2023).

MyclassrequiredAI.Here'swhatI'velearnedsofar

.OneUsefulThing

LinkedIn(2023).

FutureofWorkReport:AIatWork

.

MicrosoftNewFutureofWorkReport

aka.ms/nfw

Constructingoptimalpromptsisdifficult

Promptsaretheprimaryinterfaceforbothusersanddeveloperstointeractwithlargelanguagemodels,butconsistentlydevelopingeffectivepromptsisachallenge

•PrecisepromptcompositioniscriticalinachievingthedesiredLLMoutput,withsemanticallysimilarpromptsyieldingsignificantlydifferent,sometimesincorrect,outputs(Jiangetal2020).

•Writingeffectivepromptscanrequiresignificanteffort,includingmultipleiterationsofmodificationandtesting(Jiangetal2022).

•Promptbehaviorcanbebrittleandnon-intuitive:

•Seeminglyminorchanges,includingcapitalizationandspacingcanresultindramaticallydifferentLLMoutputs(Holtzman2021,Aroraetal.2023)

•Theorderofpromptelements,suchassections,few-shotexamplesorevenwordscansignificantlyimpactaccuracy,insomecasesvaryingfromnearrandomchancetostate-of-the-art(Zhaoetal.2021,Kaddouretal.2023).

•Thesamepromptcanresultinsignificantlydifferentperformanceacrossmodelfamilies,evenwithmodelsofsimilarparametersize(Sanhetal.2022).

•Whilemanypromptingtechniqueshavebeendeveloped,thereislittletheoreticalunderstandingforwhyanyparticulartechniqueissuitedtoanyparticulartask(Zhaoetal.2021).

•Endusersofprompt-basedapplicationsstrugglemorethanpromptengineerstoformulateeffectiveprompts(Zamfirescu-Pereiraetal.2023).

Jiang,Z.etal.(2020).

HowCanWeKnowWhatLanguageModelsKnow?

TransactionsoftheAssociationforComputationalLinguistics,8.

Jiang,E.etal.(2022).

PromptMaker:Prompt-basedPrototypingwithLargeLanguageModels

.ExtendedAbstractsofthe2022CHIConferenceonHumanFactorsinComputingSystems

Holtzman,A.etal.(2021).SurfaceFormCompetition:WhytheHighestProbabilityAnswerIsn’tAlwaysRight.EMNLP.

Arora,S.etal.(2023).

Askmeanything:Asimplestrategyforpromptinglanguagemodels

.TheEleventhInternationalConferenceonLearningRepresentations.

Zhao,Z.,etal.(2021).

CalibrateBeforeUse:ImprovingFew-shotPerformanceofLanguageModels

.Proceedingsofthe38thInternationalConferenceonMachineLearning.

Kaddour,J.,etal.(2023).

ChallengesandApplicationsofLargeLanguageModels

.arXivpreprint.

12

Sanh,V.etal.(2022)

MultitaskPromptedTrainingEnablesZero-ShotTaskGeneralization

.InternationalConferenceonLearningRepresentations

Zamfirescu-Pereira,J.D.,etal.(2023).

WhyJohnnyCan’tPrompt:HowNon-AIExpertsTry(andFail)toDesignLLMPrompts

.(CHI'23).

MicrosoftNewFutureofWorkReport

aka.ms/nfw

Butconstructingeffectivepromptsisbecomingeasier

Basemodeltraining,toolsandLLMsthemselvesarehelpingimprovepromptperformance

•Significantresearchisdevotedtoimprovingmodelinstructionfollowing.

•Fine-tuningwithhumanfeedbackcandramaticallyimproveLLMsabilitytofollowpromptinstructions,evenwhencomparedtomodelswith100xparameters(Ouyangetal.2022).

•Utilizingmulti-taskandchain-of-thoughttrainingdatasignificantlyimprovedinstruction-followingcapabilities(Chungetal.2022).

•LLMshavebeenshowntobeeffectivepromptoptimizers.

•PromptoptimizationtechniquesthatutilizeanLLMtoiterativelyprovidefeedbackandproducenewversionsofahand-craftedseedpromptcansignificantlyimproveperformance(Pryzantetal.2023).

•Multi-stepoptimizationwithnaturallanguagetaskdescriptionsandscoredoptimizationexamplescaninduceanLLMtogeneratenew,higherperformingpromptvariations(Yangetal.2023).

•Inspiredbyevolutionaryalgorithms,anLLMcanbeusedtogeneratenewpromptcandidatesbymutatingpromptsfromapopulation,evaluatingtheirfitnessagainstatestsetovermultiplegenerations(Fernandoetal.2023).

•Recentworksuggestsoptimizedpromptscanoutperformspecificallyfine-tunedmodelsinanumberofimportantdomains,especiallymedicine(Norietal.2023).

Ouyang,L.,etal.(2022).Traininglanguagemodelstofollowinstructionswithhumanfeedback.AdvancesinNeuralInformationProcessingSystems,35.

Chung,H.W.,etal.(2023)

Scalinginstruction-finetunedlanguagemodels

.arXivpreprint.

Pryzant,R.,etal.(2023).

AutomaticPromptOptimizationwithGradientDescentandBeamSearch

.arXivpreprint.

Yang,C.,etal.(2023).

Largelanguagemodelsasoptimizers

.arXivpreprint.

13

Nori,Harsha,etal.

CanGeneralistFoundationModelsOutcompeteSpecial-PurposeTuning?CaseStudyinMedicine

arXivpreprint.

Fernando,C.,etal.(2023).

Promptbreeder:Self-referentialself-improvementviapromptevolution

.arXivpreprint.

MicrosoftNewFutureofWorkReport

aka.ms/nfw

Peoplearealsolearningtopromptmoreeffectively

AspeoplegetbetteratcommunicatingwithLLMs,theyaregettingbetterresults

•Promptguidanceiscommonlyusedasawayforpeopletolearntopromptbetter.

•ResearchsuggeststhattrainingonhowtopromptcanleadtogreaterproductivitygainsfromLLMtools(Dell’Acquaetal.2023).

•Usingalensinformedbythepsycholinguisticconceptofgrounding(Clark1996),Teevan(2023)arguesinHBRthateffectivecommunicationwithgenerativeAIrequiresprovidingcontextualinformation,specifyingthedesiredoutput,andverifyingtheaccuracyofthegeneratedcontent.

•Manyotherguidesandreferencematerialsarealsoavailable,includingarecentWorkLabarticle(Microsoft2023)andOpenAI’sdocumentation